Sep 16 17:22:36.409654 ip-10-0-129-160 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 17:22:36.409664 ip-10-0-129-160 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 17:22:36.409672 ip-10-0-129-160 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 17:22:36.409939 ip-10-0-129-160 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 17:22:46.502330 ip-10-0-129-160 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 17:22:46.502343 ip-10-0-129-160 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 282ca857efc64e05a1ebfa5cc4fea074 --
Sep 16 17:24:36.300104 ip-10-0-129-160 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 17:24:36.694333 ip-10-0-129-160 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 17:24:36.694333 ip-10-0-129-160 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 17:24:36.694333 ip-10-0-129-160 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 17:24:36.694333 ip-10-0-129-160 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 17:24:36.694333 ip-10-0-129-160 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 17:24:36.695198 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.695114    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 17:24:36.698747 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698732    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:36.698747 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698746    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:36.698834 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698750    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:36.698834 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698753    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:36.698834 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698757    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:36.698834 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698760    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:36.698834 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698763    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:36.698834 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698766    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:36.698834 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698769    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:36.698834 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698772    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:36.698834 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698775    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:36.698834 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698778    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:36.698834 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698780    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:36.698834 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698783    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:36.698834 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698785    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:36.698834 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698788    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:36.698834 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698791    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:36.698834 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698793    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:36.698834 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698796    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:36.698834 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698799    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:36.698834 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698818    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:36.698834 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698821    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:36.699303 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698824    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:36.699303 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698826    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:36.699303 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698829    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:36.699303 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698832    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:36.699303 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698834    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:36.699303 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698837    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:36.699303 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698839    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:36.699303 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698842    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:36.699303 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698845    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:36.699303 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698848    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:36.699303 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698851    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:36.699303 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698853    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:36.699303 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698856    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:36.699303 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698858    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:36.699303 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698861    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:36.699303 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698864    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:36.699303 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698866    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:36.699303 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698869    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:36.699303 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698872    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:36.699303 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698874    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:36.699773 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698877    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:36.699773 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698879    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:36.699773 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698882    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:36.699773 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698884    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:36.699773 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698889    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:36.699773 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698893    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:36.699773 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698896    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:36.699773 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698899    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:36.699773 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698901    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:36.699773 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698904    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:36.699773 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698906    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:36.699773 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698909    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:36.699773 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698912    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:36.699773 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698915    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:36.699773 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698918    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:36.699773 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698920    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:36.699773 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698923    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:36.699773 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698925    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:36.699773 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698928    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:36.699773 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698932    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:36.700360 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698935    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:36.700360 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698938    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:36.700360 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698941    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:36.700360 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698945    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:36.700360 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698948    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:36.700360 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698951    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:36.700360 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698954    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:36.700360 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698957    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:36.700360 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698960    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:36.700360 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698962    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:36.700360 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698965    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:36.700360 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698968    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:36.700360 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698971    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:36.700360 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698973    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:36.700360 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698976    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:36.700360 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698979    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:36.700360 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698982    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:36.700360 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698985    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:36.700360 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698987    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:36.700360 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698991    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:36.701202 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698994    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:36.701202 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698996    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:36.701202 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.698999    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:36.701202 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699001    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:36.701202 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699009    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:36.701202 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699013    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:36.701202 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699016    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:36.701202 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699018    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:36.701202 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699021    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:36.701202 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699023    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:36.701202 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699026    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:36.701202 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699028    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:36.701202 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699031    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:36.701202 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699033    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:36.701202 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699036    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:36.701202 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699038    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:36.701202 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699041    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:36.701202 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699709    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:36.701202 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699715    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:36.701677 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699718    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:36.701677 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699721    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:36.701677 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699726    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:36.701677 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699730    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:36.701677 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699733    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:36.701677 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699736    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:36.701677 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699739    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:36.701677 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699742    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:36.701677 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699745    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:36.701677 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699748    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:36.701677 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699751    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:36.701677 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699753    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:36.701677 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699756    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:36.701677 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699759    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:36.701677 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699762    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:36.701677 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699765    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:36.701677 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699767    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:36.701677 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699769    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:36.701677 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699773    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:36.701677 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699776    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:36.702191 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699779    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:36.702191 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699782    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:36.702191 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699785    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:36.702191 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699787    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:36.702191 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699790    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:36.702191 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699792    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:36.702191 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699795    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:36.702191 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699798    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:36.702191 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699814    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:36.702191 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699817    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:36.702191 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699819    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:36.702191 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699822    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:36.702191 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699825    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:36.702191 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699827    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:36.702191 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699830    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:36.702191 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699833    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:36.702191 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699835    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:36.702191 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699838    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:36.702191 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699841    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:36.702191 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699844    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:36.702780 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699847    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:36.702780 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699849    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:36.702780 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699852    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:36.702780 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699854    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:36.702780 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699858    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:36.702780 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699862    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:36.702780 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699865    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:36.702780 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699868    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:36.702780 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699871    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:36.702780 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699874    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:36.702780 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699878    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:36.702780 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699881    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:36.702780 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699884    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:36.702780 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699886    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:36.702780 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699889    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:36.702780 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699891    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:36.702780 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699894    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:36.702780 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699896    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:36.702780 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699899    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:36.703270 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699901    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:36.703270 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699904    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:36.703270 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699907    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:36.703270 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699909    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:36.703270 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699912    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:36.703270 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699914    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:36.703270 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699917    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:36.703270 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699919    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:36.703270 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699921    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:36.703270 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699924    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:36.703270 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699928    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:36.703270 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699930    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:36.703270 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699933    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:36.703270 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699936    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:36.703270 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699938    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:36.703270 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699941    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:36.703270 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699943    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:36.703270 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699946    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:36.703270 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699949    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:36.703270 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699951    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:36.703764 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699954    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:36.703764 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699957    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:36.703764 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699959    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:36.703764 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699962    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:36.703764 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699971    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:36.703764 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699976    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:36.703764 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699978    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:36.703764 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699981    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:36.703764 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699983    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:36.703764 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699986    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:36.703764 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699988    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:36.703764 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699991    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:36.703764 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699993    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:36.703764 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699996    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:36.703764 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.699998    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:36.703764 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.700001    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:36.703764 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.700003    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:36.703764 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.700007    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:36.703764 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703049    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:36.704257 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703060    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:36.704257 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703064    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:36.704257 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703067    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:36.704257 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703070    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:36.704257 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703075    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:36.704257 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703078    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:36.704257 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703080    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:36.704257 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703083    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:36.704257 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703086    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:36.704257 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703089    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:36.704257 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703092    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:36.704257 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703095    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:36.704257 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703097    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:36.704257 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703100    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:36.704257 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703103    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:36.704257 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703105    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:36.704257 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703108    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:36.704257 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703111    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:36.704257 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703113    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:36.704257 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703116    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:36.704738 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703118    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:36.704738 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703121    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:36.704738 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703123    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:36.704738 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703126    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:36.704738 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703129    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:36.704738 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703131    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:36.704738 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703134    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:36.704738 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703136    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:36.704738 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703138    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:36.704738 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703141    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:36.704738 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703144    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:36.704738 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703147    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:36.704738 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703150    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:36.704738 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703153    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:36.704738 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703156    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:36.704738 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703158    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:36.704738 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703161    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:36.704738 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703163    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:36.704738 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703166    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:36.704738 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703168    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:36.705253 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703171    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:36.705253 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703173    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:36.705253 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703176    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:36.705253 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703178    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:36.705253 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703181    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:36.705253 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703183    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:36.705253 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703186    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:36.705253 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703190    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:36.705253 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703194    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:36.705253 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703196    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:36.705253 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703199    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:36.705253 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703202    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:36.705253 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703204    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:36.705253 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703206    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:36.705253 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703209    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:36.705253 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703212    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:36.705253 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703214    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:36.705253 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703217    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:36.705253 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703219    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:36.705253 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703222    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:36.705774 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703225    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:36.705774 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703228    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:36.705774 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703231    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:36.705774 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703233    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:36.705774 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703237    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:36.705774 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703239    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:36.705774 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703242    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:36.705774 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703245    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:36.705774 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703248    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:36.705774 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703250    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:36.705774 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703253    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:36.705774 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703256    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:36.705774 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703258    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:36.705774 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703261    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:36.705774 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703264    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:36.705774 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703267    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:36.705774 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703271    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:36.705774 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703274    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:36.705774 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703278    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:36.706258 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703282    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:36.706258 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703286    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:36.706258 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703289    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:36.706258 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703292    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:36.706258 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703295    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:36.706258 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703298    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:36.706258 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703301    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:36.706258 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703304    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:36.706258 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703307    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:36.706258 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703309    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:36.706258 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703312    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:36.706258 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703315    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:36.706258 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703318    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:36.706258 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703321    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:36.706258 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703323    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:36.706258 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703326    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:36.706258 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703329    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:36.706258 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703332    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:36.706258 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.703334    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:36.706258 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703420    2562 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 17:24:36.706725 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703465    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 17:24:36.706725 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703471    2562 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 17:24:36.706725 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703476    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 17:24:36.706725 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703480    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 17:24:36.706725 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703483    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 17:24:36.706725 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703487    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 17:24:36.706725 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703491    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 17:24:36.706725 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703495    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 17:24:36.706725 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703498    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 17:24:36.706725 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703501    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 17:24:36.706725 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703505    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 17:24:36.706725 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703508    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 17:24:36.706725 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703511    2562 flags.go:64] FLAG: --cgroup-root=""
Sep 16 17:24:36.706725 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703514    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 17:24:36.706725 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703517    2562 flags.go:64] FLAG: --client-ca-file=""
Sep 16 17:24:36.706725 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703521    2562 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 17:24:36.706725 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703524    2562 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 17:24:36.706725 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703529    2562 flags.go:64] FLAG: --cluster-domain=""
Sep 16 17:24:36.706725 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703532    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 17:24:36.706725 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703535    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 17:24:36.706725 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703538    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 17:24:36.706725 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703542    2562 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 17:24:36.706725 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703545    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 17:24:36.706725 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703548    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 17:24:36.707320 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703551    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 17:24:36.707320 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703555    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 17:24:36.707320 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703558    2562 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 17:24:36.707320 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703561    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 17:24:36.707320 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703564    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 17:24:36.707320 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703567    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 17:24:36.707320 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703570    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 17:24:36.707320 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703575    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 17:24:36.707320 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703578    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 17:24:36.707320 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703580    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 17:24:36.707320 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703583    2562 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 17:24:36.707320 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703586    2562 flags.go:64] FLAG: --enable-server="true"
Sep 16 17:24:36.707320 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703589    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 17:24:36.707320 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703593    2562 flags.go:64] FLAG: --event-burst="100"
Sep 16 17:24:36.707320 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703596    2562 flags.go:64] FLAG: --event-qps="50"
Sep 16 17:24:36.707320 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703599    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 17:24:36.707320 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703602    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 17:24:36.707320 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703605    2562 flags.go:64] FLAG: --eviction-hard=""
Sep 16 17:24:36.707320 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703609    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 17:24:36.707320 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703612    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 17:24:36.707320 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703615    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 17:24:36.707320 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703618    2562 flags.go:64] FLAG: --eviction-soft=""
Sep 16 17:24:36.707320 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703621    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 17:24:36.707320 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703624    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 17:24:36.707320 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703627    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 17:24:36.707957 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703630    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 17:24:36.707957 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703633    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 17:24:36.707957 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703636    2562 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 17:24:36.707957 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703639    2562 flags.go:64] FLAG: --feature-gates=""
Sep 16 17:24:36.707957 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703642    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 17:24:36.707957 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703645    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 17:24:36.707957 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703648    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 17:24:36.707957 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703651    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 17:24:36.707957 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703654    2562 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 17:24:36.707957 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703657    2562 flags.go:64] FLAG: --help="false"
Sep 16 17:24:36.707957 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703660    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-129-160.ec2.internal"
Sep 16 17:24:36.707957 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703663    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 17:24:36.707957 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703666    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 17:24:36.707957 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703669    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 17:24:36.707957 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703673    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 17:24:36.707957 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703676    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 17:24:36.707957 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703679    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 17:24:36.707957 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703682    2562 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 17:24:36.707957 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703685    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 17:24:36.707957 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703687    2562 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 17:24:36.707957 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703690    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 17:24:36.707957 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703693    2562 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 17:24:36.707957 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703696    2562 flags.go:64] FLAG: --kube-reserved=""
Sep 16 17:24:36.707957 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703699    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 17:24:36.708549 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703702    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 17:24:36.708549 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703704    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 17:24:36.708549 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703710    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 17:24:36.708549 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703713    2562 flags.go:64] FLAG: --lock-file=""
Sep 16 17:24:36.708549 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703716    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 17:24:36.708549 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703719    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 17:24:36.708549 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703722    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 17:24:36.708549 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703728    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 17:24:36.708549 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703731    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 17:24:36.708549 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703734    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 17:24:36.708549 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703737    2562 flags.go:64] FLAG: --logging-format="text"
Sep 16 17:24:36.708549 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703740    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 17:24:36.708549 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703744    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 17:24:36.708549 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703747    2562 flags.go:64] FLAG: --manifest-url=""
Sep 16 17:24:36.708549 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703750    2562 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 17:24:36.708549 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703755    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 17:24:36.708549 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703758    2562 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 17:24:36.708549 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703762    2562 flags.go:64] FLAG: --max-pods="110"
Sep 16 17:24:36.708549 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703765    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 17:24:36.708549 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703768    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 17:24:36.708549 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703771    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 17:24:36.708549 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703774    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 17:24:36.708549 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703778    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 17:24:36.708549 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703781    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 17:24:36.708549 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703783    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 17:24:36.709222 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703791    2562 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 17:24:36.709222 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703794    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 17:24:36.709222 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703797    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 17:24:36.709222 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703813    2562 flags.go:64] FLAG: --pod-cidr=""
Sep 16 17:24:36.709222 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703819    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 17:24:36.709222 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703825    2562 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 17:24:36.709222 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703828    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 17:24:36.709222 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703831    2562 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 17:24:36.709222 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703834    2562 flags.go:64] FLAG: --port="10250"
Sep 16 17:24:36.709222 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703837    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 17:24:36.709222 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703842    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d49190a4ae8803b9"
Sep 16 17:24:36.709222 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703845    2562 flags.go:64] FLAG: --qos-reserved=""
Sep 16 17:24:36.709222 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703848    2562 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 17:24:36.709222 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703851    2562 flags.go:64] FLAG: --register-node="true"
Sep 16 17:24:36.709222 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703854    2562 flags.go:64] FLAG: --register-with-taints=""
Sep 16 17:24:36.709222 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703857    2562 flags.go:64] FLAG: --registry-burst="10"
Sep 16 17:24:36.709222 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703861    2562 flags.go:64] FLAG: --registry-qps="5"
Sep 16 17:24:36.709222 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703864    2562 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 17:24:36.709222 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703867    2562 flags.go:64] FLAG: --reserved-memory=""
Sep 16 17:24:36.709222 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703871    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 17:24:36.709222 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703874    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 17:24:36.709222 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703877    2562 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 17:24:36.709222 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703880    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 17:24:36.709222 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703883    2562 flags.go:64] FLAG: --runonce="false"
Sep 16 17:24:36.709222 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703885    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 17:24:36.709940 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703889    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 17:24:36.709940 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703892    2562 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 17:24:36.709940 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703895    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 17:24:36.709940 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703898    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 17:24:36.709940 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703901    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 17:24:36.709940 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703904    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 17:24:36.709940 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703907    2562 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 17:24:36.709940 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703909    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 17:24:36.709940 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703912    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 17:24:36.709940 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703915    2562 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 17:24:36.709940 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703918    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 17:24:36.709940 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703921    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 17:24:36.709940 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703924    2562 flags.go:64] FLAG: --system-cgroups=""
Sep 16 17:24:36.709940 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703927    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 17:24:36.709940 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703933    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 17:24:36.709940 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703936    2562 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 17:24:36.709940 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703939    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 17:24:36.709940 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703976    2562 flags.go:64] FLAG: --tls-min-version=""
Sep 16 17:24:36.709940 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703980    2562 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 17:24:36.709940 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703983    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 17:24:36.709940 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703986    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 17:24:36.709940 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703990    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 17:24:36.709940 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703993    2562 flags.go:64] FLAG: --v="2"
Sep 16 17:24:36.709940 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.703997    2562 flags.go:64] FLAG: --version="false"
Sep 16 17:24:36.709940 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.704003    2562 flags.go:64] FLAG: --vmodule=""
Sep 16 17:24:36.710564 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.704007    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 17:24:36.710564 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.704010    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 17:24:36.710564 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704135    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:36.710564 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704140    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:36.710564 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704143    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:36.710564 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704146    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:36.710564 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704149    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:36.710564 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704152    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:36.710564 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704155    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:36.710564 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704157    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:36.710564 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704160    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:36.710564 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704163    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:36.710564 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704165    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:36.710564 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704168    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:36.710564 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704170    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:36.710564 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704173    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:36.710564 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704175    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:36.710564 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704178    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:36.710564 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704181    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:36.711054 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704184    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:36.711054 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704188    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:36.711054 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704191    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:36.711054 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704194    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:36.711054 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704197    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:36.711054 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704203    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:36.711054 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704206    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:36.711054 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704209    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:36.711054 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704212    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:36.711054 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704215    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:36.711054 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704217    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:36.711054 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704221    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:36.711054 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704225    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:36.711054 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704228    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:36.711054 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704231    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:36.711054 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704234    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:36.711054 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704237    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:36.711054 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704240    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:36.711054 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704242    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:36.711054 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704245    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:36.711542 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704248    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:36.711542 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704252    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:36.711542 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704256    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:36.711542 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704258    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:36.711542 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704261    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:36.711542 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704265    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:36.711542 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704268    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:36.711542 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704271    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:36.711542 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704273    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:36.711542 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704277    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:36.711542 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704279    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:36.711542 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704283    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:36.711542 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704287    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:36.711542 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704290    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:36.711542 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704295    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:36.711542 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704298    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:36.711542 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704301    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:36.711542 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704305    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:36.711542 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704308    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:36.712041 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704310    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:36.712041 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704313    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:36.712041 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704316    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:36.712041 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704318    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:36.712041 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704321    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:36.712041 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704324    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:36.712041 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704327    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:36.712041 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704330    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:36.712041 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704333    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:36.712041 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704336    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:36.712041 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704338    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:36.712041 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704342    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:36.712041 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704344    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:36.712041 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704347    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:36.712041 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704349    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:36.712041 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704352    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:36.712041 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704354    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:36.712041 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704357    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:36.712041 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704360    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:36.712517 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704364    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:36.712517 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704367    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:36.712517 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704370    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:36.712517 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704372    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:36.712517 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704375    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:36.712517 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704378    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:36.712517 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704381    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:36.712517 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704383    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:36.712517 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704386    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:36.712517 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704389    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:36.712517 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704391    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:36.712517 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704395    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:36.712517 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704397    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:36.712517 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704400    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:36.712517 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704402    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:36.712517 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704405    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:36.712517 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704407    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:36.712517 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704410    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:36.712517 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704414    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:36.712517 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704416    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:36.713058 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704419    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:36.713058 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704422    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:36.713058 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704424    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:36.713058 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.704427    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:36.713058 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.704924    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 17:24:36.713058 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.711907    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 17:24:36.713058 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.711922    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 17:24:36.713058 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.711936    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 17:24:36.713058 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.711941    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 17:24:36.713058 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712003    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:36.713058 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712008    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:36.713058 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712011    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:36.713058 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712015    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:36.713058 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712018    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:36.713058 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712021    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:36.713432 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712024    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:36.713432 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712026    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:36.713432 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712029    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:36.713432 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712032    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:36.713432 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712035    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:36.713432 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712038    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:36.713432 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712040    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:36.713432 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712043    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:36.713432 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712047    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:36.713432 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712051    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:36.713432 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712055    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:36.713432 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712058    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:36.713432 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712061    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:36.713432 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712064    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:36.713432 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712067    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:36.713432 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712070    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:36.713432 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712073    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:36.713432 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712076    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:36.713432 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712079    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:36.713904 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712082    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:36.713904 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712085    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:36.713904 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712088    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:36.713904 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712091    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:36.713904 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712094    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:36.713904 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712097    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:36.713904 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712100    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:36.713904 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712103    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:36.713904 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712105    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:36.713904 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712108    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:36.713904 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712111    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:36.713904 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712114    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:36.713904 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712117    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:36.713904 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712120    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:36.713904 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712123    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:36.713904 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712125    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:36.713904 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712128    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:36.713904 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712131    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:36.713904 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712133    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:36.714372 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712136    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:36.714372 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712139    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:36.714372 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712141    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:36.714372 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712144    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:36.714372 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712147    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:36.714372 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712149    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:36.714372 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712152    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:36.714372 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712154    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:36.714372 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712159    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:36.714372 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712162    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:36.714372 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712165    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:36.714372 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712168    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:36.714372 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712171    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:36.714372 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712175    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:36.714372 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712178    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:36.714372 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712181    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:36.714372 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712185    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:36.714372 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712188    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:36.714372 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712191    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:36.714854 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712194    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:36.714854 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712196    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:36.714854 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712199    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:36.714854 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712202    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:36.714854 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712204    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:36.714854 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712207    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:36.714854 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712210    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:36.714854 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712212    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:36.714854 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712215    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:36.714854 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712217    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:36.714854 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712220    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:36.714854 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712222    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:36.714854 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712225    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:36.714854 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712228    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:36.714854 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712230    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:36.714854 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712233    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:36.714854 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712235    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:36.714854 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712238    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:36.714854 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712241    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:36.714854 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712243    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:36.715375 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712246    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:36.715375 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712248    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:36.715375 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712251    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:36.715375 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712254    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:36.715375 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712256    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:36.715375 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712259    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:36.715375 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712262    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:36.715375 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712265    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:36.715375 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712268    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:36.715375 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712270    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:36.715375 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712273    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:36.715375 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712276    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:36.715375 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712279    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:36.715375 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712281    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:36.715375 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712284    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:36.715375 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712288    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:36.715770 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.712293    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 17:24:36.715770 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712402    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:36.715770 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712407    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:36.715770 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712410    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:36.715770 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712413    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:36.715770 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712416    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:36.715770 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712419    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:36.715770 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712421    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:36.715770 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712426    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:36.715770 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712429    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:36.715770 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712432    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:36.715770 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712436    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:36.715770 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712440    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:36.715770 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712444    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:36.716154 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712448    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:36.716154 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712451    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:36.716154 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712454    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:36.716154 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712457    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:36.716154 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712460    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:36.716154 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712462    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:36.716154 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712465    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:36.716154 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712467    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:36.716154 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712470    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:36.716154 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712473    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:36.716154 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712476    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:36.716154 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712479    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:36.716154 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712482    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:36.716154 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712485    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:36.716154 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712488    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:36.716154 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712490    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:36.716154 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712493    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:36.716154 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712495    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:36.716154 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712498    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:36.716608 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712501    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:36.716608 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712504    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:36.716608 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712507    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:36.716608 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712510    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:36.716608 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712512    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:36.716608 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712515    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:36.716608 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712518    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:36.716608 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712521    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:36.716608 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712524    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:36.716608 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712526    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:36.716608 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712529    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:36.716608 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712532    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:36.716608 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712534    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:36.716608 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712537    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:36.716608 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712540    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:36.716608 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712542    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:36.716608 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712545    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:36.716608 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712548    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:36.716608 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712550    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:36.716608 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712553    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:36.717098 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712555    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:36.717098 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712558    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:36.717098 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712561    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:36.717098 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712563    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:36.717098 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712572    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:36.717098 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712575    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:36.717098 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712577    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:36.717098 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712580    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:36.717098 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712583    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:36.717098 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712585    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:36.717098 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712588    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:36.717098 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712590    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:36.717098 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712593    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:36.717098 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712596    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:36.717098 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712598    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:36.717098 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712601    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:36.717098 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712603    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:36.717098 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712606    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:36.717098 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712609    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:36.717098 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712611    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:36.717588 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712614    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:36.717588 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712616    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:36.717588 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712619    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:36.717588 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712622    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:36.717588 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712625    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:36.717588 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712627    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:36.717588 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712630    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:36.717588 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712632    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:36.717588 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712635    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:36.717588 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712638    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:36.717588 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712641    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:36.717588 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712643    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:36.717588 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712646    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:36.717588 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712649    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:36.717588 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712652    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:36.717588 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712654    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:36.717588 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712657    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:36.717588 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712660    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:36.717588 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712663    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:36.717588 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712666    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:36.718087 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712668    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:36.718087 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712671    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:36.718087 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712673    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:36.718087 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712676    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:36.718087 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712679    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:36.718087 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712681    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:36.718087 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:36.712684    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:36.718087 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.712689    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 17:24:36.718087 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.712817    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 17:24:36.718087 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.714787    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 17:24:36.718087 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.715892    2562 server.go:1019] "Starting client certificate rotation"
Sep 16 17:24:36.718087 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.715988    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 17:24:36.718087 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.716578    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 17:24:36.718513 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.718499    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 17:24:36.719616 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.719596    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 17:24:36.730013 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.729995    2562 log.go:25] "Validated CRI v1 runtime API"
Sep 16 17:24:36.732692 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.732678    2562 log.go:25] "Validated CRI v1 image API"
Sep 16 17:24:36.734299 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.734283    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 17:24:36.737307 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.737285    2562 fs.go:135] Filesystem UUIDs: map[4514b457-19d0-44d9-99f5-9b296c756499:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b4b95144-2a2c-4cd3-9bff-a17356f0664d:/dev/nvme0n1p3]
Sep 16 17:24:36.737376 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.737305    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 17:24:36.743068 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.742963    2562 manager.go:216] Machine: {Timestamp:2026-09-16 17:24:36.741348969 +0000 UTC m=+0.329738948 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102055 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ff8cb415bedbe8347d3b07b1572ce SystemUUID:ec2ff8cb-415b-edbe-8347-d3b07b1572ce BootID:282ca857-efc6-4e05-a1eb-fa5cc4fea074 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c4:36:96:cb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c4:36:96:cb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:5f:f3:ac:8f:e5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 17:24:36.743068 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.743066    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 17:24:36.743185 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.743147    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 17:24:36.745993 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.745972    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 17:24:36.746126 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.745997    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-160.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 17:24:36.746726 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.746718    2562 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 17:24:36.746756 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.746729    2562 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 17:24:36.746756 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.746737    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 17:24:36.747323 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.747313    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 17:24:36.748060 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.748050    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 17:24:36.752268 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.752256    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 17:24:36.752379 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.752370    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 17:24:36.752411 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.752405    2562 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 17:24:36.752439 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.752415    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 17:24:36.752439 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.752428    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 17:24:36.752439 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.752437    2562 kubelet.go:398] "Adding apiserver pod source"
Sep 16 17:24:36.752530 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.752446    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 17:24:36.752767 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.752757    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 17:24:36.752798 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.752775    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 17:24:36.754887 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.754868    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 17:24:36.756437 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.756418    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 17:24:36.756755 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.756743    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 17:24:36.756826 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.756773    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 17:24:36.758229 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.758211    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 17:24:36.758229 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.758229    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 17:24:36.758348 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.758235    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 17:24:36.758348 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.758243    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 17:24:36.758348 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.758255    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 17:24:36.758348 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.758263    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 17:24:36.758348 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.758277    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 17:24:36.758348 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.758282    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 17:24:36.758348 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.758288    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 17:24:36.758348 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.758294    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 17:24:36.758348 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.758311    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 17:24:36.758348 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.758316    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 17:24:36.759023 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.759014    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 17:24:36.759056 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.759024    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 17:24:36.762600 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.762587    2562 server.go:1268] "Started kubelet"
Sep 16 17:24:36.762697 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.762674    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 17:24:36.762788 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.762694    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 17:24:36.762855 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.762800    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 17:24:36.763474 ip-10-0-129-160 systemd[1]: Started Kubernetes Kubelet.
Sep 16 17:24:36.765638 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.763760    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 17:24:36.765638 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.764982    2562 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 17:24:36.767756 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:36.767736    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 17:24:36.769839 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.769817    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 17:24:36.770026 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.769991    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-160.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 17:24:36.770099 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:36.770018    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 17:24:36.770099 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:36.770058    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-160.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 17:24:36.770568 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.770519    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 17:24:36.770850 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:36.770829    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-160.ec2.internal\" not found"
Sep 16 17:24:36.771137 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.771114    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 17:24:36.771137 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.771125    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 17:24:36.771270 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.771142    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 17:24:36.771319 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.771296    2562 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 17:24:36.771319 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.771307    2562 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 17:24:36.771991 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.771977    2562 factory.go:153] Registering CRI-O factory
Sep 16 17:24:36.772057 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.771997    2562 factory.go:223] Registration of the crio container factory successfully
Sep 16 17:24:36.772057 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.772045    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 17:24:36.772169 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.772060    2562 factory.go:55] Registering systemd factory
Sep 16 17:24:36.772169 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.772068    2562 factory.go:223] Registration of the systemd container factory successfully
Sep 16 17:24:36.772169 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.772085    2562 factory.go:103] Registering Raw factory
Sep 16 17:24:36.772169 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.772099    2562 manager.go:1196] Started watching for new ooms in manager
Sep 16 17:24:36.772700 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:36.770489    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-160.ec2.internal.18d5dd0d71c311a0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-160.ec2.internal,UID:ip-10-0-129-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-160.ec2.internal,},FirstTimestamp:2026-09-16 17:24:36.762562976 +0000 UTC m=+0.350952951,LastTimestamp:2026-09-16 17:24:36.762562976 +0000 UTC m=+0.350952951,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-160.ec2.internal,}"
Sep 16 17:24:36.772838 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.772818    2562 manager.go:349] Starting recovery of all containers
Sep 16 17:24:36.779426 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:36.779402    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 17:24:36.779535 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:36.779506    2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-160.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 17:24:36.782729 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.782712    2562 manager.go:354] Recovery completed
Sep 16 17:24:36.787135 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.787026    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:36.789385 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.789367    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-160.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:36.789454 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.789402    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-160.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:36.789454 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.789418    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-160.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:36.790055 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.790039    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 17:24:36.790055 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.790049    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:36.790191 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.790052    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 17:24:36.790191 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.790129    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 17:24:36.792435 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.792319    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-160.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:36.792435 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.792345    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-160.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:36.792435 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.792358    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-160.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:36.792755 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:36.792677    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-160.ec2.internal.18d5dd0d735c55a2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-160.ec2.internal,UID:ip-10-0-129-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-160.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-160.ec2.internal,},FirstTimestamp:2026-09-16 17:24:36.78938461 +0000 UTC m=+0.377774589,LastTimestamp:2026-09-16 17:24:36.78938461 +0000 UTC m=+0.377774589,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-160.ec2.internal,}"
Sep 16 17:24:36.794120 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.794104    2562 policy_none.go:49] "None policy: Start"
Sep 16 17:24:36.794201 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.794124    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 17:24:36.794201 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.794138    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 17:24:36.795885 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.795873    2562 policy_none.go:47] "Start"
Sep 16 17:24:36.805467 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:36.805402    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-160.ec2.internal.18d5dd0d735cb28d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-160.ec2.internal,UID:ip-10-0-129-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-160.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-160.ec2.internal,},FirstTimestamp:2026-09-16 17:24:36.789408397 +0000 UTC m=+0.377798372,LastTimestamp:2026-09-16 17:24:36.789408397 +0000 UTC m=+0.377798372,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-160.ec2.internal,}"
Sep 16 17:24:36.819989 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:36.819912    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-160.ec2.internal.18d5dd0d735ceef3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-160.ec2.internal,UID:ip-10-0-129-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-160.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-160.ec2.internal,},FirstTimestamp:2026-09-16 17:24:36.789423859 +0000 UTC m=+0.377813839,LastTimestamp:2026-09-16 17:24:36.789423859 +0000 UTC m=+0.377813839,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-160.ec2.internal,}"
Sep 16 17:24:36.823396 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.823381    2562 manager.go:337] "Starting Device Plugin manager"
Sep 16 17:24:36.823493 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:36.823421    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 17:24:36.823493 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.823435    2562 server.go:87] "Starting device plugin registration server"
Sep 16 17:24:36.823817 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.823786    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 17:24:36.823874 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.823814    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 17:24:36.823999 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.823982    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 17:24:36.824086 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.824070    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 17:24:36.824086 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.824084    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 17:24:36.824710 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:36.824674    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 17:24:36.824787 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:36.824750    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-160.ec2.internal\" not found"
Sep 16 17:24:36.837090 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.837067    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h8jvh"
Sep 16 17:24:36.837893 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:36.837835    2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-160.ec2.internal.18d5dd0d735c55a2\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-160.ec2.internal.18d5dd0d735c55a2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-160.ec2.internal,UID:ip-10-0-129-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-160.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-160.ec2.internal,},FirstTimestamp:2026-09-16 17:24:36.78938461 +0000 UTC m=+0.377774589,LastTimestamp:2026-09-16 17:24:36.792334676 +0000 UTC m=+0.380724651,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-160.ec2.internal,}"
Sep 16 17:24:36.838236 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.838219    2562 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2589 into the system container \"/system.slice\": <nil>"
Sep 16 17:24:36.846797 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.846781    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h8jvh"
Sep 16 17:24:36.852525 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:36.852457    2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-160.ec2.internal.18d5dd0d735cb28d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-160.ec2.internal.18d5dd0d735cb28d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-160.ec2.internal,UID:ip-10-0-129-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-160.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-160.ec2.internal,},FirstTimestamp:2026-09-16 17:24:36.789408397 +0000 UTC m=+0.377798372,LastTimestamp:2026-09-16 17:24:36.792350454 +0000 UTC m=+0.380740430,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-160.ec2.internal,}"
Sep 16 17:24:36.862134 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:36.862080    2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-160.ec2.internal.18d5dd0d735ceef3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-160.ec2.internal.18d5dd0d735ceef3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-160.ec2.internal,UID:ip-10-0-129-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-160.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-160.ec2.internal,},FirstTimestamp:2026-09-16 17:24:36.789423859 +0000 UTC m=+0.377813839,LastTimestamp:2026-09-16 17:24:36.792433426 +0000 UTC m=+0.380823403,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-160.ec2.internal,}"
Sep 16 17:24:36.877195 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:36.877133    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-160.ec2.internal.18d5dd0d75dab16c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-160.ec2.internal,UID:ip-10-0-129-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-129-160.ec2.internal,},FirstTimestamp:2026-09-16 17:24:36.831220076 +0000 UTC m=+0.419610043,LastTimestamp:2026-09-16 17:24:36.831220076 +0000 UTC m=+0.419610043,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-160.ec2.internal,}"
Sep 16 17:24:36.924299 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.924269    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:36.926279 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.926260    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-160.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:36.926379 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.926293    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-160.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:36.926379 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.926304    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-160.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:36.926379 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.926341    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-160.ec2.internal"
Sep 16 17:24:36.935449 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:36.935376    2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-160.ec2.internal.18d5dd0d735c55a2\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-160.ec2.internal.18d5dd0d735c55a2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-160.ec2.internal,UID:ip-10-0-129-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-160.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-160.ec2.internal,},FirstTimestamp:2026-09-16 17:24:36.78938461 +0000 UTC m=+0.377774589,LastTimestamp:2026-09-16 17:24:36.926277936 +0000 UTC m=+0.514667911,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-160.ec2.internal,}"
Sep 16 17:24:36.949225 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:36.949203    2562 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-129-160.ec2.internal\" not found" node="ip-10-0-129-160.ec2.internal"
Sep 16 17:24:36.987237 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:36.987213    2562 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-160.ec2.internal\" not found" node="ip-10-0-129-160.ec2.internal"
Sep 16 17:24:36.993946 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.993920    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 17:24:36.995053 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.995025    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 17:24:36.995136 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.995086    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 17:24:36.995136 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.995112    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 17:24:36.995250 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:36.995219    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 17:24:36.998042 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:36.998023    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 17:24:37.095929 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.095890    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-160.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-160.ec2.internal"]
Sep 16 17:24:37.096092 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.095981    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:37.097094 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.097077    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-160.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:37.097154 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.097109    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-160.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:37.097154 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.097131    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-160.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:37.099451 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.099437    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:37.099565 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.099552    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-160.ec2.internal"
Sep 16 17:24:37.099605 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.099579    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:37.100421 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.100400    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-160.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:37.100533 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.100431    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-160.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:37.100533 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.100444    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-160.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:37.100533 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.100448    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-160.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:37.100533 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.100470    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-160.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:37.100533 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.100480    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-160.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:37.102725 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.102703    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-160.ec2.internal"
Sep 16 17:24:37.102833 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.102729    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:37.103520 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.103504    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-160.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:37.103609 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.103533    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-160.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:37.103609 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.103546    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-160.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:37.125437 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:37.125418    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-160.ec2.internal\" not found" node="ip-10-0-129-160.ec2.internal"
Sep 16 17:24:37.128764 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:37.128749    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-160.ec2.internal\" not found" node="ip-10-0-129-160.ec2.internal"
Sep 16 17:24:37.149623 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.149607    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:37.150542 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.150525    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-160.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:37.150630 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.150552    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-160.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:37.150630 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.150561    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-160.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:37.150630 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.150584    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-160.ec2.internal"
Sep 16 17:24:37.163073 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.163057    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-160.ec2.internal"
Sep 16 17:24:37.163132 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:37.163076    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-160.ec2.internal\": node \"ip-10-0-129-160.ec2.internal\" not found"
Sep 16 17:24:37.173640 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.173619    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0df2195e00f1241c11a998a6a4518995-config\") pod \"kube-apiserver-proxy-ip-10-0-129-160.ec2.internal\" (UID: \"0df2195e00f1241c11a998a6a4518995\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-160.ec2.internal"
Sep 16 17:24:37.173719 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.173659    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/82ef5bcb9fdb0c81f8cc0a52c6999929-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-160.ec2.internal\" (UID: \"82ef5bcb9fdb0c81f8cc0a52c6999929\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-160.ec2.internal"
Sep 16 17:24:37.173719 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.173690    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82ef5bcb9fdb0c81f8cc0a52c6999929-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-160.ec2.internal\" (UID: \"82ef5bcb9fdb0c81f8cc0a52c6999929\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-160.ec2.internal"
Sep 16 17:24:37.211536 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:37.211480    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-160.ec2.internal\" not found"
Sep 16 17:24:37.274214 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.274187    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/82ef5bcb9fdb0c81f8cc0a52c6999929-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-160.ec2.internal\" (UID: \"82ef5bcb9fdb0c81f8cc0a52c6999929\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-160.ec2.internal"
Sep 16 17:24:37.274316 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.274216    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82ef5bcb9fdb0c81f8cc0a52c6999929-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-160.ec2.internal\" (UID: \"82ef5bcb9fdb0c81f8cc0a52c6999929\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-160.ec2.internal"
Sep 16 17:24:37.274316 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.274240    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0df2195e00f1241c11a998a6a4518995-config\") pod \"kube-apiserver-proxy-ip-10-0-129-160.ec2.internal\" (UID: \"0df2195e00f1241c11a998a6a4518995\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-160.ec2.internal"
Sep 16 17:24:37.274316 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.274293    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82ef5bcb9fdb0c81f8cc0a52c6999929-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-160.ec2.internal\" (UID: \"82ef5bcb9fdb0c81f8cc0a52c6999929\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-160.ec2.internal"
Sep 16 17:24:37.274316 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.274297    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/82ef5bcb9fdb0c81f8cc0a52c6999929-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-160.ec2.internal\" (UID: \"82ef5bcb9fdb0c81f8cc0a52c6999929\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-160.ec2.internal"
Sep 16 17:24:37.274448 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.274330    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0df2195e00f1241c11a998a6a4518995-config\") pod \"kube-apiserver-proxy-ip-10-0-129-160.ec2.internal\" (UID: \"0df2195e00f1241c11a998a6a4518995\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-160.ec2.internal"
Sep 16 17:24:37.311921 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:37.311891    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-160.ec2.internal\" not found"
Sep 16 17:24:37.412867 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:37.412836    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-160.ec2.internal\" not found"
Sep 16 17:24:37.428001 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.427982    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-160.ec2.internal"
Sep 16 17:24:37.431084 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.431072    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-160.ec2.internal"
Sep 16 17:24:37.515492 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:37.515442    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-160.ec2.internal\" not found"
Sep 16 17:24:37.618477 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:37.618449    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-160.ec2.internal\" not found"
Sep 16 17:24:37.717774 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.717744    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 17:24:37.718290 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.717942    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 17:24:37.718939 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:37.718917    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-160.ec2.internal\" not found"
Sep 16 17:24:37.819792 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:37.819754    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-160.ec2.internal\" not found"
Sep 16 17:24:37.832977 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.832955    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 17:24:37.870888 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.870858    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-160.ec2.internal"
Sep 16 17:24:37.872730 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.872705    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 17:24:37.939046 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.939004    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 17:19:37 +0000 UTC" deadline="2028-05-29 07:22:23.170291068 +0000 UTC"
Sep 16 17:24:37.939046 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.939039    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14893h57m45.231256395s"
Sep 16 17:24:37.955095 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.955072    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 17:24:37.957040 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.957025    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-160.ec2.internal"
Sep 16 17:24:37.976044 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:37.975971    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 17:24:38.144069 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.144041    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 17:24:38.161014 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:38.160981    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df2195e00f1241c11a998a6a4518995.slice/crio-8eb128d47875f28cb57068b74bf2881d6c811c941db9344654a2f2f21b82d3e5 WatchSource:0}: Error finding container 8eb128d47875f28cb57068b74bf2881d6c811c941db9344654a2f2f21b82d3e5: Status 404 returned error can't find the container with id 8eb128d47875f28cb57068b74bf2881d6c811c941db9344654a2f2f21b82d3e5
Sep 16 17:24:38.161252 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:38.161234    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ef5bcb9fdb0c81f8cc0a52c6999929.slice/crio-854bfed690430a4d16e0a22bc627e9b5af4a3cefec1ed95a056769d86d3c4250 WatchSource:0}: Error finding container 854bfed690430a4d16e0a22bc627e9b5af4a3cefec1ed95a056769d86d3c4250: Status 404 returned error can't find the container with id 854bfed690430a4d16e0a22bc627e9b5af4a3cefec1ed95a056769d86d3c4250
Sep 16 17:24:38.166190 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.166169    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:24:38.754275 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.754250    2562 apiserver.go:52] "Watching apiserver"
Sep 16 17:24:38.769451 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.769421    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 17:24:38.770743 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.770711    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-160.ec2.internal","openshift-cluster-node-tuning-operator/tuned-jdd4s","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-160.ec2.internal","openshift-multus/multus-additional-cni-plugins-tkzcs","openshift-multus/multus-lpd4g","openshift-ovn-kubernetes/ovnkube-node-t6p26","kube-system/global-pull-secret-syncer-scrxg","kube-system/konnectivity-agent-r67pk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj","openshift-dns/node-resolver-2r4vj","openshift-image-registry/node-ca-ddqwd","openshift-multus/network-metrics-daemon-fw78m","openshift-network-diagnostics/network-check-target-89r27","openshift-network-operator/iptables-alerter-hvbv7"]
Sep 16 17:24:38.770893 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.770725    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 17:24:38.773650 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.773626    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:24:38.773755 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:38.773730    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scrxg" podUID="4b47007d-992e-4660-900c-b4807ce86b3f"
Sep 16 17:24:38.775738 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.775715    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:24:38.775840 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:38.775792    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89r27" podUID="c88f1596-e3ef-45ce-b8f6-55fc246bfdab"
Sep 16 17:24:38.777594 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.777577    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 17:24:38.778278 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.778256    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hvbv7"
Sep 16 17:24:38.780635 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.780612    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tkzcs"
Sep 16 17:24:38.781220 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.781199    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 17:24:38.781220 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.781211    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 17:24:38.781381 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.781321    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7gtfd\""
Sep 16 17:24:38.781752 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.781730    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 17:24:38.782740 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.782718    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.782881 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.782860    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4b47007d-992e-4660-900c-b4807ce86b3f-kubelet-config\") pod \"global-pull-secret-syncer-scrxg\" (UID: \"4b47007d-992e-4660-900c-b4807ce86b3f\") " pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:24:38.782967 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.782911    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4b47007d-992e-4660-900c-b4807ce86b3f-dbus\") pod \"global-pull-secret-syncer-scrxg\" (UID: \"4b47007d-992e-4660-900c-b4807ce86b3f\") " pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:24:38.783030 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.782982    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b47007d-992e-4660-900c-b4807ce86b3f-original-pull-secret\") pod \"global-pull-secret-syncer-scrxg\" (UID: \"4b47007d-992e-4660-900c-b4807ce86b3f\") " pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:24:38.783642 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.783236    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 17:24:38.784818 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.784799    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 17:24:38.788296 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.787876    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 17:24:38.788296 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.788109    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4tk98\""
Sep 16 17:24:38.788416 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.788300    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 17:24:38.790505 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.790085    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 17:24:38.790505 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.790232    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 17:24:38.790505 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.790328    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-82pvq\""
Sep 16 17:24:38.790704 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.790554    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.794447 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.794383    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj"
Sep 16 17:24:38.796284 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.796263    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 17:24:38.796474 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.796457    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 17:24:38.796723 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.796707    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r67pk"
Sep 16 17:24:38.797817 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.797798    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mgz9t\""
Sep 16 17:24:38.797932 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.797832    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 17:24:38.798085 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.798053    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 17:24:38.798085 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.798064    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 17:24:38.798224 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.798140    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 17:24:38.798312 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.798299    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 17:24:38.800619 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.800351    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.802120 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.801173    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 17:24:38.802120 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.801411    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 17:24:38.802120 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.801660    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 17:24:38.802120 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.801914    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 17:24:38.802373 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.802197    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-kbrdz\""
Sep 16 17:24:38.802373 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.802318    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x99mg\""
Sep 16 17:24:38.803213 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.803044    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 17:24:38.804042 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.804015    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2r4vj"
Sep 16 17:24:38.804674 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.804656    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 17:24:38.804765 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.804710    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mpxnr\""
Sep 16 17:24:38.811136 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.809132    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ddqwd"
Sep 16 17:24:38.811136 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.809209    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 17:24:38.811136 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.809617    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 17:24:38.811136 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.809693    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:24:38.811136 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:38.809774    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fw78m" podUID="9d27ec00-06e7-431b-925f-addd45ff0115"
Sep 16 17:24:38.811937 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.811657    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pbxrr\""
Sep 16 17:24:38.813300 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.812612    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 17:24:38.813300 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.813271    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 17:24:38.813451 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.813314    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 17:24:38.813451 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.813384    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pvqfb\""
Sep 16 17:24:38.815132 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.815107    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 17:24:38.859402 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.859374    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-r9qr4"
Sep 16 17:24:38.867197 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.867172    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r9qr4"
Sep 16 17:24:38.872418 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.872397    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 17:24:38.883363 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.883337    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-host-run-k8s-cni-cncf-io\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.883482 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.883376    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-etc-sysconfig\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.883482 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.883403    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f689aa90-24ff-48a5-8e66-f2e5d749dbd4-host\") pod \"node-ca-ddqwd\" (UID: \"f689aa90-24ff-48a5-8e66-f2e5d749dbd4\") " pod="openshift-image-registry/node-ca-ddqwd"
Sep 16 17:24:38.883482 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.883429    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05821b99-6706-481a-a80c-0a86275f520a-host-slash\") pod \"iptables-alerter-hvbv7\" (UID: \"05821b99-6706-481a-a80c-0a86275f520a\") " pod="openshift-network-operator/iptables-alerter-hvbv7"
Sep 16 17:24:38.883482 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.883456    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-host-var-lib-cni-bin\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.883700 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.883481    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d970e5c-29fc-44af-b138-0e60309f2ac0-ovn-node-metrics-cert\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.883700 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.883508    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af87e079-c946-4c98-86fc-4be5a3098d8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tkzcs\" (UID: \"af87e079-c946-4c98-86fc-4be5a3098d8a\") " pod="openshift-multus/multus-additional-cni-plugins-tkzcs"
Sep 16 17:24:38.883700 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.883551    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-system-cni-dir\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.883700 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.883586    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-run\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.883700 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.883624    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-lib-modules\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.883700 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.883651    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-var-lib-kubelet\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.883700 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.883685    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sr5g\" (UniqueName: \"kubernetes.io/projected/c88f1596-e3ef-45ce-b8f6-55fc246bfdab-kube-api-access-7sr5g\") pod \"network-check-target-89r27\" (UID: \"c88f1596-e3ef-45ce-b8f6-55fc246bfdab\") " pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:24:38.884079 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.883723    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79c16240-59ee-4b5e-98c9-86ef04179efa-tmp-dir\") pod \"node-resolver-2r4vj\" (UID: \"79c16240-59ee-4b5e-98c9-86ef04179efa\") " pod="openshift-dns/node-resolver-2r4vj"
Sep 16 17:24:38.884079 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.883751    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1d970e5c-29fc-44af-b138-0e60309f2ac0-ovnkube-script-lib\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.884079 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.883791    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-etc-sysctl-d\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.884079 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.883835    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5189ad61-97f5-4970-b0f5-acc0606b13f8-sys-fs\") pod \"aws-ebs-csi-driver-node-489qj\" (UID: \"5189ad61-97f5-4970-b0f5-acc0606b13f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj"
Sep 16 17:24:38.884079 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.883863    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-host-run-netns\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.884079 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.883899    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1d970e5c-29fc-44af-b138-0e60309f2ac0-ovnkube-config\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.884079 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.883943    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-hostroot\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.884079 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.883973    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-var-lib-openvswitch\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.884079 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.883999    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-etc-openvswitch\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.884079 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.884029    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-sys\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.884079 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.884054    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj9m9\" (UniqueName: \"kubernetes.io/projected/f689aa90-24ff-48a5-8e66-f2e5d749dbd4-kube-api-access-wj9m9\") pod \"node-ca-ddqwd\" (UID: \"f689aa90-24ff-48a5-8e66-f2e5d749dbd4\") " pod="openshift-image-registry/node-ca-ddqwd"
Sep 16 17:24:38.884079 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.884078    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs\") pod \"network-metrics-daemon-fw78m\" (UID: \"9d27ec00-06e7-431b-925f-addd45ff0115\") " pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:24:38.884662 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.884150    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4b47007d-992e-4660-900c-b4807ce86b3f-dbus\") pod \"global-pull-secret-syncer-scrxg\" (UID: \"4b47007d-992e-4660-900c-b4807ce86b3f\") " pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:24:38.884662 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.884181    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-multus-cni-dir\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.884662 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.884258    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4b47007d-992e-4660-900c-b4807ce86b3f-dbus\") pod \"global-pull-secret-syncer-scrxg\" (UID: \"4b47007d-992e-4660-900c-b4807ce86b3f\") " pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:24:38.884662 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.884258    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1d970e5c-29fc-44af-b138-0e60309f2ac0-env-overrides\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.884662 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.884315    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bd7c\" (UniqueName: \"kubernetes.io/projected/9d27ec00-06e7-431b-925f-addd45ff0115-kube-api-access-7bd7c\") pod \"network-metrics-daemon-fw78m\" (UID: \"9d27ec00-06e7-431b-925f-addd45ff0115\") " pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:24:38.884662 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.884347    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5189ad61-97f5-4970-b0f5-acc0606b13f8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-489qj\" (UID: \"5189ad61-97f5-4970-b0f5-acc0606b13f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj"
Sep 16 17:24:38.884662 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.884373    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5189ad61-97f5-4970-b0f5-acc0606b13f8-etc-selinux\") pod \"aws-ebs-csi-driver-node-489qj\" (UID: \"5189ad61-97f5-4970-b0f5-acc0606b13f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj"
Sep 16 17:24:38.884662 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.884397    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/15ea9542-9e0b-4000-8e55-0a4b734105ae-konnectivity-ca\") pod \"konnectivity-agent-r67pk\" (UID: \"15ea9542-9e0b-4000-8e55-0a4b734105ae\") " pod="kube-system/konnectivity-agent-r67pk"
Sep 16 17:24:38.884662 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.884421    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a83c017c-9dbf-4051-a74c-e5fa85eda19f-cni-binary-copy\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.884662 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.884444    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/05821b99-6706-481a-a80c-0a86275f520a-iptables-alerter-script\") pod \"iptables-alerter-hvbv7\" (UID: \"05821b99-6706-481a-a80c-0a86275f520a\") " pod="openshift-network-operator/iptables-alerter-hvbv7"
Sep 16 17:24:38.884662 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.884468    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-systemd-units\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.884662 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.884491    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-run-systemd\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.884662 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.884522    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-etc-modprobe-d\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.884662 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.884545    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f689aa90-24ff-48a5-8e66-f2e5d749dbd4-serviceca\") pod \"node-ca-ddqwd\" (UID: \"f689aa90-24ff-48a5-8e66-f2e5d749dbd4\") " pod="openshift-image-registry/node-ca-ddqwd"
Sep 16 17:24:38.884662 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.884570    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cp86\" (UniqueName: \"kubernetes.io/projected/05821b99-6706-481a-a80c-0a86275f520a-kube-api-access-4cp86\") pod \"iptables-alerter-hvbv7\" (UID: \"05821b99-6706-481a-a80c-0a86275f520a\") " pod="openshift-network-operator/iptables-alerter-hvbv7"
Sep 16 17:24:38.884662 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.884617    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af87e079-c946-4c98-86fc-4be5a3098d8a-cnibin\") pod \"multus-additional-cni-plugins-tkzcs\" (UID: \"af87e079-c946-4c98-86fc-4be5a3098d8a\") " pod="openshift-multus/multus-additional-cni-plugins-tkzcs"
Sep 16 17:24:38.885451 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.884643    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp5hx\" (UniqueName: \"kubernetes.io/projected/af87e079-c946-4c98-86fc-4be5a3098d8a-kube-api-access-sp5hx\") pod \"multus-additional-cni-plugins-tkzcs\" (UID: \"af87e079-c946-4c98-86fc-4be5a3098d8a\") " pod="openshift-multus/multus-additional-cni-plugins-tkzcs"
Sep 16 17:24:38.885451 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.884669    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-host-var-lib-cni-multus\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.885451 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.884695    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-host-slash\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.885451 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.884720    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-host-run-netns\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.885451 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.884744    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-host-cni-bin\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.885451 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.884789    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b47007d-992e-4660-900c-b4807ce86b3f-original-pull-secret\") pod \"global-pull-secret-syncer-scrxg\" (UID: \"4b47007d-992e-4660-900c-b4807ce86b3f\") " pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:24:38.885451 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.884891    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-cnibin\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.885451 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:38.884910    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:38.885451 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.884949    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.885451 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885007    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-etc-kubernetes\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.885451 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:38.885026    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b47007d-992e-4660-900c-b4807ce86b3f-original-pull-secret podName:4b47007d-992e-4660-900c-b4807ce86b3f nodeName:}" failed. No retries permitted until 2026-09-16 17:24:39.385006073 +0000 UTC m=+2.973396059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b47007d-992e-4660-900c-b4807ce86b3f-original-pull-secret") pod "global-pull-secret-syncer-scrxg" (UID: "4b47007d-992e-4660-900c-b4807ce86b3f") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:38.885451 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885060    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fa114745-379e-413e-8a75-75c24ffa9ee7-etc-tuned\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.885451 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885086    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/af87e079-c946-4c98-86fc-4be5a3098d8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tkzcs\" (UID: \"af87e079-c946-4c98-86fc-4be5a3098d8a\") " pod="openshift-multus/multus-additional-cni-plugins-tkzcs"
Sep 16 17:24:38.885451 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885112    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdgfq\" (UniqueName: \"kubernetes.io/projected/79c16240-59ee-4b5e-98c9-86ef04179efa-kube-api-access-qdgfq\") pod \"node-resolver-2r4vj\" (UID: \"79c16240-59ee-4b5e-98c9-86ef04179efa\") " pod="openshift-dns/node-resolver-2r4vj"
Sep 16 17:24:38.885451 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885137    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa114745-379e-413e-8a75-75c24ffa9ee7-tmp\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.885451 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885160    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d55mj\" (UniqueName: \"kubernetes.io/projected/fa114745-379e-413e-8a75-75c24ffa9ee7-kube-api-access-d55mj\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.886356 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885184    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-log-socket\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.886356 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885219    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-host-run-ovn-kubernetes\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.886356 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885248    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj6th\" (UniqueName: \"kubernetes.io/projected/1d970e5c-29fc-44af-b138-0e60309f2ac0-kube-api-access-xj6th\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.886356 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885279    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-etc-sysctl-conf\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.886356 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885308    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af87e079-c946-4c98-86fc-4be5a3098d8a-system-cni-dir\") pod \"multus-additional-cni-plugins-tkzcs\" (UID: \"af87e079-c946-4c98-86fc-4be5a3098d8a\") " pod="openshift-multus/multus-additional-cni-plugins-tkzcs"
Sep 16 17:24:38.886356 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885342    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4b47007d-992e-4660-900c-b4807ce86b3f-kubelet-config\") pod \"global-pull-secret-syncer-scrxg\" (UID: \"4b47007d-992e-4660-900c-b4807ce86b3f\") " pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:24:38.886356 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885367    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-host-var-lib-kubelet\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.886356 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885391    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-etc-kubernetes\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.886356 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885454    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4b47007d-992e-4660-900c-b4807ce86b3f-kubelet-config\") pod \"global-pull-secret-syncer-scrxg\" (UID: \"4b47007d-992e-4660-900c-b4807ce86b3f\") " pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:24:38.886356 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885471    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-etc-systemd\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.886356 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885502    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af87e079-c946-4c98-86fc-4be5a3098d8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-tkzcs\" (UID: \"af87e079-c946-4c98-86fc-4be5a3098d8a\") " pod="openshift-multus/multus-additional-cni-plugins-tkzcs"
Sep 16 17:24:38.886356 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885529    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5189ad61-97f5-4970-b0f5-acc0606b13f8-device-dir\") pod \"aws-ebs-csi-driver-node-489qj\" (UID: \"5189ad61-97f5-4970-b0f5-acc0606b13f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj"
Sep 16 17:24:38.886356 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885555    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/15ea9542-9e0b-4000-8e55-0a4b734105ae-agent-certs\") pod \"konnectivity-agent-r67pk\" (UID: \"15ea9542-9e0b-4000-8e55-0a4b734105ae\") " pod="kube-system/konnectivity-agent-r67pk"
Sep 16 17:24:38.886356 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885580    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/79c16240-59ee-4b5e-98c9-86ef04179efa-hosts-file\") pod \"node-resolver-2r4vj\" (UID: \"79c16240-59ee-4b5e-98c9-86ef04179efa\") " pod="openshift-dns/node-resolver-2r4vj"
Sep 16 17:24:38.886356 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885606    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-run-openvswitch\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.886356 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885631    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-host-cni-netd\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.887173 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885656    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-host\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.887173 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885679    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5189ad61-97f5-4970-b0f5-acc0606b13f8-socket-dir\") pod \"aws-ebs-csi-driver-node-489qj\" (UID: \"5189ad61-97f5-4970-b0f5-acc0606b13f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj"
Sep 16 17:24:38.887173 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885704    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb4jn\" (UniqueName: \"kubernetes.io/projected/5189ad61-97f5-4970-b0f5-acc0606b13f8-kube-api-access-mb4jn\") pod \"aws-ebs-csi-driver-node-489qj\" (UID: \"5189ad61-97f5-4970-b0f5-acc0606b13f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj"
Sep 16 17:24:38.887173 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885735    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-multus-socket-dir-parent\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.887173 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885766    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-multus-conf-dir\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.887173 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885797    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a83c017c-9dbf-4051-a74c-e5fa85eda19f-multus-daemon-config\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.887173 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885841    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-host-kubelet\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.887173 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885877    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-run-ovn\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.887173 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885904    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-node-log\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.887173 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885931    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/af87e079-c946-4c98-86fc-4be5a3098d8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tkzcs\" (UID: \"af87e079-c946-4c98-86fc-4be5a3098d8a\") " pod="openshift-multus/multus-additional-cni-plugins-tkzcs"
Sep 16 17:24:38.887173 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885955    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-host-run-multus-certs\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.887173 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.885983    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvjzr\" (UniqueName: \"kubernetes.io/projected/a83c017c-9dbf-4051-a74c-e5fa85eda19f-kube-api-access-cvjzr\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.887173 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.886007    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5189ad61-97f5-4970-b0f5-acc0606b13f8-registration-dir\") pod \"aws-ebs-csi-driver-node-489qj\" (UID: \"5189ad61-97f5-4970-b0f5-acc0606b13f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj"
Sep 16 17:24:38.987078 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987038    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-host-run-netns\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.987270 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987087    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1d970e5c-29fc-44af-b138-0e60309f2ac0-ovnkube-config\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.987270 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987116    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-hostroot\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.987270 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987141    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-var-lib-openvswitch\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.987270 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987166    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-etc-openvswitch\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.987270 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987191    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-sys\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.987270 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987216    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj9m9\" (UniqueName: \"kubernetes.io/projected/f689aa90-24ff-48a5-8e66-f2e5d749dbd4-kube-api-access-wj9m9\") pod \"node-ca-ddqwd\" (UID: \"f689aa90-24ff-48a5-8e66-f2e5d749dbd4\") " pod="openshift-image-registry/node-ca-ddqwd"
Sep 16 17:24:38.987270 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987242    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs\") pod \"network-metrics-daemon-fw78m\" (UID: \"9d27ec00-06e7-431b-925f-addd45ff0115\") " pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:24:38.987270 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987268    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-multus-cni-dir\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.987591 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987292    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1d970e5c-29fc-44af-b138-0e60309f2ac0-env-overrides\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.987591 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987317    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bd7c\" (UniqueName: \"kubernetes.io/projected/9d27ec00-06e7-431b-925f-addd45ff0115-kube-api-access-7bd7c\") pod \"network-metrics-daemon-fw78m\" (UID: \"9d27ec00-06e7-431b-925f-addd45ff0115\") " pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:24:38.987591 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987342    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5189ad61-97f5-4970-b0f5-acc0606b13f8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-489qj\" (UID: \"5189ad61-97f5-4970-b0f5-acc0606b13f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj"
Sep 16 17:24:38.987591 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987367    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5189ad61-97f5-4970-b0f5-acc0606b13f8-etc-selinux\") pod \"aws-ebs-csi-driver-node-489qj\" (UID: \"5189ad61-97f5-4970-b0f5-acc0606b13f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj"
Sep 16 17:24:38.987591 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987393    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/15ea9542-9e0b-4000-8e55-0a4b734105ae-konnectivity-ca\") pod \"konnectivity-agent-r67pk\" (UID: \"15ea9542-9e0b-4000-8e55-0a4b734105ae\") " pod="kube-system/konnectivity-agent-r67pk"
Sep 16 17:24:38.987591 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987418    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a83c017c-9dbf-4051-a74c-e5fa85eda19f-cni-binary-copy\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.987591 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987444    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/05821b99-6706-481a-a80c-0a86275f520a-iptables-alerter-script\") pod \"iptables-alerter-hvbv7\" (UID: \"05821b99-6706-481a-a80c-0a86275f520a\") " pod="openshift-network-operator/iptables-alerter-hvbv7"
Sep 16 17:24:38.987591 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987473    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-systemd-units\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.987591 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987499    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-run-systemd\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.987591 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987535    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-etc-modprobe-d\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.987591 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987559    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f689aa90-24ff-48a5-8e66-f2e5d749dbd4-serviceca\") pod \"node-ca-ddqwd\" (UID: \"f689aa90-24ff-48a5-8e66-f2e5d749dbd4\") " pod="openshift-image-registry/node-ca-ddqwd"
Sep 16 17:24:38.987591 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987583    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cp86\" (UniqueName: \"kubernetes.io/projected/05821b99-6706-481a-a80c-0a86275f520a-kube-api-access-4cp86\") pod \"iptables-alerter-hvbv7\" (UID: \"05821b99-6706-481a-a80c-0a86275f520a\") " pod="openshift-network-operator/iptables-alerter-hvbv7"
Sep 16 17:24:38.988212 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987610    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af87e079-c946-4c98-86fc-4be5a3098d8a-cnibin\") pod \"multus-additional-cni-plugins-tkzcs\" (UID: \"af87e079-c946-4c98-86fc-4be5a3098d8a\") " pod="openshift-multus/multus-additional-cni-plugins-tkzcs"
Sep 16 17:24:38.988212 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987636    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp5hx\" (UniqueName: \"kubernetes.io/projected/af87e079-c946-4c98-86fc-4be5a3098d8a-kube-api-access-sp5hx\") pod \"multus-additional-cni-plugins-tkzcs\" (UID: \"af87e079-c946-4c98-86fc-4be5a3098d8a\") " pod="openshift-multus/multus-additional-cni-plugins-tkzcs"
Sep 16 17:24:38.988212 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987664    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-host-var-lib-cni-multus\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.988212 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987691    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-host-slash\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.988212 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987717    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-host-run-netns\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.988212 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987746    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-host-cni-bin\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.988212 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987787    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-cnibin\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.988212 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987813    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.988212 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987862    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-etc-kubernetes\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.988212 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987908    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fa114745-379e-413e-8a75-75c24ffa9ee7-etc-tuned\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.988212 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987936    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/af87e079-c946-4c98-86fc-4be5a3098d8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tkzcs\" (UID: \"af87e079-c946-4c98-86fc-4be5a3098d8a\") " pod="openshift-multus/multus-additional-cni-plugins-tkzcs"
Sep 16 17:24:38.988212 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987967    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdgfq\" (UniqueName: \"kubernetes.io/projected/79c16240-59ee-4b5e-98c9-86ef04179efa-kube-api-access-qdgfq\") pod \"node-resolver-2r4vj\" (UID: \"79c16240-59ee-4b5e-98c9-86ef04179efa\") " pod="openshift-dns/node-resolver-2r4vj"
Sep 16 17:24:38.988212 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.987996    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa114745-379e-413e-8a75-75c24ffa9ee7-tmp\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.988212 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988021    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d55mj\" (UniqueName: \"kubernetes.io/projected/fa114745-379e-413e-8a75-75c24ffa9ee7-kube-api-access-d55mj\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.988212 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988054    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-log-socket\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.988212 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988084    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-host-run-ovn-kubernetes\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.988212 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988115    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj6th\" (UniqueName: \"kubernetes.io/projected/1d970e5c-29fc-44af-b138-0e60309f2ac0-kube-api-access-xj6th\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.989097 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988143    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-etc-sysctl-conf\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.989097 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988169    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af87e079-c946-4c98-86fc-4be5a3098d8a-system-cni-dir\") pod \"multus-additional-cni-plugins-tkzcs\" (UID: \"af87e079-c946-4c98-86fc-4be5a3098d8a\") " pod="openshift-multus/multus-additional-cni-plugins-tkzcs"
Sep 16 17:24:38.989097 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988197    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-host-var-lib-kubelet\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.989097 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988223    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-etc-kubernetes\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.989097 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988248    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-etc-systemd\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.989097 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988274    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af87e079-c946-4c98-86fc-4be5a3098d8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-tkzcs\" (UID: \"af87e079-c946-4c98-86fc-4be5a3098d8a\") " pod="openshift-multus/multus-additional-cni-plugins-tkzcs"
Sep 16 17:24:38.989097 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988299    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5189ad61-97f5-4970-b0f5-acc0606b13f8-device-dir\") pod \"aws-ebs-csi-driver-node-489qj\" (UID: \"5189ad61-97f5-4970-b0f5-acc0606b13f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj"
Sep 16 17:24:38.989097 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988325    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/15ea9542-9e0b-4000-8e55-0a4b734105ae-agent-certs\") pod \"konnectivity-agent-r67pk\" (UID: \"15ea9542-9e0b-4000-8e55-0a4b734105ae\") " pod="kube-system/konnectivity-agent-r67pk"
Sep 16 17:24:38.989097 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988351    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/79c16240-59ee-4b5e-98c9-86ef04179efa-hosts-file\") pod \"node-resolver-2r4vj\" (UID: \"79c16240-59ee-4b5e-98c9-86ef04179efa\") " pod="openshift-dns/node-resolver-2r4vj"
Sep 16 17:24:38.989097 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988379    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-run-openvswitch\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.989097 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988404    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-host-cni-netd\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.989097 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988430    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-host\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.989097 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988456    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5189ad61-97f5-4970-b0f5-acc0606b13f8-socket-dir\") pod \"aws-ebs-csi-driver-node-489qj\" (UID: \"5189ad61-97f5-4970-b0f5-acc0606b13f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj"
Sep 16 17:24:38.989097 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988482    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb4jn\" (UniqueName: \"kubernetes.io/projected/5189ad61-97f5-4970-b0f5-acc0606b13f8-kube-api-access-mb4jn\") pod \"aws-ebs-csi-driver-node-489qj\" (UID: \"5189ad61-97f5-4970-b0f5-acc0606b13f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj"
Sep 16 17:24:38.989097 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988508    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-multus-socket-dir-parent\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.989097 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988543    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-multus-conf-dir\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.989097 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988568    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a83c017c-9dbf-4051-a74c-e5fa85eda19f-multus-daemon-config\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.989930 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988593    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-host-kubelet\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.989930 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988618    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-run-ovn\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.989930 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988643    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-node-log\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.989930 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988670    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/af87e079-c946-4c98-86fc-4be5a3098d8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tkzcs\" (UID: \"af87e079-c946-4c98-86fc-4be5a3098d8a\") " pod="openshift-multus/multus-additional-cni-plugins-tkzcs"
Sep 16 17:24:38.989930 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988697    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-host-run-multus-certs\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.989930 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988726    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvjzr\" (UniqueName: \"kubernetes.io/projected/a83c017c-9dbf-4051-a74c-e5fa85eda19f-kube-api-access-cvjzr\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.989930 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988753    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5189ad61-97f5-4970-b0f5-acc0606b13f8-registration-dir\") pod \"aws-ebs-csi-driver-node-489qj\" (UID: \"5189ad61-97f5-4970-b0f5-acc0606b13f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj"
Sep 16 17:24:38.989930 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988780    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-host-run-k8s-cni-cncf-io\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.989930 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988806    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-etc-sysconfig\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.989930 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988831    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f689aa90-24ff-48a5-8e66-f2e5d749dbd4-host\") pod \"node-ca-ddqwd\" (UID: \"f689aa90-24ff-48a5-8e66-f2e5d749dbd4\") " pod="openshift-image-registry/node-ca-ddqwd"
Sep 16 17:24:38.989930 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988858    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05821b99-6706-481a-a80c-0a86275f520a-host-slash\") pod \"iptables-alerter-hvbv7\" (UID: \"05821b99-6706-481a-a80c-0a86275f520a\") " pod="openshift-network-operator/iptables-alerter-hvbv7"
Sep 16 17:24:38.989930 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988900    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-host-var-lib-cni-bin\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.989930 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988945    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d970e5c-29fc-44af-b138-0e60309f2ac0-ovn-node-metrics-cert\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.989930 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.988976    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af87e079-c946-4c98-86fc-4be5a3098d8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tkzcs\" (UID: \"af87e079-c946-4c98-86fc-4be5a3098d8a\") " pod="openshift-multus/multus-additional-cni-plugins-tkzcs"
Sep 16 17:24:38.989930 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.989004    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-system-cni-dir\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.989930 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.989033    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-run\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.989930 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.989059    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-lib-modules\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.990762 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.989086    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-var-lib-kubelet\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.990762 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.989116    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sr5g\" (UniqueName: \"kubernetes.io/projected/c88f1596-e3ef-45ce-b8f6-55fc246bfdab-kube-api-access-7sr5g\") pod \"network-check-target-89r27\" (UID: \"c88f1596-e3ef-45ce-b8f6-55fc246bfdab\") " pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:24:38.990762 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.989143    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79c16240-59ee-4b5e-98c9-86ef04179efa-tmp-dir\") pod \"node-resolver-2r4vj\" (UID: \"79c16240-59ee-4b5e-98c9-86ef04179efa\") " pod="openshift-dns/node-resolver-2r4vj"
Sep 16 17:24:38.990762 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.989170    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1d970e5c-29fc-44af-b138-0e60309f2ac0-ovnkube-script-lib\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.990762 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.989197    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-etc-sysctl-d\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.990762 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.989224    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5189ad61-97f5-4970-b0f5-acc0606b13f8-sys-fs\") pod \"aws-ebs-csi-driver-node-489qj\" (UID: \"5189ad61-97f5-4970-b0f5-acc0606b13f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj"
Sep 16 17:24:38.990762 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.989359    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5189ad61-97f5-4970-b0f5-acc0606b13f8-sys-fs\") pod \"aws-ebs-csi-driver-node-489qj\" (UID: \"5189ad61-97f5-4970-b0f5-acc0606b13f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj"
Sep 16 17:24:38.990762 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.989411    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-host-run-netns\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.990762 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.989970    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af87e079-c946-4c98-86fc-4be5a3098d8a-system-cni-dir\") pod \"multus-additional-cni-plugins-tkzcs\" (UID: \"af87e079-c946-4c98-86fc-4be5a3098d8a\") " pod="openshift-multus/multus-additional-cni-plugins-tkzcs"
Sep 16 17:24:38.990762 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.990019    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af87e079-c946-4c98-86fc-4be5a3098d8a-cnibin\") pod \"multus-additional-cni-plugins-tkzcs\" (UID: \"af87e079-c946-4c98-86fc-4be5a3098d8a\") " pod="openshift-multus/multus-additional-cni-plugins-tkzcs"
Sep 16 17:24:38.990762 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.990037    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-sys\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.990762 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.990108    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-hostroot\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.990762 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.990199    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-var-lib-openvswitch\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.990762 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.990227    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-host-var-lib-cni-multus\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.990762 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.990269    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-etc-openvswitch\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.990762 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:38.990345    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:38.990762 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.990383    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-systemd-units\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.991594 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:38.990425    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs podName:9d27ec00-06e7-431b-925f-addd45ff0115 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:39.490407386 +0000 UTC m=+3.078797357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs") pod "network-metrics-daemon-fw78m" (UID: "9d27ec00-06e7-431b-925f-addd45ff0115") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:38.991594 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.990435    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-run-systemd\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.991594 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.990489    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-multus-cni-dir\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.991594 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.990523    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-host-slash\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.991594 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.990571    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-host-run-netns\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.991594 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.990581    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-etc-modprobe-d\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.991594 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.990611    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-host-cni-bin\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.991594 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.990671    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-cnibin\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.991594 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.990714    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.991594 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.990838    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-etc-kubernetes\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.991594 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.990944    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1d970e5c-29fc-44af-b138-0e60309f2ac0-env-overrides\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.991594 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.991081    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f689aa90-24ff-48a5-8e66-f2e5d749dbd4-serviceca\") pod \"node-ca-ddqwd\" (UID: \"f689aa90-24ff-48a5-8e66-f2e5d749dbd4\") " pod="openshift-image-registry/node-ca-ddqwd"
Sep 16 17:24:38.991594 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.991160    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5189ad61-97f5-4970-b0f5-acc0606b13f8-etc-selinux\") pod \"aws-ebs-csi-driver-node-489qj\" (UID: \"5189ad61-97f5-4970-b0f5-acc0606b13f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj"
Sep 16 17:24:38.991594 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.991182    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5189ad61-97f5-4970-b0f5-acc0606b13f8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-489qj\" (UID: \"5189ad61-97f5-4970-b0f5-acc0606b13f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj"
Sep 16 17:24:38.991594 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.991081    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/05821b99-6706-481a-a80c-0a86275f520a-iptables-alerter-script\") pod \"iptables-alerter-hvbv7\" (UID: \"05821b99-6706-481a-a80c-0a86275f520a\") " pod="openshift-network-operator/iptables-alerter-hvbv7"
Sep 16 17:24:38.991594 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.991324    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 17:24:38.992428 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.991923    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a83c017c-9dbf-4051-a74c-e5fa85eda19f-cni-binary-copy\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.992428 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.991965    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1d970e5c-29fc-44af-b138-0e60309f2ac0-ovnkube-config\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.992428 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.992000    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/15ea9542-9e0b-4000-8e55-0a4b734105ae-konnectivity-ca\") pod \"konnectivity-agent-r67pk\" (UID: \"15ea9542-9e0b-4000-8e55-0a4b734105ae\") " pod="kube-system/konnectivity-agent-r67pk"
Sep 16 17:24:38.992428 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.992054    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-host-var-lib-kubelet\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.992428 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.992104    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-host-run-multus-certs\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.992428 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.992214    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-system-cni-dir\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.992428 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.992257    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/af87e079-c946-4c98-86fc-4be5a3098d8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tkzcs\" (UID: \"af87e079-c946-4c98-86fc-4be5a3098d8a\") " pod="openshift-multus/multus-additional-cni-plugins-tkzcs"
Sep 16 17:24:38.992428 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.992267    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-run\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.992428 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.992293    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af87e079-c946-4c98-86fc-4be5a3098d8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tkzcs\" (UID: \"af87e079-c946-4c98-86fc-4be5a3098d8a\") " pod="openshift-multus/multus-additional-cni-plugins-tkzcs"
Sep 16 17:24:38.992428 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.992327    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-etc-sysconfig\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.992428 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.992356    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f689aa90-24ff-48a5-8e66-f2e5d749dbd4-host\") pod \"node-ca-ddqwd\" (UID: \"f689aa90-24ff-48a5-8e66-f2e5d749dbd4\") " pod="openshift-image-registry/node-ca-ddqwd"
Sep 16 17:24:38.992428 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.992346    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-lib-modules\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.992428 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.992408    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05821b99-6706-481a-a80c-0a86275f520a-host-slash\") pod \"iptables-alerter-hvbv7\" (UID: \"05821b99-6706-481a-a80c-0a86275f520a\") " pod="openshift-network-operator/iptables-alerter-hvbv7"
Sep 16 17:24:38.992428 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.992419    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-var-lib-kubelet\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.993150 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.992460    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-host-var-lib-cni-bin\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.993150 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.992509    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/af87e079-c946-4c98-86fc-4be5a3098d8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tkzcs\" (UID: \"af87e079-c946-4c98-86fc-4be5a3098d8a\") " pod="openshift-multus/multus-additional-cni-plugins-tkzcs"
Sep 16 17:24:38.993150 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.992610    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-etc-systemd\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.993150 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.992675    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-etc-kubernetes\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.993150 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.992651    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-host-run-k8s-cni-cncf-io\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.993150 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.992951    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79c16240-59ee-4b5e-98c9-86ef04179efa-tmp-dir\") pod \"node-resolver-2r4vj\" (UID: \"79c16240-59ee-4b5e-98c9-86ef04179efa\") " pod="openshift-dns/node-resolver-2r4vj"
Sep 16 17:24:38.993150 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.992995    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-log-socket\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.993150 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.993072    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-host-run-ovn-kubernetes\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.993150 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.993128    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-etc-sysctl-d\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.993606 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.993251    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af87e079-c946-4c98-86fc-4be5a3098d8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-tkzcs\" (UID: \"af87e079-c946-4c98-86fc-4be5a3098d8a\") " pod="openshift-multus/multus-additional-cni-plugins-tkzcs"
Sep 16 17:24:38.993606 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.993308    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1d970e5c-29fc-44af-b138-0e60309f2ac0-ovnkube-script-lib\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.993606 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.993291    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-etc-sysctl-conf\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.993606 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.993375    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5189ad61-97f5-4970-b0f5-acc0606b13f8-device-dir\") pod \"aws-ebs-csi-driver-node-489qj\" (UID: \"5189ad61-97f5-4970-b0f5-acc0606b13f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj"
Sep 16 17:24:38.993606 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.993382    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-host-kubelet\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.993606 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.993446    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/79c16240-59ee-4b5e-98c9-86ef04179efa-hosts-file\") pod \"node-resolver-2r4vj\" (UID: \"79c16240-59ee-4b5e-98c9-86ef04179efa\") " pod="openshift-dns/node-resolver-2r4vj"
Sep 16 17:24:38.993606 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.993454    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-multus-socket-dir-parent\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.993606 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.993494    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a83c017c-9dbf-4051-a74c-e5fa85eda19f-multus-conf-dir\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.993606 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.993551    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa114745-379e-413e-8a75-75c24ffa9ee7-host\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.993606 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.993608    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-host-cni-netd\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.994165 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.993672    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-node-log\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.994165 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.993702    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5189ad61-97f5-4970-b0f5-acc0606b13f8-socket-dir\") pod \"aws-ebs-csi-driver-node-489qj\" (UID: \"5189ad61-97f5-4970-b0f5-acc0606b13f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj"
Sep 16 17:24:38.994165 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.993716    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-run-ovn\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.994165 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.992610    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5189ad61-97f5-4970-b0f5-acc0606b13f8-registration-dir\") pod \"aws-ebs-csi-driver-node-489qj\" (UID: \"5189ad61-97f5-4970-b0f5-acc0606b13f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj"
Sep 16 17:24:38.994165 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.993756    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d970e5c-29fc-44af-b138-0e60309f2ac0-run-openvswitch\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:38.996009 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.994520    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a83c017c-9dbf-4051-a74c-e5fa85eda19f-multus-daemon-config\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:38.996257 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.996233    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa114745-379e-413e-8a75-75c24ffa9ee7-tmp\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:38.996760 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.996741    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/15ea9542-9e0b-4000-8e55-0a4b734105ae-agent-certs\") pod \"konnectivity-agent-r67pk\" (UID: \"15ea9542-9e0b-4000-8e55-0a4b734105ae\") " pod="kube-system/konnectivity-agent-r67pk"
Sep 16 17:24:39.001361 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.997044    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d970e5c-29fc-44af-b138-0e60309f2ac0-ovn-node-metrics-cert\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:39.001361 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:38.999030    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fa114745-379e-413e-8a75-75c24ffa9ee7-etc-tuned\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:39.004785 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:39.004670    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-160.ec2.internal" event={"ID":"0df2195e00f1241c11a998a6a4518995","Type":"ContainerStarted","Data":"8eb128d47875f28cb57068b74bf2881d6c811c941db9344654a2f2f21b82d3e5"}
Sep 16 17:24:39.007252 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:39.007227    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-160.ec2.internal" event={"ID":"82ef5bcb9fdb0c81f8cc0a52c6999929","Type":"ContainerStarted","Data":"854bfed690430a4d16e0a22bc627e9b5af4a3cefec1ed95a056769d86d3c4250"}
Sep 16 17:24:39.012434 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:39.012407    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cp86\" (UniqueName: \"kubernetes.io/projected/05821b99-6706-481a-a80c-0a86275f520a-kube-api-access-4cp86\") pod \"iptables-alerter-hvbv7\" (UID: \"05821b99-6706-481a-a80c-0a86275f520a\") " pod="openshift-network-operator/iptables-alerter-hvbv7"
Sep 16 17:24:39.019691 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:39.019336    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:39.019691 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:39.019362    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:39.019691 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:39.019380    2562 projected.go:196] Error preparing data for projected volume kube-api-access-7sr5g for pod openshift-network-diagnostics/network-check-target-89r27: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:39.019691 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:39.019452    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c88f1596-e3ef-45ce-b8f6-55fc246bfdab-kube-api-access-7sr5g podName:c88f1596-e3ef-45ce-b8f6-55fc246bfdab nodeName:}" failed. No retries permitted until 2026-09-16 17:24:39.519431253 +0000 UTC m=+3.107821238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7sr5g" (UniqueName: "kubernetes.io/projected/c88f1596-e3ef-45ce-b8f6-55fc246bfdab-kube-api-access-7sr5g") pod "network-check-target-89r27" (UID: "c88f1596-e3ef-45ce-b8f6-55fc246bfdab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:39.020151 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:39.020127    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdgfq\" (UniqueName: \"kubernetes.io/projected/79c16240-59ee-4b5e-98c9-86ef04179efa-kube-api-access-qdgfq\") pod \"node-resolver-2r4vj\" (UID: \"79c16240-59ee-4b5e-98c9-86ef04179efa\") " pod="openshift-dns/node-resolver-2r4vj"
Sep 16 17:24:39.020399 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:39.020375    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bd7c\" (UniqueName: \"kubernetes.io/projected/9d27ec00-06e7-431b-925f-addd45ff0115-kube-api-access-7bd7c\") pod \"network-metrics-daemon-fw78m\" (UID: \"9d27ec00-06e7-431b-925f-addd45ff0115\") " pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:24:39.021132 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:39.021099    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp5hx\" (UniqueName: \"kubernetes.io/projected/af87e079-c946-4c98-86fc-4be5a3098d8a-kube-api-access-sp5hx\") pod \"multus-additional-cni-plugins-tkzcs\" (UID: \"af87e079-c946-4c98-86fc-4be5a3098d8a\") " pod="openshift-multus/multus-additional-cni-plugins-tkzcs"
Sep 16 17:24:39.021285 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:39.021256    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvjzr\" (UniqueName: \"kubernetes.io/projected/a83c017c-9dbf-4051-a74c-e5fa85eda19f-kube-api-access-cvjzr\") pod \"multus-lpd4g\" (UID: \"a83c017c-9dbf-4051-a74c-e5fa85eda19f\") " pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:39.022587 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:39.022558    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d55mj\" (UniqueName: \"kubernetes.io/projected/fa114745-379e-413e-8a75-75c24ffa9ee7-kube-api-access-d55mj\") pod \"tuned-jdd4s\" (UID: \"fa114745-379e-413e-8a75-75c24ffa9ee7\") " pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:39.022873 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:39.022853    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb4jn\" (UniqueName: \"kubernetes.io/projected/5189ad61-97f5-4970-b0f5-acc0606b13f8-kube-api-access-mb4jn\") pod \"aws-ebs-csi-driver-node-489qj\" (UID: \"5189ad61-97f5-4970-b0f5-acc0606b13f8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj"
Sep 16 17:24:39.025064 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:39.025040    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj9m9\" (UniqueName: \"kubernetes.io/projected/f689aa90-24ff-48a5-8e66-f2e5d749dbd4-kube-api-access-wj9m9\") pod \"node-ca-ddqwd\" (UID: \"f689aa90-24ff-48a5-8e66-f2e5d749dbd4\") " pod="openshift-image-registry/node-ca-ddqwd"
Sep 16 17:24:39.025826 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:39.025803    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj6th\" (UniqueName: \"kubernetes.io/projected/1d970e5c-29fc-44af-b138-0e60309f2ac0-kube-api-access-xj6th\") pod \"ovnkube-node-t6p26\" (UID: \"1d970e5c-29fc-44af-b138-0e60309f2ac0\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:39.097265 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:39.097229    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hvbv7"
Sep 16 17:24:39.110294 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:39.110250    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tkzcs"
Sep 16 17:24:39.111073 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:39.111041    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05821b99_6706_481a_a80c_0a86275f520a.slice/crio-7295639d7a3ae8ee7cd1546d8ea6f74c21a5cb48ff77c4489b51d359913c6258 WatchSource:0}: Error finding container 7295639d7a3ae8ee7cd1546d8ea6f74c21a5cb48ff77c4489b51d359913c6258: Status 404 returned error can't find the container with id 7295639d7a3ae8ee7cd1546d8ea6f74c21a5cb48ff77c4489b51d359913c6258
Sep 16 17:24:39.119041 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:39.119017    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lpd4g"
Sep 16 17:24:39.120921 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:39.120893    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf87e079_c946_4c98_86fc_4be5a3098d8a.slice/crio-c57112ac84383357288eba11c8698951b8ac865bb961ca30754e4af12b8e9dbe WatchSource:0}: Error finding container c57112ac84383357288eba11c8698951b8ac865bb961ca30754e4af12b8e9dbe: Status 404 returned error can't find the container with id c57112ac84383357288eba11c8698951b8ac865bb961ca30754e4af12b8e9dbe
Sep 16 17:24:39.126950 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:39.126928    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:24:39.135466 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:39.135443    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj"
Sep 16 17:24:39.144340 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:39.144318    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r67pk"
Sep 16 17:24:39.152077 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:39.152053    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jdd4s"
Sep 16 17:24:39.160876 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:39.160841    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2r4vj"
Sep 16 17:24:39.168513 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:39.168491    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ddqwd"
Sep 16 17:24:39.393151 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:39.393055    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b47007d-992e-4660-900c-b4807ce86b3f-original-pull-secret\") pod \"global-pull-secret-syncer-scrxg\" (UID: \"4b47007d-992e-4660-900c-b4807ce86b3f\") " pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:24:39.393306 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:39.393224    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:39.393306 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:39.393294    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b47007d-992e-4660-900c-b4807ce86b3f-original-pull-secret podName:4b47007d-992e-4660-900c-b4807ce86b3f nodeName:}" failed. No retries permitted until 2026-09-16 17:24:40.393272496 +0000 UTC m=+3.981662460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b47007d-992e-4660-900c-b4807ce86b3f-original-pull-secret") pod "global-pull-secret-syncer-scrxg" (UID: "4b47007d-992e-4660-900c-b4807ce86b3f") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:39.494151 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:39.494117    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs\") pod \"network-metrics-daemon-fw78m\" (UID: \"9d27ec00-06e7-431b-925f-addd45ff0115\") " pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:24:39.494340 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:39.494301    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:39.494427 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:39.494413    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs podName:9d27ec00-06e7-431b-925f-addd45ff0115 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:40.494389824 +0000 UTC m=+4.082779813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs") pod "network-metrics-daemon-fw78m" (UID: "9d27ec00-06e7-431b-925f-addd45ff0115") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:39.595421 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:39.595371    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sr5g\" (UniqueName: \"kubernetes.io/projected/c88f1596-e3ef-45ce-b8f6-55fc246bfdab-kube-api-access-7sr5g\") pod \"network-check-target-89r27\" (UID: \"c88f1596-e3ef-45ce-b8f6-55fc246bfdab\") " pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:24:39.595699 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:39.595580    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:39.595699 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:39.595615    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:39.595699 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:39.595629    2562 projected.go:196] Error preparing data for projected volume kube-api-access-7sr5g for pod openshift-network-diagnostics/network-check-target-89r27: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:39.595909 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:39.595706    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c88f1596-e3ef-45ce-b8f6-55fc246bfdab-kube-api-access-7sr5g podName:c88f1596-e3ef-45ce-b8f6-55fc246bfdab nodeName:}" failed. No retries permitted until 2026-09-16 17:24:40.595686061 +0000 UTC m=+4.184076039 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7sr5g" (UniqueName: "kubernetes.io/projected/c88f1596-e3ef-45ce-b8f6-55fc246bfdab-kube-api-access-7sr5g") pod "network-check-target-89r27" (UID: "c88f1596-e3ef-45ce-b8f6-55fc246bfdab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:39.815667 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:39.815629    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83c017c_9dbf_4051_a74c_e5fa85eda19f.slice/crio-5a73d8f167adef9b33df3233e15e459fad99b4d7d4387b3426755b005b3f5b4d WatchSource:0}: Error finding container 5a73d8f167adef9b33df3233e15e459fad99b4d7d4387b3426755b005b3f5b4d: Status 404 returned error can't find the container with id 5a73d8f167adef9b33df3233e15e459fad99b4d7d4387b3426755b005b3f5b4d
Sep 16 17:24:39.871417 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:39.871381    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 17:19:39 +0000 UTC" deadline="2028-05-27 07:32:59.573025636 +0000 UTC"
Sep 16 17:24:39.871417 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:39.871412    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14846h8m19.701616768s"
Sep 16 17:24:40.010443 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:40.010398    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lpd4g" event={"ID":"a83c017c-9dbf-4051-a74c-e5fa85eda19f","Type":"ContainerStarted","Data":"5a73d8f167adef9b33df3233e15e459fad99b4d7d4387b3426755b005b3f5b4d"}
Sep 16 17:24:40.011819 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:40.011766    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tkzcs" event={"ID":"af87e079-c946-4c98-86fc-4be5a3098d8a","Type":"ContainerStarted","Data":"c57112ac84383357288eba11c8698951b8ac865bb961ca30754e4af12b8e9dbe"}
Sep 16 17:24:40.012995 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:40.012970    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hvbv7" event={"ID":"05821b99-6706-481a-a80c-0a86275f520a","Type":"ContainerStarted","Data":"7295639d7a3ae8ee7cd1546d8ea6f74c21a5cb48ff77c4489b51d359913c6258"}
Sep 16 17:24:40.224032 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:40.223768    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf689aa90_24ff_48a5_8e66_f2e5d749dbd4.slice/crio-3b3dd269820eb19f504a4b95e9df8f8453c7ca878e58078dfb9b7d5d80b3855d WatchSource:0}: Error finding container 3b3dd269820eb19f504a4b95e9df8f8453c7ca878e58078dfb9b7d5d80b3855d: Status 404 returned error can't find the container with id 3b3dd269820eb19f504a4b95e9df8f8453c7ca878e58078dfb9b7d5d80b3855d
Sep 16 17:24:40.224655 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:40.224628    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5189ad61_97f5_4970_b0f5_acc0606b13f8.slice/crio-f366a69e0d9ada86e3e5a2cc1c7560e20eaf6526a8719759043eeade49e6cce4 WatchSource:0}: Error finding container f366a69e0d9ada86e3e5a2cc1c7560e20eaf6526a8719759043eeade49e6cce4: Status 404 returned error can't find the container with id f366a69e0d9ada86e3e5a2cc1c7560e20eaf6526a8719759043eeade49e6cce4
Sep 16 17:24:40.225587 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:40.225500    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa114745_379e_413e_8a75_75c24ffa9ee7.slice/crio-cff7fdd555fb18d358eb92a314a8485c4367d0e7fdb72eada96c177cef96a911 WatchSource:0}: Error finding container cff7fdd555fb18d358eb92a314a8485c4367d0e7fdb72eada96c177cef96a911: Status 404 returned error can't find the container with id cff7fdd555fb18d358eb92a314a8485c4367d0e7fdb72eada96c177cef96a911
Sep 16 17:24:40.226951 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:40.226866    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15ea9542_9e0b_4000_8e55_0a4b734105ae.slice/crio-98ae92703413321d0ac1072b8b44f0f59d6fb1b240aa026b0774b100a3a62e02 WatchSource:0}: Error finding container 98ae92703413321d0ac1072b8b44f0f59d6fb1b240aa026b0774b100a3a62e02: Status 404 returned error can't find the container with id 98ae92703413321d0ac1072b8b44f0f59d6fb1b240aa026b0774b100a3a62e02
Sep 16 17:24:40.227974 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:40.227953    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c16240_59ee_4b5e_98c9_86ef04179efa.slice/crio-31021a52280335d7c5f681e142ed0cfddae95e93e6bb702aca2ce05e4e7b6d5d WatchSource:0}: Error finding container 31021a52280335d7c5f681e142ed0cfddae95e93e6bb702aca2ce05e4e7b6d5d: Status 404 returned error can't find the container with id 31021a52280335d7c5f681e142ed0cfddae95e93e6bb702aca2ce05e4e7b6d5d
Sep 16 17:24:40.230183 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:24:40.230149    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d970e5c_29fc_44af_b138_0e60309f2ac0.slice/crio-a1baee5104d3496d525ac3a64dabd77392a3c41254ddc90bf8a253c90d08663d WatchSource:0}: Error finding container a1baee5104d3496d525ac3a64dabd77392a3c41254ddc90bf8a253c90d08663d: Status 404 returned error can't find the container with id a1baee5104d3496d525ac3a64dabd77392a3c41254ddc90bf8a253c90d08663d
Sep 16 17:24:40.403151 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:40.403046    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b47007d-992e-4660-900c-b4807ce86b3f-original-pull-secret\") pod \"global-pull-secret-syncer-scrxg\" (UID: \"4b47007d-992e-4660-900c-b4807ce86b3f\") " pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:24:40.403323 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:40.403168    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:40.403323 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:40.403234    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b47007d-992e-4660-900c-b4807ce86b3f-original-pull-secret podName:4b47007d-992e-4660-900c-b4807ce86b3f nodeName:}" failed. No retries permitted until 2026-09-16 17:24:42.403212736 +0000 UTC m=+5.991602703 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b47007d-992e-4660-900c-b4807ce86b3f-original-pull-secret") pod "global-pull-secret-syncer-scrxg" (UID: "4b47007d-992e-4660-900c-b4807ce86b3f") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:40.504333 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:40.504250    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs\") pod \"network-metrics-daemon-fw78m\" (UID: \"9d27ec00-06e7-431b-925f-addd45ff0115\") " pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:24:40.504555 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:40.504536    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:40.504700 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:40.504686    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs podName:9d27ec00-06e7-431b-925f-addd45ff0115 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:42.504665875 +0000 UTC m=+6.093055862 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs") pod "network-metrics-daemon-fw78m" (UID: "9d27ec00-06e7-431b-925f-addd45ff0115") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:40.605010 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:40.604979    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sr5g\" (UniqueName: \"kubernetes.io/projected/c88f1596-e3ef-45ce-b8f6-55fc246bfdab-kube-api-access-7sr5g\") pod \"network-check-target-89r27\" (UID: \"c88f1596-e3ef-45ce-b8f6-55fc246bfdab\") " pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:24:40.605228 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:40.605210    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:40.605314 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:40.605235    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:40.605314 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:40.605248    2562 projected.go:196] Error preparing data for projected volume kube-api-access-7sr5g for pod openshift-network-diagnostics/network-check-target-89r27: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:40.605314 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:40.605312    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c88f1596-e3ef-45ce-b8f6-55fc246bfdab-kube-api-access-7sr5g podName:c88f1596-e3ef-45ce-b8f6-55fc246bfdab nodeName:}" failed. No retries permitted until 2026-09-16 17:24:42.605291257 +0000 UTC m=+6.193681236 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7sr5g" (UniqueName: "kubernetes.io/projected/c88f1596-e3ef-45ce-b8f6-55fc246bfdab-kube-api-access-7sr5g") pod "network-check-target-89r27" (UID: "c88f1596-e3ef-45ce-b8f6-55fc246bfdab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:40.872119 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:40.871999    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 17:19:39 +0000 UTC" deadline="2028-03-23 09:06:41.525337508 +0000 UTC"
Sep 16 17:24:40.872119 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:40.872069    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13287h42m0.653272721s"
Sep 16 17:24:40.998138 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:40.998108    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:24:40.998312 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:40.998251    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fw78m" podUID="9d27ec00-06e7-431b-925f-addd45ff0115"
Sep 16 17:24:40.999024 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:40.998998    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:24:40.999151 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:40.999108    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scrxg" podUID="4b47007d-992e-4660-900c-b4807ce86b3f"
Sep 16 17:24:40.999212 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:40.999168    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:24:40.999272 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:40.999239    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89r27" podUID="c88f1596-e3ef-45ce-b8f6-55fc246bfdab"
Sep 16 17:24:41.024083 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:41.023950    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj" event={"ID":"5189ad61-97f5-4970-b0f5-acc0606b13f8","Type":"ContainerStarted","Data":"f366a69e0d9ada86e3e5a2cc1c7560e20eaf6526a8719759043eeade49e6cce4"}
Sep 16 17:24:41.026504 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:41.026409    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ddqwd" event={"ID":"f689aa90-24ff-48a5-8e66-f2e5d749dbd4","Type":"ContainerStarted","Data":"3b3dd269820eb19f504a4b95e9df8f8453c7ca878e58078dfb9b7d5d80b3855d"}
Sep 16 17:24:41.031861 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:41.031807    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-160.ec2.internal" event={"ID":"0df2195e00f1241c11a998a6a4518995","Type":"ContainerStarted","Data":"ec5d017317264894f9fa30d8d1b4ca1f80750921a0300c3e5fec5d22bf252989"}
Sep 16 17:24:41.037483 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:41.036857    2562 generic.go:356] "Generic (PLEG): container finished" podID="82ef5bcb9fdb0c81f8cc0a52c6999929" containerID="48c102a5fd4d576ff90cd3107651b2d5614964a1e2a9fd0b31b10088f9db5b7d" exitCode=0
Sep 16 17:24:41.037483 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:41.036920    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-160.ec2.internal" event={"ID":"82ef5bcb9fdb0c81f8cc0a52c6999929","Type":"ContainerDied","Data":"48c102a5fd4d576ff90cd3107651b2d5614964a1e2a9fd0b31b10088f9db5b7d"}
Sep 16 17:24:41.039517 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:41.039491    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6p26" event={"ID":"1d970e5c-29fc-44af-b138-0e60309f2ac0","Type":"ContainerStarted","Data":"a1baee5104d3496d525ac3a64dabd77392a3c41254ddc90bf8a253c90d08663d"}
Sep 16 17:24:41.048283 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:41.048234    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-160.ec2.internal" podStartSLOduration=3.048215762 podStartE2EDuration="3.048215762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:24:41.047437993 +0000 UTC m=+4.635827983" watchObservedRunningTime="2026-09-16 17:24:41.048215762 +0000 UTC m=+4.636605785"
Sep 16 17:24:41.050945 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:41.050918    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2r4vj" event={"ID":"79c16240-59ee-4b5e-98c9-86ef04179efa","Type":"ContainerStarted","Data":"31021a52280335d7c5f681e142ed0cfddae95e93e6bb702aca2ce05e4e7b6d5d"}
Sep 16 17:24:41.057039 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:41.057013    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jdd4s" event={"ID":"fa114745-379e-413e-8a75-75c24ffa9ee7","Type":"ContainerStarted","Data":"cff7fdd555fb18d358eb92a314a8485c4367d0e7fdb72eada96c177cef96a911"}
Sep 16 17:24:41.061311 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:41.061285    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r67pk" event={"ID":"15ea9542-9e0b-4000-8e55-0a4b734105ae","Type":"ContainerStarted","Data":"98ae92703413321d0ac1072b8b44f0f59d6fb1b240aa026b0774b100a3a62e02"}
Sep 16 17:24:42.082483 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:42.082376    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-160.ec2.internal" event={"ID":"82ef5bcb9fdb0c81f8cc0a52c6999929","Type":"ContainerStarted","Data":"1aed5062b4f55c4ce2606dc0d90d60542a8c389ecd67b1bbd333225935ebe551"}
Sep 16 17:24:42.418792 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:42.418753    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b47007d-992e-4660-900c-b4807ce86b3f-original-pull-secret\") pod \"global-pull-secret-syncer-scrxg\" (UID: \"4b47007d-992e-4660-900c-b4807ce86b3f\") " pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:24:42.418990 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:42.418944    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:42.419058 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:42.419010    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b47007d-992e-4660-900c-b4807ce86b3f-original-pull-secret podName:4b47007d-992e-4660-900c-b4807ce86b3f nodeName:}" failed. No retries permitted until 2026-09-16 17:24:46.418991989 +0000 UTC m=+10.007381965 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b47007d-992e-4660-900c-b4807ce86b3f-original-pull-secret") pod "global-pull-secret-syncer-scrxg" (UID: "4b47007d-992e-4660-900c-b4807ce86b3f") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:42.519979 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:42.519388    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs\") pod \"network-metrics-daemon-fw78m\" (UID: \"9d27ec00-06e7-431b-925f-addd45ff0115\") " pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:24:42.519979 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:42.519561    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:42.519979 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:42.519626    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs podName:9d27ec00-06e7-431b-925f-addd45ff0115 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:46.519607509 +0000 UTC m=+10.107997491 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs") pod "network-metrics-daemon-fw78m" (UID: "9d27ec00-06e7-431b-925f-addd45ff0115") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:42.620279 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:42.620242    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sr5g\" (UniqueName: \"kubernetes.io/projected/c88f1596-e3ef-45ce-b8f6-55fc246bfdab-kube-api-access-7sr5g\") pod \"network-check-target-89r27\" (UID: \"c88f1596-e3ef-45ce-b8f6-55fc246bfdab\") " pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:24:42.620482 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:42.620434    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:42.620482 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:42.620470    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:42.620596 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:42.620486    2562 projected.go:196] Error preparing data for projected volume kube-api-access-7sr5g for pod openshift-network-diagnostics/network-check-target-89r27: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:42.620596 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:42.620548    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c88f1596-e3ef-45ce-b8f6-55fc246bfdab-kube-api-access-7sr5g podName:c88f1596-e3ef-45ce-b8f6-55fc246bfdab nodeName:}" failed. No retries permitted until 2026-09-16 17:24:46.620527943 +0000 UTC m=+10.208917923 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7sr5g" (UniqueName: "kubernetes.io/projected/c88f1596-e3ef-45ce-b8f6-55fc246bfdab-kube-api-access-7sr5g") pod "network-check-target-89r27" (UID: "c88f1596-e3ef-45ce-b8f6-55fc246bfdab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:42.997907 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:42.997876    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:24:42.998095 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:42.998006    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scrxg" podUID="4b47007d-992e-4660-900c-b4807ce86b3f"
Sep 16 17:24:42.998095 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:42.998043    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:24:42.998207 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:42.998117    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89r27" podUID="c88f1596-e3ef-45ce-b8f6-55fc246bfdab"
Sep 16 17:24:42.998207 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:42.998163    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:24:42.998302 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:42.998235    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fw78m" podUID="9d27ec00-06e7-431b-925f-addd45ff0115"
Sep 16 17:24:44.997458 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:44.996731    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:24:44.997458 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:44.996888    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fw78m" podUID="9d27ec00-06e7-431b-925f-addd45ff0115"
Sep 16 17:24:44.997458 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:44.997316    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:24:44.997458 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:44.997408    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scrxg" podUID="4b47007d-992e-4660-900c-b4807ce86b3f"
Sep 16 17:24:44.998187 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:44.998159    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:24:44.998317 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:44.998256    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89r27" podUID="c88f1596-e3ef-45ce-b8f6-55fc246bfdab"
Sep 16 17:24:46.451220 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:46.451188    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b47007d-992e-4660-900c-b4807ce86b3f-original-pull-secret\") pod \"global-pull-secret-syncer-scrxg\" (UID: \"4b47007d-992e-4660-900c-b4807ce86b3f\") " pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:24:46.451659 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:46.451356    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:46.451659 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:46.451425    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b47007d-992e-4660-900c-b4807ce86b3f-original-pull-secret podName:4b47007d-992e-4660-900c-b4807ce86b3f nodeName:}" failed. No retries permitted until 2026-09-16 17:24:54.451406688 +0000 UTC m=+18.039796672 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b47007d-992e-4660-900c-b4807ce86b3f-original-pull-secret") pod "global-pull-secret-syncer-scrxg" (UID: "4b47007d-992e-4660-900c-b4807ce86b3f") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:46.552169 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:46.552130    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs\") pod \"network-metrics-daemon-fw78m\" (UID: \"9d27ec00-06e7-431b-925f-addd45ff0115\") " pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:24:46.552352 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:46.552236    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:46.552352 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:46.552307    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs podName:9d27ec00-06e7-431b-925f-addd45ff0115 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:54.552287391 +0000 UTC m=+18.140677362 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs") pod "network-metrics-daemon-fw78m" (UID: "9d27ec00-06e7-431b-925f-addd45ff0115") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:46.653172 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:46.653134    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sr5g\" (UniqueName: \"kubernetes.io/projected/c88f1596-e3ef-45ce-b8f6-55fc246bfdab-kube-api-access-7sr5g\") pod \"network-check-target-89r27\" (UID: \"c88f1596-e3ef-45ce-b8f6-55fc246bfdab\") " pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:24:46.653366 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:46.653336    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:46.653366 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:46.653359    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:46.653486 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:46.653374    2562 projected.go:196] Error preparing data for projected volume kube-api-access-7sr5g for pod openshift-network-diagnostics/network-check-target-89r27: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:46.653486 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:46.653429    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c88f1596-e3ef-45ce-b8f6-55fc246bfdab-kube-api-access-7sr5g podName:c88f1596-e3ef-45ce-b8f6-55fc246bfdab nodeName:}" failed. No retries permitted until 2026-09-16 17:24:54.65341529 +0000 UTC m=+18.241805265 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7sr5g" (UniqueName: "kubernetes.io/projected/c88f1596-e3ef-45ce-b8f6-55fc246bfdab-kube-api-access-7sr5g") pod "network-check-target-89r27" (UID: "c88f1596-e3ef-45ce-b8f6-55fc246bfdab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:46.997565 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:46.996877    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:24:46.997565 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:46.997003    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fw78m" podUID="9d27ec00-06e7-431b-925f-addd45ff0115"
Sep 16 17:24:46.997565 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:46.997320    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:24:46.997565 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:46.997410    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scrxg" podUID="4b47007d-992e-4660-900c-b4807ce86b3f"
Sep 16 17:24:46.997565 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:46.997459    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:24:46.997565 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:46.997525    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89r27" podUID="c88f1596-e3ef-45ce-b8f6-55fc246bfdab"
Sep 16 17:24:48.996240 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:48.996200    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:24:48.996697 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:48.996242    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:24:48.996697 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:48.996245    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:24:48.996697 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:48.996337    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89r27" podUID="c88f1596-e3ef-45ce-b8f6-55fc246bfdab"
Sep 16 17:24:48.996697 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:48.996442    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scrxg" podUID="4b47007d-992e-4660-900c-b4807ce86b3f"
Sep 16 17:24:48.996697 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:48.996550    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fw78m" podUID="9d27ec00-06e7-431b-925f-addd45ff0115"
Sep 16 17:24:50.995361 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:50.995328    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:24:50.995838 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:50.995328    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:24:50.995838 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:50.995467    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fw78m" podUID="9d27ec00-06e7-431b-925f-addd45ff0115"
Sep 16 17:24:50.995838 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:50.995328    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:24:50.995838 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:50.995530    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scrxg" podUID="4b47007d-992e-4660-900c-b4807ce86b3f"
Sep 16 17:24:50.995838 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:50.995642    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89r27" podUID="c88f1596-e3ef-45ce-b8f6-55fc246bfdab"
Sep 16 17:24:52.996835 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:52.996723    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:24:52.997501 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:52.997468    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89r27" podUID="c88f1596-e3ef-45ce-b8f6-55fc246bfdab"
Sep 16 17:24:52.997963 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:52.997946    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:24:52.998106 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:52.998088    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fw78m" podUID="9d27ec00-06e7-431b-925f-addd45ff0115"
Sep 16 17:24:52.998167 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:52.998154    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:24:52.998276 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:52.998261    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scrxg" podUID="4b47007d-992e-4660-900c-b4807ce86b3f"
Sep 16 17:24:54.508703 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:54.508500    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b47007d-992e-4660-900c-b4807ce86b3f-original-pull-secret\") pod \"global-pull-secret-syncer-scrxg\" (UID: \"4b47007d-992e-4660-900c-b4807ce86b3f\") " pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:24:54.509187 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:54.508663    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:54.509187 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:54.508818    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b47007d-992e-4660-900c-b4807ce86b3f-original-pull-secret podName:4b47007d-992e-4660-900c-b4807ce86b3f nodeName:}" failed. No retries permitted until 2026-09-16 17:25:10.50879197 +0000 UTC m=+34.097181931 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b47007d-992e-4660-900c-b4807ce86b3f-original-pull-secret") pod "global-pull-secret-syncer-scrxg" (UID: "4b47007d-992e-4660-900c-b4807ce86b3f") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:54.610047 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:54.610012    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs\") pod \"network-metrics-daemon-fw78m\" (UID: \"9d27ec00-06e7-431b-925f-addd45ff0115\") " pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:24:54.610208 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:54.610186    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:54.610279 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:54.610267    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs podName:9d27ec00-06e7-431b-925f-addd45ff0115 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:10.61024716 +0000 UTC m=+34.198637137 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs") pod "network-metrics-daemon-fw78m" (UID: "9d27ec00-06e7-431b-925f-addd45ff0115") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:54.711034 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:54.710995    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sr5g\" (UniqueName: \"kubernetes.io/projected/c88f1596-e3ef-45ce-b8f6-55fc246bfdab-kube-api-access-7sr5g\") pod \"network-check-target-89r27\" (UID: \"c88f1596-e3ef-45ce-b8f6-55fc246bfdab\") " pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:24:54.711213 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:54.711191    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:54.711291 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:54.711221    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:54.711291 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:54.711236    2562 projected.go:196] Error preparing data for projected volume kube-api-access-7sr5g for pod openshift-network-diagnostics/network-check-target-89r27: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:54.711385 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:54.711301    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c88f1596-e3ef-45ce-b8f6-55fc246bfdab-kube-api-access-7sr5g podName:c88f1596-e3ef-45ce-b8f6-55fc246bfdab nodeName:}" failed. No retries permitted until 2026-09-16 17:25:10.711284255 +0000 UTC m=+34.299674219 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7sr5g" (UniqueName: "kubernetes.io/projected/c88f1596-e3ef-45ce-b8f6-55fc246bfdab-kube-api-access-7sr5g") pod "network-check-target-89r27" (UID: "c88f1596-e3ef-45ce-b8f6-55fc246bfdab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:54.995683 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:54.995650    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:24:54.995683 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:54.995668    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:24:54.995926 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:54.995770    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:24:54.995926 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:54.995769    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scrxg" podUID="4b47007d-992e-4660-900c-b4807ce86b3f"
Sep 16 17:24:54.995926 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:54.995907    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fw78m" podUID="9d27ec00-06e7-431b-925f-addd45ff0115"
Sep 16 17:24:54.996033 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:54.996005    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89r27" podUID="c88f1596-e3ef-45ce-b8f6-55fc246bfdab"
Sep 16 17:24:56.996005 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:56.995970    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:24:56.996392 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:56.996087    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fw78m" podUID="9d27ec00-06e7-431b-925f-addd45ff0115"
Sep 16 17:24:56.996392 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:56.996098    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:24:56.996392 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:56.996115    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:24:56.996392 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:56.996187    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scrxg" podUID="4b47007d-992e-4660-900c-b4807ce86b3f"
Sep 16 17:24:56.996392 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:56.996235    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89r27" podUID="c88f1596-e3ef-45ce-b8f6-55fc246bfdab"
Sep 16 17:24:58.114062 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:58.114030    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jdd4s" event={"ID":"fa114745-379e-413e-8a75-75c24ffa9ee7","Type":"ContainerStarted","Data":"b5024f9c7a6ea0a2ec709a24034c2ed845bcd7a68d43ccc214d99f30f4afc6b0"}
Sep 16 17:24:58.115371 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:58.115350    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r67pk" event={"ID":"15ea9542-9e0b-4000-8e55-0a4b734105ae","Type":"ContainerStarted","Data":"db95c8ea9dd655066892a7a45b4e564a8074a0307f8ff7180705459e7b04e2c1"}
Sep 16 17:24:58.116749 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:58.116725    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj" event={"ID":"5189ad61-97f5-4970-b0f5-acc0606b13f8","Type":"ContainerStarted","Data":"3a7aa5ecb5064b69684115c650688a468a3895573999db87ab54456732e3bc25"}
Sep 16 17:24:58.118203 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:58.118183    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ddqwd" event={"ID":"f689aa90-24ff-48a5-8e66-f2e5d749dbd4","Type":"ContainerStarted","Data":"1dcf77847c9a1dd56f4f09f730106c98cd8fec049af0d5b8b47384bb42043d50"}
Sep 16 17:24:58.119605 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:58.119581    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tkzcs" event={"ID":"af87e079-c946-4c98-86fc-4be5a3098d8a","Type":"ContainerStarted","Data":"166369cb9bc369da39d8eefdfc552fee9a16b48c124b63c2ca1f06739e867ae7"}
Sep 16 17:24:58.121420 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:58.121401    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6p26" event={"ID":"1d970e5c-29fc-44af-b138-0e60309f2ac0","Type":"ContainerStarted","Data":"fe42f4b4bfba6b06ea6c8c1a7604082c25bf5b275434ad9ed1494ef5e4277681"}
Sep 16 17:24:58.121488 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:58.121454    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6p26" event={"ID":"1d970e5c-29fc-44af-b138-0e60309f2ac0","Type":"ContainerStarted","Data":"59afbbd3ee326c83e3a28f5ded4b840813b6b6fe137a4c8f1d954717d791488b"}
Sep 16 17:24:58.122730 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:58.122711    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2r4vj" event={"ID":"79c16240-59ee-4b5e-98c9-86ef04179efa","Type":"ContainerStarted","Data":"e9ab72015eca7f1ab73b4eb70985b50a7221d977eb27fb2563fe8740e96d19fa"}
Sep 16 17:24:58.158781 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:58.158730    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-160.ec2.internal" podStartSLOduration=20.158708993 podStartE2EDuration="20.158708993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:24:42.098955373 +0000 UTC m=+5.687345358" watchObservedRunningTime="2026-09-16 17:24:58.158708993 +0000 UTC m=+21.747098986"
Sep 16 17:24:58.199782 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:58.199742    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jdd4s" podStartSLOduration=3.659127111 podStartE2EDuration="21.199725524s" totalImagesPullingTime="17.540598413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:24:58.159614192 +0000 UTC m=+21.748004168" watchObservedRunningTime="2026-09-16 17:24:58.199725524 +0000 UTC m=+21.788115508"
Sep 16 17:24:58.200346 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:58.200313    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2r4vj" podStartSLOduration=3.703593989 podStartE2EDuration="21.20030206s" totalImagesPullingTime="17.496708071s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:24:58.200131052 +0000 UTC m=+21.788521037" watchObservedRunningTime="2026-09-16 17:24:58.20030206 +0000 UTC m=+21.788692045"
Sep 16 17:24:58.420047 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:58.419845    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ddqwd" podStartSLOduration=3.879672907 podStartE2EDuration="21.419818165s" totalImagesPullingTime="17.540145258s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:24:58.302428416 +0000 UTC m=+21.890818401" watchObservedRunningTime="2026-09-16 17:24:58.419818165 +0000 UTC m=+22.008208140"
Sep 16 17:24:58.995772 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:58.995746    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:24:58.995912 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:58.995746    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:24:58.995912 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:58.995866    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89r27" podUID="c88f1596-e3ef-45ce-b8f6-55fc246bfdab"
Sep 16 17:24:58.995912 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:58.995746    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:24:58.995912 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:58.995905    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scrxg" podUID="4b47007d-992e-4660-900c-b4807ce86b3f"
Sep 16 17:24:58.996128 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:24:58.995970    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fw78m" podUID="9d27ec00-06e7-431b-925f-addd45ff0115"
Sep 16 17:24:59.126315 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:59.126282    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lpd4g" event={"ID":"a83c017c-9dbf-4051-a74c-e5fa85eda19f","Type":"ContainerStarted","Data":"b00ca92437b4570a129718f9e972a588e99b90d821e0f7c46f0287b257edc9ef"}
Sep 16 17:24:59.127711 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:59.127688    2562 generic.go:356] "Generic (PLEG): container finished" podID="af87e079-c946-4c98-86fc-4be5a3098d8a" containerID="166369cb9bc369da39d8eefdfc552fee9a16b48c124b63c2ca1f06739e867ae7" exitCode=0
Sep 16 17:24:59.127830 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:59.127769    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tkzcs" event={"ID":"af87e079-c946-4c98-86fc-4be5a3098d8a","Type":"ContainerDied","Data":"166369cb9bc369da39d8eefdfc552fee9a16b48c124b63c2ca1f06739e867ae7"}
Sep 16 17:24:59.129186 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:59.129165    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hvbv7" event={"ID":"05821b99-6706-481a-a80c-0a86275f520a","Type":"ContainerStarted","Data":"f745df90c0d40ee5f0e49b7fdb87b5e4b83fd2957a08bcb6744f05c43a0cc66d"}
Sep 16 17:24:59.131907 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:59.131893    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6p26_1d970e5c-29fc-44af-b138-0e60309f2ac0/ovn-acl-logging/0.log"
Sep 16 17:24:59.132221 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:59.132203    2562 generic.go:356] "Generic (PLEG): container finished" podID="1d970e5c-29fc-44af-b138-0e60309f2ac0" containerID="fe42f4b4bfba6b06ea6c8c1a7604082c25bf5b275434ad9ed1494ef5e4277681" exitCode=1
Sep 16 17:24:59.132305 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:59.132286    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6p26" event={"ID":"1d970e5c-29fc-44af-b138-0e60309f2ac0","Type":"ContainerDied","Data":"fe42f4b4bfba6b06ea6c8c1a7604082c25bf5b275434ad9ed1494ef5e4277681"}
Sep 16 17:24:59.132386 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:59.132372    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6p26" event={"ID":"1d970e5c-29fc-44af-b138-0e60309f2ac0","Type":"ContainerStarted","Data":"50454243d51bbe30c6ab0ebd306afd9745f67700522c601fab25863b6783af3b"}
Sep 16 17:24:59.132445 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:59.132427    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6p26" event={"ID":"1d970e5c-29fc-44af-b138-0e60309f2ac0","Type":"ContainerStarted","Data":"138b349508a10ed462dba26e25da0cb561a0be205f0cb19c50343690fbf7fd55"}
Sep 16 17:24:59.132527 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:59.132515    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6p26" event={"ID":"1d970e5c-29fc-44af-b138-0e60309f2ac0","Type":"ContainerStarted","Data":"b66982f5cb254ce87dc1a98ff9bc39dcc10fe5ed5ebe2afe8c9cba1a0be35a2c"}
Sep 16 17:24:59.132583 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:59.132566    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6p26" event={"ID":"1d970e5c-29fc-44af-b138-0e60309f2ac0","Type":"ContainerStarted","Data":"0100e049f89a107a79069f810b60b2e0f66b7580012c84c54d0d9fc418accda8"}
Sep 16 17:24:59.150009 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:59.149966    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-r67pk" podStartSLOduration=12.028784283 podStartE2EDuration="22.149950708s" totalImagesPullingTime="10.121166425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:24:58.419704084 +0000 UTC m=+22.008094067" watchObservedRunningTime="2026-09-16 17:24:59.149950708 +0000 UTC m=+22.738340692"
Sep 16 17:24:59.172783 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:59.172736    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lpd4g" podStartSLOduration=3.81938321 podStartE2EDuration="22.172720454s" totalImagesPullingTime="18.353337244s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:24:59.150160146 +0000 UTC m=+22.738550130" watchObservedRunningTime="2026-09-16 17:24:59.172720454 +0000 UTC m=+22.761110438"
Sep 16 17:24:59.218889 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:59.218788    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hvbv7" podStartSLOduration=3.56634253 podStartE2EDuration="22.21877598s" totalImagesPullingTime="18.65243345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:24:59.173028699 +0000 UTC m=+22.761418682" watchObservedRunningTime="2026-09-16 17:24:59.21877598 +0000 UTC m=+22.807165999"
Sep 16 17:24:59.386479 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:59.386454    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 17:24:59.833655 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:59.833628    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-r67pk"
Sep 16 17:24:59.834341 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:59.834319    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-r67pk"
Sep 16 17:24:59.838459 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:59.838333    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T17:24:59.386472999Z","UUID":"6967b21a-01b4-496d-a6b5-0044e89ffa64","Handler":null,"Name":"","Endpoint":""}
Sep 16 17:24:59.839887 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:59.839868    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 17:24:59.839887 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:24:59.839901    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 17:25:00.137238 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:00.136961    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj" event={"ID":"5189ad61-97f5-4970-b0f5-acc0606b13f8","Type":"ContainerStarted","Data":"146192aed114fbbe62dee17172b4940ad5b07c28cc29ddffa397917a6be56eef"}
Sep 16 17:25:00.996285 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:00.996251    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:25:00.996481 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:00.996257    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:25:00.996481 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:00.996395    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:25:00.996481 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:00.996408    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fw78m" podUID="9d27ec00-06e7-431b-925f-addd45ff0115"
Sep 16 17:25:00.996646 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:00.996479    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scrxg" podUID="4b47007d-992e-4660-900c-b4807ce86b3f"
Sep 16 17:25:00.996646 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:00.996532    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89r27" podUID="c88f1596-e3ef-45ce-b8f6-55fc246bfdab"
Sep 16 17:25:01.142738 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:01.142710    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6p26_1d970e5c-29fc-44af-b138-0e60309f2ac0/ovn-acl-logging/0.log"
Sep 16 17:25:01.143213 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:01.143139    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6p26" event={"ID":"1d970e5c-29fc-44af-b138-0e60309f2ac0","Type":"ContainerStarted","Data":"2e5cfa19b832527121d459c32d82ec7c09c02955650331719a9bc083f8e20d04"}
Sep 16 17:25:01.145575 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:01.145545    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj" event={"ID":"5189ad61-97f5-4970-b0f5-acc0606b13f8","Type":"ContainerStarted","Data":"179a77d619470bec2c55e832d78212f31cd57594209a618f9c2f8c796d825fcf"}
Sep 16 17:25:01.145575 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:01.145571    2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 17:25:01.201028 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:01.200976    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-489qj" podStartSLOduration=4.595868822 podStartE2EDuration="24.200957735s" totalImagesPullingTime="19.605088913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:01.200707366 +0000 UTC m=+24.789097354" watchObservedRunningTime="2026-09-16 17:25:01.200957735 +0000 UTC m=+24.789347722"
Sep 16 17:25:02.996478 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:02.996431    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:25:02.996478 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:02.996477    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:25:02.997038 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:02.996432    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:25:02.997038 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:02.996570    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89r27" podUID="c88f1596-e3ef-45ce-b8f6-55fc246bfdab"
Sep 16 17:25:02.997038 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:02.996627    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fw78m" podUID="9d27ec00-06e7-431b-925f-addd45ff0115"
Sep 16 17:25:02.997038 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:02.996730    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scrxg" podUID="4b47007d-992e-4660-900c-b4807ce86b3f"
Sep 16 17:25:04.157933 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:04.157746    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6p26_1d970e5c-29fc-44af-b138-0e60309f2ac0/ovn-acl-logging/0.log"
Sep 16 17:25:04.158314 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:04.158273    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6p26" event={"ID":"1d970e5c-29fc-44af-b138-0e60309f2ac0","Type":"ContainerStarted","Data":"e00b034813d7f6570d9f5ad8d67a6aa43475156e919a8d16a17e8a6b563b9f86"}
Sep 16 17:25:04.158540 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:04.158518    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:25:04.158666 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:04.158588    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:25:04.158666 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:04.158633    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:25:04.158666 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:04.158659    2562 scope.go:117] "RemoveContainer" containerID="fe42f4b4bfba6b06ea6c8c1a7604082c25bf5b275434ad9ed1494ef5e4277681"
Sep 16 17:25:04.160375 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:04.160343    2562 generic.go:356] "Generic (PLEG): container finished" podID="af87e079-c946-4c98-86fc-4be5a3098d8a" containerID="3ea9431eac1382b0b13722e51031bf85544b6ef1758a1dfc92d7f26d02a23b39" exitCode=0
Sep 16 17:25:04.160479 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:04.160385    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tkzcs" event={"ID":"af87e079-c946-4c98-86fc-4be5a3098d8a","Type":"ContainerDied","Data":"3ea9431eac1382b0b13722e51031bf85544b6ef1758a1dfc92d7f26d02a23b39"}
Sep 16 17:25:04.174790 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:04.174770    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:25:04.175289 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:04.175269    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:25:04.352527 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:04.352496    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-r67pk"
Sep 16 17:25:04.352708 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:04.352660    2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 17:25:04.353053 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:04.353036    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r67pk"
Sep 16 17:25:04.995326 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:04.995303    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:25:04.995439 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:04.995303    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:25:04.995439 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:04.995406    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scrxg" podUID="4b47007d-992e-4660-900c-b4807ce86b3f"
Sep 16 17:25:04.995539 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:04.995303    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:25:04.995539 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:04.995463    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89r27" podUID="c88f1596-e3ef-45ce-b8f6-55fc246bfdab"
Sep 16 17:25:04.995632 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:04.995538    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fw78m" podUID="9d27ec00-06e7-431b-925f-addd45ff0115"
Sep 16 17:25:05.165665 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:05.165636    2562 generic.go:356] "Generic (PLEG): container finished" podID="af87e079-c946-4c98-86fc-4be5a3098d8a" containerID="531b06cf543620af629a635aaaa41db5bf2ab60a7b563dbf0757b144b30f0373" exitCode=0
Sep 16 17:25:05.166368 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:05.165706    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tkzcs" event={"ID":"af87e079-c946-4c98-86fc-4be5a3098d8a","Type":"ContainerDied","Data":"531b06cf543620af629a635aaaa41db5bf2ab60a7b563dbf0757b144b30f0373"}
Sep 16 17:25:05.169704 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:05.169688    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6p26_1d970e5c-29fc-44af-b138-0e60309f2ac0/ovn-acl-logging/0.log"
Sep 16 17:25:05.170019 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:05.169996    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6p26" event={"ID":"1d970e5c-29fc-44af-b138-0e60309f2ac0","Type":"ContainerStarted","Data":"10551e85292e198301f1a02993afbf5288f737b18820a7a27e16a66cef39ec46"}
Sep 16 17:25:05.365165 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:05.365054    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-t6p26" podStartSLOduration=10.834714944 podStartE2EDuration="28.36503s" totalImagesPullingTime="17.530315056s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:05.257882184 +0000 UTC m=+28.846272168" watchObservedRunningTime="2026-09-16 17:25:05.36503 +0000 UTC m=+28.953419982"
Sep 16 17:25:05.365986 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:05.365956    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-scrxg"]
Sep 16 17:25:05.366107 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:05.366051    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:25:05.366143 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:05.366133    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scrxg" podUID="4b47007d-992e-4660-900c-b4807ce86b3f"
Sep 16 17:25:05.375429 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:05.375405    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-89r27"]
Sep 16 17:25:05.375539 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:05.375494    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:25:05.375595 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:05.375579    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89r27" podUID="c88f1596-e3ef-45ce-b8f6-55fc246bfdab"
Sep 16 17:25:05.384048 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:05.384026    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fw78m"]
Sep 16 17:25:05.384146 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:05.384138    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:25:05.384274 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:05.384238    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fw78m" podUID="9d27ec00-06e7-431b-925f-addd45ff0115"
Sep 16 17:25:06.177815 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:06.177587    2562 generic.go:356] "Generic (PLEG): container finished" podID="af87e079-c946-4c98-86fc-4be5a3098d8a" containerID="b068d2f3205e2f89475a07dcf44e321e4141aaf8d73e62a27039c2bedddd6650" exitCode=0
Sep 16 17:25:06.178177 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:06.177672    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tkzcs" event={"ID":"af87e079-c946-4c98-86fc-4be5a3098d8a","Type":"ContainerDied","Data":"b068d2f3205e2f89475a07dcf44e321e4141aaf8d73e62a27039c2bedddd6650"}
Sep 16 17:25:06.996314 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:06.996191    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:25:06.996314 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:06.996279    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:25:06.996314 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:06.996291    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89r27" podUID="c88f1596-e3ef-45ce-b8f6-55fc246bfdab"
Sep 16 17:25:06.996568 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:06.996397    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fw78m" podUID="9d27ec00-06e7-431b-925f-addd45ff0115"
Sep 16 17:25:06.996568 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:06.996431    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:25:06.996568 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:06.996513    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scrxg" podUID="4b47007d-992e-4660-900c-b4807ce86b3f"
Sep 16 17:25:08.999541 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:08.999506    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:25:08.999541 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:08.999542    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:25:08.999985 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:08.999614    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:25:08.999985 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:08.999632    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89r27" podUID="c88f1596-e3ef-45ce-b8f6-55fc246bfdab"
Sep 16 17:25:08.999985 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:08.999704    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scrxg" podUID="4b47007d-992e-4660-900c-b4807ce86b3f"
Sep 16 17:25:08.999985 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:08.999779    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fw78m" podUID="9d27ec00-06e7-431b-925f-addd45ff0115"
Sep 16 17:25:10.523083 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:10.523054    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b47007d-992e-4660-900c-b4807ce86b3f-original-pull-secret\") pod \"global-pull-secret-syncer-scrxg\" (UID: \"4b47007d-992e-4660-900c-b4807ce86b3f\") " pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:25:10.523369 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:10.523178    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:25:10.523369 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:10.523232    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b47007d-992e-4660-900c-b4807ce86b3f-original-pull-secret podName:4b47007d-992e-4660-900c-b4807ce86b3f nodeName:}" failed. No retries permitted until 2026-09-16 17:25:42.523217671 +0000 UTC m=+66.111607634 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b47007d-992e-4660-900c-b4807ce86b3f-original-pull-secret") pod "global-pull-secret-syncer-scrxg" (UID: "4b47007d-992e-4660-900c-b4807ce86b3f") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:25:10.624324 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:10.624243    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs\") pod \"network-metrics-daemon-fw78m\" (UID: \"9d27ec00-06e7-431b-925f-addd45ff0115\") " pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:25:10.624473 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:10.624395    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:25:10.624473 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:10.624461    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs podName:9d27ec00-06e7-431b-925f-addd45ff0115 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:42.624444496 +0000 UTC m=+66.212834457 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs") pod "network-metrics-daemon-fw78m" (UID: "9d27ec00-06e7-431b-925f-addd45ff0115") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:25:10.725248 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:10.725216    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sr5g\" (UniqueName: \"kubernetes.io/projected/c88f1596-e3ef-45ce-b8f6-55fc246bfdab-kube-api-access-7sr5g\") pod \"network-check-target-89r27\" (UID: \"c88f1596-e3ef-45ce-b8f6-55fc246bfdab\") " pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:25:10.725402 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:10.725384    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:25:10.725445 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:10.725408    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:25:10.725445 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:10.725419    2562 projected.go:196] Error preparing data for projected volume kube-api-access-7sr5g for pod openshift-network-diagnostics/network-check-target-89r27: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:25:10.725510 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:10.725470    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c88f1596-e3ef-45ce-b8f6-55fc246bfdab-kube-api-access-7sr5g podName:c88f1596-e3ef-45ce-b8f6-55fc246bfdab nodeName:}" failed. No retries permitted until 2026-09-16 17:25:42.725454142 +0000 UTC m=+66.313844104 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7sr5g" (UniqueName: "kubernetes.io/projected/c88f1596-e3ef-45ce-b8f6-55fc246bfdab-kube-api-access-7sr5g") pod "network-check-target-89r27" (UID: "c88f1596-e3ef-45ce-b8f6-55fc246bfdab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:25:10.996083 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:10.996050    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:25:10.996308 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:10.996057    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:25:10.996308 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:10.996173    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-scrxg" podUID="4b47007d-992e-4660-900c-b4807ce86b3f"
Sep 16 17:25:10.996308 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:10.996057    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:25:10.996308 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:10.996257    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89r27" podUID="c88f1596-e3ef-45ce-b8f6-55fc246bfdab"
Sep 16 17:25:10.996449 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:10.996359    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fw78m" podUID="9d27ec00-06e7-431b-925f-addd45ff0115"
Sep 16 17:25:11.192424 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:11.192395    2562 generic.go:356] "Generic (PLEG): container finished" podID="af87e079-c946-4c98-86fc-4be5a3098d8a" containerID="5ccc6c1a9101d7c002f4a311e34648976b4c7899ca56be6605278cbeafe2521a" exitCode=0
Sep 16 17:25:11.192589 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:11.192449    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tkzcs" event={"ID":"af87e079-c946-4c98-86fc-4be5a3098d8a","Type":"ContainerDied","Data":"5ccc6c1a9101d7c002f4a311e34648976b4c7899ca56be6605278cbeafe2521a"}
Sep 16 17:25:11.713364 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:11.713336    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-160.ec2.internal" event="NodeReady"
Sep 16 17:25:11.713738 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:11.713480    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 17:25:11.834333 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:11.834303    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4lcwz"]
Sep 16 17:25:11.856448 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:11.856372    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2n9xt"]
Sep 16 17:25:11.856611 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:11.856490    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4lcwz"
Sep 16 17:25:11.861832 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:11.861794    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 17:25:11.862365 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:11.862347    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 17:25:11.862474 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:11.862459    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v8tmk\""
Sep 16 17:25:11.876702 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:11.876682    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4lcwz"]
Sep 16 17:25:11.876828 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:11.876771    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2n9xt"
Sep 16 17:25:11.877972 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:11.877954    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2n9xt"]
Sep 16 17:25:11.884217 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:11.884200    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 17:25:11.884413 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:11.884398    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nqgn2\""
Sep 16 17:25:11.884507 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:11.884491    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 17:25:11.884786 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:11.884771    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 17:25:11.934363 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:11.934323    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzxpx\" (UniqueName: \"kubernetes.io/projected/64f7e8df-d87e-426f-acca-39781d05567c-kube-api-access-qzxpx\") pod \"ingress-canary-2n9xt\" (UID: \"64f7e8df-d87e-426f-acca-39781d05567c\") " pod="openshift-ingress-canary/ingress-canary-2n9xt"
Sep 16 17:25:11.934363 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:11.934362    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwzmf\" (UniqueName: \"kubernetes.io/projected/afb12787-82e5-40ce-b4f5-b34655ba032f-kube-api-access-zwzmf\") pod \"dns-default-4lcwz\" (UID: \"afb12787-82e5-40ce-b4f5-b34655ba032f\") " pod="openshift-dns/dns-default-4lcwz"
Sep 16 17:25:11.934546 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:11.934383    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb12787-82e5-40ce-b4f5-b34655ba032f-tmp-dir\") pod \"dns-default-4lcwz\" (UID: \"afb12787-82e5-40ce-b4f5-b34655ba032f\") " pod="openshift-dns/dns-default-4lcwz"
Sep 16 17:25:11.934546 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:11.934412    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afb12787-82e5-40ce-b4f5-b34655ba032f-metrics-tls\") pod \"dns-default-4lcwz\" (UID: \"afb12787-82e5-40ce-b4f5-b34655ba032f\") " pod="openshift-dns/dns-default-4lcwz"
Sep 16 17:25:11.934546 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:11.934480    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/afb12787-82e5-40ce-b4f5-b34655ba032f-config-volume\") pod \"dns-default-4lcwz\" (UID: \"afb12787-82e5-40ce-b4f5-b34655ba032f\") " pod="openshift-dns/dns-default-4lcwz"
Sep 16 17:25:11.934546 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:11.934500    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64f7e8df-d87e-426f-acca-39781d05567c-cert\") pod \"ingress-canary-2n9xt\" (UID: \"64f7e8df-d87e-426f-acca-39781d05567c\") " pod="openshift-ingress-canary/ingress-canary-2n9xt"
Sep 16 17:25:12.035217 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:12.035184    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afb12787-82e5-40ce-b4f5-b34655ba032f-metrics-tls\") pod \"dns-default-4lcwz\" (UID: \"afb12787-82e5-40ce-b4f5-b34655ba032f\") " pod="openshift-dns/dns-default-4lcwz"
Sep 16 17:25:12.035378 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:12.035241    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/afb12787-82e5-40ce-b4f5-b34655ba032f-config-volume\") pod \"dns-default-4lcwz\" (UID: \"afb12787-82e5-40ce-b4f5-b34655ba032f\") " pod="openshift-dns/dns-default-4lcwz"
Sep 16 17:25:12.035378 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:12.035261    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64f7e8df-d87e-426f-acca-39781d05567c-cert\") pod \"ingress-canary-2n9xt\" (UID: \"64f7e8df-d87e-426f-acca-39781d05567c\") " pod="openshift-ingress-canary/ingress-canary-2n9xt"
Sep 16 17:25:12.035378 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:12.035332    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:12.035519 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:12.035391    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb12787-82e5-40ce-b4f5-b34655ba032f-metrics-tls podName:afb12787-82e5-40ce-b4f5-b34655ba032f nodeName:}" failed. No retries permitted until 2026-09-16 17:25:12.535374684 +0000 UTC m=+36.123764647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/afb12787-82e5-40ce-b4f5-b34655ba032f-metrics-tls") pod "dns-default-4lcwz" (UID: "afb12787-82e5-40ce-b4f5-b34655ba032f") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:12.035519 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:12.035429    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzxpx\" (UniqueName: \"kubernetes.io/projected/64f7e8df-d87e-426f-acca-39781d05567c-kube-api-access-qzxpx\") pod \"ingress-canary-2n9xt\" (UID: \"64f7e8df-d87e-426f-acca-39781d05567c\") " pod="openshift-ingress-canary/ingress-canary-2n9xt"
Sep 16 17:25:12.035519 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:12.035465    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwzmf\" (UniqueName: \"kubernetes.io/projected/afb12787-82e5-40ce-b4f5-b34655ba032f-kube-api-access-zwzmf\") pod \"dns-default-4lcwz\" (UID: \"afb12787-82e5-40ce-b4f5-b34655ba032f\") " pod="openshift-dns/dns-default-4lcwz"
Sep 16 17:25:12.035519 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:12.035492    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb12787-82e5-40ce-b4f5-b34655ba032f-tmp-dir\") pod \"dns-default-4lcwz\" (UID: \"afb12787-82e5-40ce-b4f5-b34655ba032f\") " pod="openshift-dns/dns-default-4lcwz"
Sep 16 17:25:12.035762 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:12.035743    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb12787-82e5-40ce-b4f5-b34655ba032f-tmp-dir\") pod \"dns-default-4lcwz\" (UID: \"afb12787-82e5-40ce-b4f5-b34655ba032f\") " pod="openshift-dns/dns-default-4lcwz"
Sep 16 17:25:12.035891 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:12.035873    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/afb12787-82e5-40ce-b4f5-b34655ba032f-config-volume\") pod \"dns-default-4lcwz\" (UID: \"afb12787-82e5-40ce-b4f5-b34655ba032f\") " pod="openshift-dns/dns-default-4lcwz"
Sep 16 17:25:12.039690 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:12.039568    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64f7e8df-d87e-426f-acca-39781d05567c-cert\") pod \"ingress-canary-2n9xt\" (UID: \"64f7e8df-d87e-426f-acca-39781d05567c\") " pod="openshift-ingress-canary/ingress-canary-2n9xt"
Sep 16 17:25:12.059740 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:12.059709    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzxpx\" (UniqueName: \"kubernetes.io/projected/64f7e8df-d87e-426f-acca-39781d05567c-kube-api-access-qzxpx\") pod \"ingress-canary-2n9xt\" (UID: \"64f7e8df-d87e-426f-acca-39781d05567c\") " pod="openshift-ingress-canary/ingress-canary-2n9xt"
Sep 16 17:25:12.064479 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:12.064455    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwzmf\" (UniqueName: \"kubernetes.io/projected/afb12787-82e5-40ce-b4f5-b34655ba032f-kube-api-access-zwzmf\") pod \"dns-default-4lcwz\" (UID: \"afb12787-82e5-40ce-b4f5-b34655ba032f\") " pod="openshift-dns/dns-default-4lcwz"
Sep 16 17:25:12.184976 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:12.184937    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2n9xt"
Sep 16 17:25:12.197894 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:12.197860    2562 generic.go:356] "Generic (PLEG): container finished" podID="af87e079-c946-4c98-86fc-4be5a3098d8a" containerID="10e062ecb2afe08211e85e0858c25b203473a9c3d2247ee6453f2a213fe7c10a" exitCode=0
Sep 16 17:25:12.198020 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:12.197929    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tkzcs" event={"ID":"af87e079-c946-4c98-86fc-4be5a3098d8a","Type":"ContainerDied","Data":"10e062ecb2afe08211e85e0858c25b203473a9c3d2247ee6453f2a213fe7c10a"}
Sep 16 17:25:12.502566 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:12.502490    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2n9xt"]
Sep 16 17:25:12.506110 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:25:12.506085    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f7e8df_d87e_426f_acca_39781d05567c.slice/crio-53d7c8e60f924ed12988f20f0db7e61ddd43fc875515464dfe1af3f47ab97cf9 WatchSource:0}: Error finding container 53d7c8e60f924ed12988f20f0db7e61ddd43fc875515464dfe1af3f47ab97cf9: Status 404 returned error can't find the container with id 53d7c8e60f924ed12988f20f0db7e61ddd43fc875515464dfe1af3f47ab97cf9
Sep 16 17:25:12.539256 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:12.539232    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afb12787-82e5-40ce-b4f5-b34655ba032f-metrics-tls\") pod \"dns-default-4lcwz\" (UID: \"afb12787-82e5-40ce-b4f5-b34655ba032f\") " pod="openshift-dns/dns-default-4lcwz"
Sep 16 17:25:12.539397 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:12.539377    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:12.539452 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:12.539442    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb12787-82e5-40ce-b4f5-b34655ba032f-metrics-tls podName:afb12787-82e5-40ce-b4f5-b34655ba032f nodeName:}" failed. No retries permitted until 2026-09-16 17:25:13.53942367 +0000 UTC m=+37.127813634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/afb12787-82e5-40ce-b4f5-b34655ba032f-metrics-tls") pod "dns-default-4lcwz" (UID: "afb12787-82e5-40ce-b4f5-b34655ba032f") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:12.996272 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:12.996235    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:25:12.997211 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:12.996264    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:25:12.997211 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:12.996235    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:25:13.001980 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:13.001956    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 17:25:13.002247 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:13.002224    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2lh24\""
Sep 16 17:25:13.003283 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:13.003207    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q452j\""
Sep 16 17:25:13.003410 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:13.003379    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 17:25:13.003410 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:13.003394    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 17:25:13.003547 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:13.003436    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 17:25:13.204722 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:13.204689    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tkzcs" event={"ID":"af87e079-c946-4c98-86fc-4be5a3098d8a","Type":"ContainerStarted","Data":"ded324ac472950258bd023e3dac30d1c959bbcd26e4a83be063d85fbd5260508"}
Sep 16 17:25:13.205912 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:13.205887    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2n9xt" event={"ID":"64f7e8df-d87e-426f-acca-39781d05567c","Type":"ContainerStarted","Data":"53d7c8e60f924ed12988f20f0db7e61ddd43fc875515464dfe1af3f47ab97cf9"}
Sep 16 17:25:13.239934 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:13.239884    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tkzcs" podStartSLOduration=7.799213821 podStartE2EDuration="36.239867466s" totalImagesPullingTime="28.440653645s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:13.23925324 +0000 UTC m=+36.827643223" watchObservedRunningTime="2026-09-16 17:25:13.239867466 +0000 UTC m=+36.828257450"
Sep 16 17:25:13.545975 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:13.545938    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afb12787-82e5-40ce-b4f5-b34655ba032f-metrics-tls\") pod \"dns-default-4lcwz\" (UID: \"afb12787-82e5-40ce-b4f5-b34655ba032f\") " pod="openshift-dns/dns-default-4lcwz"
Sep 16 17:25:13.546144 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:13.546060    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:13.546144 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:13.546132    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb12787-82e5-40ce-b4f5-b34655ba032f-metrics-tls podName:afb12787-82e5-40ce-b4f5-b34655ba032f nodeName:}" failed. No retries permitted until 2026-09-16 17:25:15.546113295 +0000 UTC m=+39.134503275 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/afb12787-82e5-40ce-b4f5-b34655ba032f-metrics-tls") pod "dns-default-4lcwz" (UID: "afb12787-82e5-40ce-b4f5-b34655ba032f") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:15.212401 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:15.212189    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2n9xt" event={"ID":"64f7e8df-d87e-426f-acca-39781d05567c","Type":"ContainerStarted","Data":"e8c1542db331d4fd0bab530215d6cff3460925e19438bd2e803ee9373636731b"}
Sep 16 17:25:15.248157 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:15.246396    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2n9xt" podStartSLOduration=1.851389736 podStartE2EDuration="4.246373796s" totalImagesPullingTime="2.39498406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:25:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:15.244292453 +0000 UTC m=+38.832682437" watchObservedRunningTime="2026-09-16 17:25:15.246373796 +0000 UTC m=+38.834763783"
Sep 16 17:25:15.563851 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:15.563734    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afb12787-82e5-40ce-b4f5-b34655ba032f-metrics-tls\") pod \"dns-default-4lcwz\" (UID: \"afb12787-82e5-40ce-b4f5-b34655ba032f\") " pod="openshift-dns/dns-default-4lcwz"
Sep 16 17:25:15.564066 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:15.563898    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:15.564066 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:15.563970    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb12787-82e5-40ce-b4f5-b34655ba032f-metrics-tls podName:afb12787-82e5-40ce-b4f5-b34655ba032f nodeName:}" failed. No retries permitted until 2026-09-16 17:25:19.563950968 +0000 UTC m=+43.152340943 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/afb12787-82e5-40ce-b4f5-b34655ba032f-metrics-tls") pod "dns-default-4lcwz" (UID: "afb12787-82e5-40ce-b4f5-b34655ba032f") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:19.589747 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:19.589711    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afb12787-82e5-40ce-b4f5-b34655ba032f-metrics-tls\") pod \"dns-default-4lcwz\" (UID: \"afb12787-82e5-40ce-b4f5-b34655ba032f\") " pod="openshift-dns/dns-default-4lcwz"
Sep 16 17:25:19.590115 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:19.589872    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:19.590115 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:19.589937    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb12787-82e5-40ce-b4f5-b34655ba032f-metrics-tls podName:afb12787-82e5-40ce-b4f5-b34655ba032f nodeName:}" failed. No retries permitted until 2026-09-16 17:25:27.589922189 +0000 UTC m=+51.178312152 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/afb12787-82e5-40ce-b4f5-b34655ba032f-metrics-tls") pod "dns-default-4lcwz" (UID: "afb12787-82e5-40ce-b4f5-b34655ba032f") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:27.645912 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:27.645870    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afb12787-82e5-40ce-b4f5-b34655ba032f-metrics-tls\") pod \"dns-default-4lcwz\" (UID: \"afb12787-82e5-40ce-b4f5-b34655ba032f\") " pod="openshift-dns/dns-default-4lcwz"
Sep 16 17:25:27.646379 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:27.646023    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:27.646379 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:27.646086    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb12787-82e5-40ce-b4f5-b34655ba032f-metrics-tls podName:afb12787-82e5-40ce-b4f5-b34655ba032f nodeName:}" failed. No retries permitted until 2026-09-16 17:25:43.646070236 +0000 UTC m=+67.234460211 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/afb12787-82e5-40ce-b4f5-b34655ba032f-metrics-tls") pod "dns-default-4lcwz" (UID: "afb12787-82e5-40ce-b4f5-b34655ba032f") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:36.188814 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:36.188777    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6p26"
Sep 16 17:25:42.547857 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:42.547785    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b47007d-992e-4660-900c-b4807ce86b3f-original-pull-secret\") pod \"global-pull-secret-syncer-scrxg\" (UID: \"4b47007d-992e-4660-900c-b4807ce86b3f\") " pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:25:42.551339 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:42.551314    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 17:25:42.561331 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:42.561309    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b47007d-992e-4660-900c-b4807ce86b3f-original-pull-secret\") pod \"global-pull-secret-syncer-scrxg\" (UID: \"4b47007d-992e-4660-900c-b4807ce86b3f\") " pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:25:42.648532 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:42.648482    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs\") pod \"network-metrics-daemon-fw78m\" (UID: \"9d27ec00-06e7-431b-925f-addd45ff0115\") " pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:25:42.651431 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:42.651414    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 17:25:42.658869 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:42.658852    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 17:25:42.658923 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:42.658914    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs podName:9d27ec00-06e7-431b-925f-addd45ff0115 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:46.658898408 +0000 UTC m=+130.247288370 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs") pod "network-metrics-daemon-fw78m" (UID: "9d27ec00-06e7-431b-925f-addd45ff0115") : secret "metrics-daemon-secret" not found
Sep 16 17:25:42.709214 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:42.709186    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-scrxg"
Sep 16 17:25:42.749632 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:42.749595    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sr5g\" (UniqueName: \"kubernetes.io/projected/c88f1596-e3ef-45ce-b8f6-55fc246bfdab-kube-api-access-7sr5g\") pod \"network-check-target-89r27\" (UID: \"c88f1596-e3ef-45ce-b8f6-55fc246bfdab\") " pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:25:42.752709 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:42.752683    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 17:25:42.762776 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:42.762753    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 17:25:42.774448 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:42.774423    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sr5g\" (UniqueName: \"kubernetes.io/projected/c88f1596-e3ef-45ce-b8f6-55fc246bfdab-kube-api-access-7sr5g\") pod \"network-check-target-89r27\" (UID: \"c88f1596-e3ef-45ce-b8f6-55fc246bfdab\") " pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:25:42.979793 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:42.979759    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-scrxg"]
Sep 16 17:25:43.025814 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:43.025786    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2lh24\""
Sep 16 17:25:43.032257 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:43.032240    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:25:43.281438 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:43.281356    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-scrxg" event={"ID":"4b47007d-992e-4660-900c-b4807ce86b3f","Type":"ContainerStarted","Data":"6e4da7ea5866c113b7566096086b146ef5895789cfb0f6ef240f91d9bde24bf0"}
Sep 16 17:25:43.306164 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:43.306137    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-89r27"]
Sep 16 17:25:43.308865 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:25:43.308842    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88f1596_e3ef_45ce_b8f6_55fc246bfdab.slice/crio-6fcc8d4fb290f48e79c4250e708890dff26527dede14e5689cbc8190a6598d38 WatchSource:0}: Error finding container 6fcc8d4fb290f48e79c4250e708890dff26527dede14e5689cbc8190a6598d38: Status 404 returned error can't find the container with id 6fcc8d4fb290f48e79c4250e708890dff26527dede14e5689cbc8190a6598d38
Sep 16 17:25:43.657046 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:43.656968    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afb12787-82e5-40ce-b4f5-b34655ba032f-metrics-tls\") pod \"dns-default-4lcwz\" (UID: \"afb12787-82e5-40ce-b4f5-b34655ba032f\") " pod="openshift-dns/dns-default-4lcwz"
Sep 16 17:25:43.657554 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:43.657135    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:43.657554 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:25:43.657211    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb12787-82e5-40ce-b4f5-b34655ba032f-metrics-tls podName:afb12787-82e5-40ce-b4f5-b34655ba032f nodeName:}" failed. No retries permitted until 2026-09-16 17:26:15.657188842 +0000 UTC m=+99.245578823 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/afb12787-82e5-40ce-b4f5-b34655ba032f-metrics-tls") pod "dns-default-4lcwz" (UID: "afb12787-82e5-40ce-b4f5-b34655ba032f") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:44.286386 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:44.286347    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-89r27" event={"ID":"c88f1596-e3ef-45ce-b8f6-55fc246bfdab","Type":"ContainerStarted","Data":"6fcc8d4fb290f48e79c4250e708890dff26527dede14e5689cbc8190a6598d38"}
Sep 16 17:25:48.298387 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:48.298349    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-scrxg" event={"ID":"4b47007d-992e-4660-900c-b4807ce86b3f","Type":"ContainerStarted","Data":"dbee08bae4c0ec7a24618b71052d20d1d25a3962ebb41ccd6d19022ff2f9e303"}
Sep 16 17:25:48.299790 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:48.299765    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-89r27" event={"ID":"c88f1596-e3ef-45ce-b8f6-55fc246bfdab","Type":"ContainerStarted","Data":"b5e59d66189299cb64a58df626a595ccce4662d490c0b7495acf26560495de38"}
Sep 16 17:25:48.299930 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:48.299883    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:25:48.332942 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:25:48.332883    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-scrxg" podStartSLOduration=66.87946104 podStartE2EDuration="1m11.332866029s" totalImagesPullingTime="4.453404989s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:48.31685584 +0000 UTC m=+71.905245825" watchObservedRunningTime="2026-09-16 17:25:48.332866029 +0000 UTC m=+71.921256013"
Sep 16 17:26:15.673991 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:15.673953    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afb12787-82e5-40ce-b4f5-b34655ba032f-metrics-tls\") pod \"dns-default-4lcwz\" (UID: \"afb12787-82e5-40ce-b4f5-b34655ba032f\") " pod="openshift-dns/dns-default-4lcwz"
Sep 16 17:26:15.674395 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:15.674061    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:26:15.674395 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:15.674111    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afb12787-82e5-40ce-b4f5-b34655ba032f-metrics-tls podName:afb12787-82e5-40ce-b4f5-b34655ba032f nodeName:}" failed. No retries permitted until 2026-09-16 17:27:19.67409755 +0000 UTC m=+163.262487512 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/afb12787-82e5-40ce-b4f5-b34655ba032f-metrics-tls") pod "dns-default-4lcwz" (UID: "afb12787-82e5-40ce-b4f5-b34655ba032f") : secret "dns-default-metrics-tls" not found
Sep 16 17:26:19.303675 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:19.303642    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-89r27"
Sep 16 17:26:19.320348 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:19.320294    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-89r27" podStartSLOduration=98.196109308 podStartE2EDuration="1m42.320280952s" totalImagesPullingTime="4.124171644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:48.332722899 +0000 UTC m=+71.921112885" watchObservedRunningTime="2026-09-16 17:26:19.320280952 +0000 UTC m=+102.908670936"
Sep 16 17:26:46.682558 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:46.682501    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs\") pod \"network-metrics-daemon-fw78m\" (UID: \"9d27ec00-06e7-431b-925f-addd45ff0115\") " pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:26:46.683061 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:46.682650    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 17:26:46.683061 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:46.682731    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs podName:9d27ec00-06e7-431b-925f-addd45ff0115 nodeName:}" failed. No retries permitted until 2026-09-16 17:28:48.682715286 +0000 UTC m=+252.271105263 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs") pod "network-metrics-daemon-fw78m" (UID: "9d27ec00-06e7-431b-925f-addd45ff0115") : secret "metrics-daemon-secret" not found
Sep 16 17:26:48.627285 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.627249    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-fpb8s"]
Sep 16 17:26:48.629224 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.629207    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-fpb8s"
Sep 16 17:26:48.631838 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.631817    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 16 17:26:48.631936 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.631824    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 16 17:26:48.633224 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.633206    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-b79bp\""
Sep 16 17:26:48.643868 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.643845    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-fpb8s"]
Sep 16 17:26:48.653609 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.653562    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-gghkt"]
Sep 16 17:26:48.655480 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.655463    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-gghkt"
Sep 16 17:26:48.658596 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.658577    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 17:26:48.658596 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.658591    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 17:26:48.658702 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.658577    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-z5gkk\""
Sep 16 17:26:48.658976 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.658951    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 17:26:48.659066 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.658985    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 17:26:48.664138 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.664123    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 17:26:48.674485 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.674462    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-gghkt"]
Sep 16 17:26:48.696357 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.696335    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t877c\" (UniqueName: \"kubernetes.io/projected/5a6c8d6d-d244-42c9-bc8a-25f5c01ae41f-kube-api-access-t877c\") pod \"volume-data-source-validator-86695b7cc7-fpb8s\" (UID: \"5a6c8d6d-d244-42c9-bc8a-25f5c01ae41f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-fpb8s"
Sep 16 17:26:48.732505 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.732480    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-czzhc"]
Sep 16 17:26:48.736479 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.736465    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-czzhc"
Sep 16 17:26:48.740280 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.740262    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 16 17:26:48.740498 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.740482    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 16 17:26:48.740795 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.740777    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-v2pxd\""
Sep 16 17:26:48.740900 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.740883    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 16 17:26:48.741176 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.741160    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 16 17:26:48.748199 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.748171    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-czzhc"]
Sep 16 17:26:48.748996 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.748974    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 16 17:26:48.797531 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.797489    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7k6x\" (UniqueName: \"kubernetes.io/projected/a1ff3897-38d1-48bd-be6e-a8a698fcdfee-kube-api-access-n7k6x\") pod \"console-operator-696884fb9c-czzhc\" (UID: \"a1ff3897-38d1-48bd-be6e-a8a698fcdfee\") " pod="openshift-console-operator/console-operator-696884fb9c-czzhc"
Sep 16 17:26:48.797698 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.797559    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a1ff3897-38d1-48bd-be6e-a8a698fcdfee-serving-cert\") pod \"console-operator-696884fb9c-czzhc\" (UID: \"a1ff3897-38d1-48bd-be6e-a8a698fcdfee\") " pod="openshift-console-operator/console-operator-696884fb9c-czzhc"
Sep 16 17:26:48.797698 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.797600    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a1ff3897-38d1-48bd-be6e-a8a698fcdfee-config\") pod \"console-operator-696884fb9c-czzhc\" (UID: \"a1ff3897-38d1-48bd-be6e-a8a698fcdfee\") " pod="openshift-console-operator/console-operator-696884fb9c-czzhc"
Sep 16 17:26:48.797698 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.797626    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a1ff3897-38d1-48bd-be6e-a8a698fcdfee-trusted-ca\") pod \"console-operator-696884fb9c-czzhc\" (UID: \"a1ff3897-38d1-48bd-be6e-a8a698fcdfee\") " pod="openshift-console-operator/console-operator-696884fb9c-czzhc"
Sep 16 17:26:48.797698 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.797664    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad7439f-6922-4550-84fa-5ea0e6ff8081-trusted-ca-bundle\") pod \"insights-operator-5844b856db-gghkt\" (UID: \"6ad7439f-6922-4550-84fa-5ea0e6ff8081\") " pod="openshift-insights/insights-operator-5844b856db-gghkt"
Sep 16 17:26:48.797698 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.797684    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6ad7439f-6922-4550-84fa-5ea0e6ff8081-serving-cert\") pod \"insights-operator-5844b856db-gghkt\" (UID: \"6ad7439f-6922-4550-84fa-5ea0e6ff8081\") " pod="openshift-insights/insights-operator-5844b856db-gghkt"
Sep 16 17:26:48.797698 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.797699    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ad7439f-6922-4550-84fa-5ea0e6ff8081-tmp\") pod \"insights-operator-5844b856db-gghkt\" (UID: \"6ad7439f-6922-4550-84fa-5ea0e6ff8081\") " pod="openshift-insights/insights-operator-5844b856db-gghkt"
Sep 16 17:26:48.797909 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.797713    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6ad7439f-6922-4550-84fa-5ea0e6ff8081-snapshots\") pod \"insights-operator-5844b856db-gghkt\" (UID: \"6ad7439f-6922-4550-84fa-5ea0e6ff8081\") " pod="openshift-insights/insights-operator-5844b856db-gghkt"
Sep 16 17:26:48.797909 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.797734    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnmjk\" (UniqueName: \"kubernetes.io/projected/6ad7439f-6922-4550-84fa-5ea0e6ff8081-kube-api-access-rnmjk\") pod \"insights-operator-5844b856db-gghkt\" (UID: \"6ad7439f-6922-4550-84fa-5ea0e6ff8081\") " pod="openshift-insights/insights-operator-5844b856db-gghkt"
Sep 16 17:26:48.797909 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.797838    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t877c\" (UniqueName: \"kubernetes.io/projected/5a6c8d6d-d244-42c9-bc8a-25f5c01ae41f-kube-api-access-t877c\") pod \"volume-data-source-validator-86695b7cc7-fpb8s\" (UID: \"5a6c8d6d-d244-42c9-bc8a-25f5c01ae41f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-fpb8s"
Sep 16 17:26:48.797909 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.797878    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad7439f-6922-4550-84fa-5ea0e6ff8081-service-ca-bundle\") pod \"insights-operator-5844b856db-gghkt\" (UID: \"6ad7439f-6922-4550-84fa-5ea0e6ff8081\") " pod="openshift-insights/insights-operator-5844b856db-gghkt"
Sep 16 17:26:48.806985 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.806954    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t877c\" (UniqueName: \"kubernetes.io/projected/5a6c8d6d-d244-42c9-bc8a-25f5c01ae41f-kube-api-access-t877c\") pod \"volume-data-source-validator-86695b7cc7-fpb8s\" (UID: \"5a6c8d6d-d244-42c9-bc8a-25f5c01ae41f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-fpb8s"
Sep 16 17:26:48.843420 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.843370    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4nsns"]
Sep 16 17:26:48.845495 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.845472    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5565b7c65b-r4644"]
Sep 16 17:26:48.845603 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.845591    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4nsns"
Sep 16 17:26:48.847168 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.847151    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:48.850394 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.850377    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 16 17:26:48.850764 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.850741    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 16 17:26:48.850764 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.850755    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 17:26:48.850967 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.850745    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 16 17:26:48.850967 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.850755    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-67s5v\""
Sep 16 17:26:48.850967 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.850943    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 17:26:48.851111 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.850995    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 17:26:48.851376 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.851363    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-45pjq\""
Sep 16 17:26:48.857323 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.857304    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 17:26:48.858095 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.858075    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4nsns"]
Sep 16 17:26:48.860983 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.860964    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5565b7c65b-r4644"]
Sep 16 17:26:48.898340 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.898265    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7e97796b-c463-41e3-b7f9-d2a4c75b7211-installation-pull-secrets\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:48.898340 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.898301    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7e97796b-c463-41e3-b7f9-d2a4c75b7211-trusted-ca\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:48.898340 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.898317    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-bound-sa-token\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:48.898340 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.898337    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bld2r\" (UniqueName: \"kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-kube-api-access-bld2r\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:48.898600 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.898408    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a1ff3897-38d1-48bd-be6e-a8a698fcdfee-serving-cert\") pod \"console-operator-696884fb9c-czzhc\" (UID: \"a1ff3897-38d1-48bd-be6e-a8a698fcdfee\") " pod="openshift-console-operator/console-operator-696884fb9c-czzhc"
Sep 16 17:26:48.898600 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.898450    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a1ff3897-38d1-48bd-be6e-a8a698fcdfee-config\") pod \"console-operator-696884fb9c-czzhc\" (UID: \"a1ff3897-38d1-48bd-be6e-a8a698fcdfee\") " pod="openshift-console-operator/console-operator-696884fb9c-czzhc"
Sep 16 17:26:48.898600 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.898468    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a1ff3897-38d1-48bd-be6e-a8a698fcdfee-trusted-ca\") pod \"console-operator-696884fb9c-czzhc\" (UID: \"a1ff3897-38d1-48bd-be6e-a8a698fcdfee\") " pod="openshift-console-operator/console-operator-696884fb9c-czzhc"
Sep 16 17:26:48.898600 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.898486    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad7439f-6922-4550-84fa-5ea0e6ff8081-trusted-ca-bundle\") pod \"insights-operator-5844b856db-gghkt\" (UID: \"6ad7439f-6922-4550-84fa-5ea0e6ff8081\") " pod="openshift-insights/insights-operator-5844b856db-gghkt"
Sep 16 17:26:48.898600 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.898504    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6ad7439f-6922-4550-84fa-5ea0e6ff8081-serving-cert\") pod \"insights-operator-5844b856db-gghkt\" (UID: \"6ad7439f-6922-4550-84fa-5ea0e6ff8081\") " pod="openshift-insights/insights-operator-5844b856db-gghkt"
Sep 16 17:26:48.898600 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.898523    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/036eb2d7-289b-4c7c-8b26-f2697a2460d0-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4nsns\" (UID: \"036eb2d7-289b-4c7c-8b26-f2697a2460d0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4nsns"
Sep 16 17:26:48.898600 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.898540    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5p7g\" (UniqueName: \"kubernetes.io/projected/036eb2d7-289b-4c7c-8b26-f2697a2460d0-kube-api-access-x5p7g\") pod \"cluster-samples-operator-86499c89c4-4nsns\" (UID: \"036eb2d7-289b-4c7c-8b26-f2697a2460d0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4nsns"
Sep 16 17:26:48.898600 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.898559    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-registry-tls\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:48.898600 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.898605    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ad7439f-6922-4550-84fa-5ea0e6ff8081-tmp\") pod \"insights-operator-5844b856db-gghkt\" (UID: \"6ad7439f-6922-4550-84fa-5ea0e6ff8081\") " pod="openshift-insights/insights-operator-5844b856db-gghkt"
Sep 16 17:26:48.899034 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.898628    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6ad7439f-6922-4550-84fa-5ea0e6ff8081-snapshots\") pod \"insights-operator-5844b856db-gghkt\" (UID: \"6ad7439f-6922-4550-84fa-5ea0e6ff8081\") " pod="openshift-insights/insights-operator-5844b856db-gghkt"
Sep 16 17:26:48.899034 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.898647    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnmjk\" (UniqueName: \"kubernetes.io/projected/6ad7439f-6922-4550-84fa-5ea0e6ff8081-kube-api-access-rnmjk\") pod \"insights-operator-5844b856db-gghkt\" (UID: \"6ad7439f-6922-4550-84fa-5ea0e6ff8081\") " pod="openshift-insights/insights-operator-5844b856db-gghkt"
Sep 16 17:26:48.899034 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.898674    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad7439f-6922-4550-84fa-5ea0e6ff8081-service-ca-bundle\") pod \"insights-operator-5844b856db-gghkt\" (UID: \"6ad7439f-6922-4550-84fa-5ea0e6ff8081\") " pod="openshift-insights/insights-operator-5844b856db-gghkt"
Sep 16 17:26:48.899034 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.898705    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7e97796b-c463-41e3-b7f9-d2a4c75b7211-ca-trust-extracted\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:48.899034 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.898736    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7e97796b-c463-41e3-b7f9-d2a4c75b7211-image-registry-private-configuration\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:48.899034 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.898761    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7e97796b-c463-41e3-b7f9-d2a4c75b7211-registry-certificates\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:48.899034 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.898788    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7k6x\" (UniqueName: \"kubernetes.io/projected/a1ff3897-38d1-48bd-be6e-a8a698fcdfee-kube-api-access-n7k6x\") pod \"console-operator-696884fb9c-czzhc\" (UID: \"a1ff3897-38d1-48bd-be6e-a8a698fcdfee\") " pod="openshift-console-operator/console-operator-696884fb9c-czzhc"
Sep 16 17:26:48.899333 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.899256    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a1ff3897-38d1-48bd-be6e-a8a698fcdfee-config\") pod \"console-operator-696884fb9c-czzhc\" (UID: \"a1ff3897-38d1-48bd-be6e-a8a698fcdfee\") " pod="openshift-console-operator/console-operator-696884fb9c-czzhc"
Sep 16 17:26:48.899333 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.899260    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad7439f-6922-4550-84fa-5ea0e6ff8081-service-ca-bundle\") pod \"insights-operator-5844b856db-gghkt\" (UID: \"6ad7439f-6922-4550-84fa-5ea0e6ff8081\") " pod="openshift-insights/insights-operator-5844b856db-gghkt"
Sep 16 17:26:48.899436 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.899382    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6ad7439f-6922-4550-84fa-5ea0e6ff8081-snapshots\") pod \"insights-operator-5844b856db-gghkt\" (UID: \"6ad7439f-6922-4550-84fa-5ea0e6ff8081\") " pod="openshift-insights/insights-operator-5844b856db-gghkt"
Sep 16 17:26:48.899492 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.899466    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ad7439f-6922-4550-84fa-5ea0e6ff8081-tmp\") pod \"insights-operator-5844b856db-gghkt\" (UID: \"6ad7439f-6922-4550-84fa-5ea0e6ff8081\") " pod="openshift-insights/insights-operator-5844b856db-gghkt"
Sep 16 17:26:48.899653 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.899633    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ad7439f-6922-4550-84fa-5ea0e6ff8081-trusted-ca-bundle\") pod \"insights-operator-5844b856db-gghkt\" (UID: \"6ad7439f-6922-4550-84fa-5ea0e6ff8081\") " pod="openshift-insights/insights-operator-5844b856db-gghkt"
Sep 16 17:26:48.899967 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.899944    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a1ff3897-38d1-48bd-be6e-a8a698fcdfee-trusted-ca\") pod \"console-operator-696884fb9c-czzhc\" (UID: \"a1ff3897-38d1-48bd-be6e-a8a698fcdfee\") " pod="openshift-console-operator/console-operator-696884fb9c-czzhc"
Sep 16 17:26:48.901067 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.901047    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a1ff3897-38d1-48bd-be6e-a8a698fcdfee-serving-cert\") pod \"console-operator-696884fb9c-czzhc\" (UID: \"a1ff3897-38d1-48bd-be6e-a8a698fcdfee\") " pod="openshift-console-operator/console-operator-696884fb9c-czzhc"
Sep 16 17:26:48.901433 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.901417    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6ad7439f-6922-4550-84fa-5ea0e6ff8081-serving-cert\") pod \"insights-operator-5844b856db-gghkt\" (UID: \"6ad7439f-6922-4550-84fa-5ea0e6ff8081\") " pod="openshift-insights/insights-operator-5844b856db-gghkt"
Sep 16 17:26:48.906965 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.906940    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7k6x\" (UniqueName: \"kubernetes.io/projected/a1ff3897-38d1-48bd-be6e-a8a698fcdfee-kube-api-access-n7k6x\") pod \"console-operator-696884fb9c-czzhc\" (UID: \"a1ff3897-38d1-48bd-be6e-a8a698fcdfee\") " pod="openshift-console-operator/console-operator-696884fb9c-czzhc"
Sep 16 17:26:48.907329 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.907310    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnmjk\" (UniqueName: \"kubernetes.io/projected/6ad7439f-6922-4550-84fa-5ea0e6ff8081-kube-api-access-rnmjk\") pod \"insights-operator-5844b856db-gghkt\" (UID: \"6ad7439f-6922-4550-84fa-5ea0e6ff8081\") " pod="openshift-insights/insights-operator-5844b856db-gghkt"
Sep 16 17:26:48.938122 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.938094    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-fpb8s"
Sep 16 17:26:48.963774 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.963746    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-gghkt"
Sep 16 17:26:48.999454 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.999425    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bld2r\" (UniqueName: \"kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-kube-api-access-bld2r\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:48.999585 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.999499    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/036eb2d7-289b-4c7c-8b26-f2697a2460d0-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4nsns\" (UID: \"036eb2d7-289b-4c7c-8b26-f2697a2460d0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4nsns"
Sep 16 17:26:48.999585 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.999528    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5p7g\" (UniqueName: \"kubernetes.io/projected/036eb2d7-289b-4c7c-8b26-f2697a2460d0-kube-api-access-x5p7g\") pod \"cluster-samples-operator-86499c89c4-4nsns\" (UID: \"036eb2d7-289b-4c7c-8b26-f2697a2460d0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4nsns"
Sep 16 17:26:48.999585 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.999552    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-registry-tls\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:48.999740 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.999588    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7e97796b-c463-41e3-b7f9-d2a4c75b7211-ca-trust-extracted\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:48.999740 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.999617    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7e97796b-c463-41e3-b7f9-d2a4c75b7211-image-registry-private-configuration\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:48.999740 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.999642    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7e97796b-c463-41e3-b7f9-d2a4c75b7211-registry-certificates\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:48.999740 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.999686    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7e97796b-c463-41e3-b7f9-d2a4c75b7211-installation-pull-secrets\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:48.999740 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.999731    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7e97796b-c463-41e3-b7f9-d2a4c75b7211-trusted-ca\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:48.999994 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:48.999754    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-bound-sa-token\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:49.000229 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:49.000200    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7e97796b-c463-41e3-b7f9-d2a4c75b7211-ca-trust-extracted\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:49.000330 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:49.000312    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 17:26:49.000390 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:49.000376    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/036eb2d7-289b-4c7c-8b26-f2697a2460d0-samples-operator-tls podName:036eb2d7-289b-4c7c-8b26-f2697a2460d0 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:49.500356752 +0000 UTC m=+133.088746738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/036eb2d7-289b-4c7c-8b26-f2697a2460d0-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-4nsns" (UID: "036eb2d7-289b-4c7c-8b26-f2697a2460d0") : secret "samples-operator-tls" not found
Sep 16 17:26:49.000877 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:49.000843    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 17:26:49.000969 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:49.000879    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5565b7c65b-r4644: secret "image-registry-tls" not found
Sep 16 17:26:49.000969 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:49.000947    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-registry-tls podName:7e97796b-c463-41e3-b7f9-d2a4c75b7211 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:49.50093153 +0000 UTC m=+133.089321511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-registry-tls") pod "image-registry-5565b7c65b-r4644" (UID: "7e97796b-c463-41e3-b7f9-d2a4c75b7211") : secret "image-registry-tls" not found
Sep 16 17:26:49.001616 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:49.001565    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7e97796b-c463-41e3-b7f9-d2a4c75b7211-registry-certificates\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:49.002290 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:49.002251    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7e97796b-c463-41e3-b7f9-d2a4c75b7211-trusted-ca\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:49.005838 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:49.005744    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7e97796b-c463-41e3-b7f9-d2a4c75b7211-image-registry-private-configuration\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:49.006145 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:49.006126    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7e97796b-c463-41e3-b7f9-d2a4c75b7211-installation-pull-secrets\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:49.011893 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:49.011868    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5p7g\" (UniqueName: \"kubernetes.io/projected/036eb2d7-289b-4c7c-8b26-f2697a2460d0-kube-api-access-x5p7g\") pod \"cluster-samples-operator-86499c89c4-4nsns\" (UID: \"036eb2d7-289b-4c7c-8b26-f2697a2460d0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4nsns"
Sep 16 17:26:49.012325 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:49.012303    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-bound-sa-token\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:49.013380 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:49.013360    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bld2r\" (UniqueName: \"kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-kube-api-access-bld2r\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:49.046125 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:49.046101    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-czzhc"
Sep 16 17:26:49.209600 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:49.209571    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-fpb8s"]
Sep 16 17:26:49.212581 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:26:49.212558    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a6c8d6d_d244_42c9_bc8a_25f5c01ae41f.slice/crio-fa84f492f75514c494d47df442c0524dab9447d5238b878ba7066b381b18f096 WatchSource:0}: Error finding container fa84f492f75514c494d47df442c0524dab9447d5238b878ba7066b381b18f096: Status 404 returned error can't find the container with id fa84f492f75514c494d47df442c0524dab9447d5238b878ba7066b381b18f096
Sep 16 17:26:49.240089 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:49.240061    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-gghkt"]
Sep 16 17:26:49.242737 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:26:49.242705    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad7439f_6922_4550_84fa_5ea0e6ff8081.slice/crio-1ab2966da091e5fce28e86bded2e3d8ea91946b01ef4c210799e76e75a7af93b WatchSource:0}: Error finding container 1ab2966da091e5fce28e86bded2e3d8ea91946b01ef4c210799e76e75a7af93b: Status 404 returned error can't find the container with id 1ab2966da091e5fce28e86bded2e3d8ea91946b01ef4c210799e76e75a7af93b
Sep 16 17:26:49.309097 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:49.309067    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-czzhc"]
Sep 16 17:26:49.313039 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:26:49.313016    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ff3897_38d1_48bd_be6e_a8a698fcdfee.slice/crio-aeff22b957e5b53b6500a82fa9bd8c0e2f78f1ed943e68019acf54c0d00af1ff WatchSource:0}: Error finding container aeff22b957e5b53b6500a82fa9bd8c0e2f78f1ed943e68019acf54c0d00af1ff: Status 404 returned error can't find the container with id aeff22b957e5b53b6500a82fa9bd8c0e2f78f1ed943e68019acf54c0d00af1ff
Sep 16 17:26:49.445825 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:49.445778    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-czzhc" event={"ID":"a1ff3897-38d1-48bd-be6e-a8a698fcdfee","Type":"ContainerStarted","Data":"aeff22b957e5b53b6500a82fa9bd8c0e2f78f1ed943e68019acf54c0d00af1ff"}
Sep 16 17:26:49.446865 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:49.446840    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-fpb8s" event={"ID":"5a6c8d6d-d244-42c9-bc8a-25f5c01ae41f","Type":"ContainerStarted","Data":"fa84f492f75514c494d47df442c0524dab9447d5238b878ba7066b381b18f096"}
Sep 16 17:26:49.447823 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:49.447794    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-gghkt" event={"ID":"6ad7439f-6922-4550-84fa-5ea0e6ff8081","Type":"ContainerStarted","Data":"1ab2966da091e5fce28e86bded2e3d8ea91946b01ef4c210799e76e75a7af93b"}
Sep 16 17:26:49.502534 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:49.502472    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/036eb2d7-289b-4c7c-8b26-f2697a2460d0-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4nsns\" (UID: \"036eb2d7-289b-4c7c-8b26-f2697a2460d0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4nsns"
Sep 16 17:26:49.502534 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:49.502501    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-registry-tls\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:49.502697 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:49.502612    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 17:26:49.502697 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:49.502676    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/036eb2d7-289b-4c7c-8b26-f2697a2460d0-samples-operator-tls podName:036eb2d7-289b-4c7c-8b26-f2697a2460d0 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:50.502658368 +0000 UTC m=+134.091048350 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/036eb2d7-289b-4c7c-8b26-f2697a2460d0-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-4nsns" (UID: "036eb2d7-289b-4c7c-8b26-f2697a2460d0") : secret "samples-operator-tls" not found
Sep 16 17:26:49.502697 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:49.502617    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 17:26:49.502790 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:49.502707    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5565b7c65b-r4644: secret "image-registry-tls" not found
Sep 16 17:26:49.502790 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:49.502751    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-registry-tls podName:7e97796b-c463-41e3-b7f9-d2a4c75b7211 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:50.50273963 +0000 UTC m=+134.091129592 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-registry-tls") pod "image-registry-5565b7c65b-r4644" (UID: "7e97796b-c463-41e3-b7f9-d2a4c75b7211") : secret "image-registry-tls" not found
Sep 16 17:26:50.510710 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:50.510671    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/036eb2d7-289b-4c7c-8b26-f2697a2460d0-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4nsns\" (UID: \"036eb2d7-289b-4c7c-8b26-f2697a2460d0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4nsns"
Sep 16 17:26:50.511168 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:50.510717    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-registry-tls\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:50.511168 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:50.510900    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 17:26:50.511168 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:50.510976    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/036eb2d7-289b-4c7c-8b26-f2697a2460d0-samples-operator-tls podName:036eb2d7-289b-4c7c-8b26-f2697a2460d0 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:52.510956655 +0000 UTC m=+136.099346621 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/036eb2d7-289b-4c7c-8b26-f2697a2460d0-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-4nsns" (UID: "036eb2d7-289b-4c7c-8b26-f2697a2460d0") : secret "samples-operator-tls" not found
Sep 16 17:26:50.511168 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:50.510908    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 17:26:50.511168 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:50.511010    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5565b7c65b-r4644: secret "image-registry-tls" not found
Sep 16 17:26:50.511168 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:50.511075    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-registry-tls podName:7e97796b-c463-41e3-b7f9-d2a4c75b7211 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:52.511058555 +0000 UTC m=+136.099448532 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-registry-tls") pod "image-registry-5565b7c65b-r4644" (UID: "7e97796b-c463-41e3-b7f9-d2a4c75b7211") : secret "image-registry-tls" not found
Sep 16 17:26:51.455523 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:51.455482    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-fpb8s" event={"ID":"5a6c8d6d-d244-42c9-bc8a-25f5c01ae41f","Type":"ContainerStarted","Data":"9a6108d49623447b062aa91a77a1cb9cc75c892db12867479f4f8a2856df99ed"}
Sep 16 17:26:52.460156 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:52.460125    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-gghkt" event={"ID":"6ad7439f-6922-4550-84fa-5ea0e6ff8081","Type":"ContainerStarted","Data":"800048d6cd9a241cf75c8363f05b3bad39c769414ba7274511bc741e06a77bcb"}
Sep 16 17:26:52.461738 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:52.461717    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-czzhc_a1ff3897-38d1-48bd-be6e-a8a698fcdfee/console-operator/0.log"
Sep 16 17:26:52.461860 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:52.461757    2562 generic.go:356] "Generic (PLEG): container finished" podID="a1ff3897-38d1-48bd-be6e-a8a698fcdfee" containerID="b41228c67800b3924dd0bf9b5c15932c105f9fa900cd30af16d19d10ed5ec66a" exitCode=255
Sep 16 17:26:52.461860 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:52.461838    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-czzhc" event={"ID":"a1ff3897-38d1-48bd-be6e-a8a698fcdfee","Type":"ContainerDied","Data":"b41228c67800b3924dd0bf9b5c15932c105f9fa900cd30af16d19d10ed5ec66a"}
Sep 16 17:26:52.462037 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:52.462024    2562 scope.go:117] "RemoveContainer" containerID="b41228c67800b3924dd0bf9b5c15932c105f9fa900cd30af16d19d10ed5ec66a"
Sep 16 17:26:52.478845 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:52.478781    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-fpb8s" podStartSLOduration=2.980064762 podStartE2EDuration="4.478763497s" totalImagesPullingTime="1.498698735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:26:51.470953915 +0000 UTC m=+135.059343901" watchObservedRunningTime="2026-09-16 17:26:52.478763497 +0000 UTC m=+136.067153481"
Sep 16 17:26:52.479331 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:52.479291    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-gghkt" podStartSLOduration=1.968800452 podStartE2EDuration="4.479283286s" totalImagesPullingTime="2.510482834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:26:52.477473512 +0000 UTC m=+136.065863491" watchObservedRunningTime="2026-09-16 17:26:52.479283286 +0000 UTC m=+136.067673270"
Sep 16 17:26:52.526342 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:52.526308    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/036eb2d7-289b-4c7c-8b26-f2697a2460d0-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4nsns\" (UID: \"036eb2d7-289b-4c7c-8b26-f2697a2460d0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4nsns"
Sep 16 17:26:52.526475 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:52.526348    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-registry-tls\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:52.526815 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:52.526783    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 17:26:52.526915 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:52.526827    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5565b7c65b-r4644: secret "image-registry-tls" not found
Sep 16 17:26:52.526915 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:52.526866    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 17:26:52.527019 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:52.526919    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-registry-tls podName:7e97796b-c463-41e3-b7f9-d2a4c75b7211 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:56.526899718 +0000 UTC m=+140.115289683 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-registry-tls") pod "image-registry-5565b7c65b-r4644" (UID: "7e97796b-c463-41e3-b7f9-d2a4c75b7211") : secret "image-registry-tls" not found
Sep 16 17:26:52.527019 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:52.526939    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/036eb2d7-289b-4c7c-8b26-f2697a2460d0-samples-operator-tls podName:036eb2d7-289b-4c7c-8b26-f2697a2460d0 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:56.526929439 +0000 UTC m=+140.115319405 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/036eb2d7-289b-4c7c-8b26-f2697a2460d0-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-4nsns" (UID: "036eb2d7-289b-4c7c-8b26-f2697a2460d0") : secret "samples-operator-tls" not found
Sep 16 17:26:53.190841 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.190789    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-stbrb"]
Sep 16 17:26:53.194283 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.194257    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:26:53.196884 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.196864    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 17:26:53.196985 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.196881    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 17:26:53.196985 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.196928    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mfnhp\""
Sep 16 17:26:53.207050 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.207030    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-stbrb"]
Sep 16 17:26:53.337671 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.334936    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2ff86f78-bedb-4545-8cdf-9eea531efcf9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-stbrb\" (UID: \"2ff86f78-bedb-4545-8cdf-9eea531efcf9\") " pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:26:53.337671 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.335055    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2ff86f78-bedb-4545-8cdf-9eea531efcf9-crio-socket\") pod \"insights-runtime-extractor-stbrb\" (UID: \"2ff86f78-bedb-4545-8cdf-9eea531efcf9\") " pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:26:53.337671 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.335133    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ff86f78-bedb-4545-8cdf-9eea531efcf9-data-volume\") pod \"insights-runtime-extractor-stbrb\" (UID: \"2ff86f78-bedb-4545-8cdf-9eea531efcf9\") " pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:26:53.337671 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.335165    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2ff86f78-bedb-4545-8cdf-9eea531efcf9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-stbrb\" (UID: \"2ff86f78-bedb-4545-8cdf-9eea531efcf9\") " pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:26:53.337671 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.335218    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rqld\" (UniqueName: \"kubernetes.io/projected/2ff86f78-bedb-4545-8cdf-9eea531efcf9-kube-api-access-5rqld\") pod \"insights-runtime-extractor-stbrb\" (UID: \"2ff86f78-bedb-4545-8cdf-9eea531efcf9\") " pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:26:53.436397 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.436356    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2ff86f78-bedb-4545-8cdf-9eea531efcf9-crio-socket\") pod \"insights-runtime-extractor-stbrb\" (UID: \"2ff86f78-bedb-4545-8cdf-9eea531efcf9\") " pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:26:53.436572 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.436416    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ff86f78-bedb-4545-8cdf-9eea531efcf9-data-volume\") pod \"insights-runtime-extractor-stbrb\" (UID: \"2ff86f78-bedb-4545-8cdf-9eea531efcf9\") " pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:26:53.436572 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.436453    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2ff86f78-bedb-4545-8cdf-9eea531efcf9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-stbrb\" (UID: \"2ff86f78-bedb-4545-8cdf-9eea531efcf9\") " pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:26:53.436572 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.436494    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rqld\" (UniqueName: \"kubernetes.io/projected/2ff86f78-bedb-4545-8cdf-9eea531efcf9-kube-api-access-5rqld\") pod \"insights-runtime-extractor-stbrb\" (UID: \"2ff86f78-bedb-4545-8cdf-9eea531efcf9\") " pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:26:53.436572 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.436495    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2ff86f78-bedb-4545-8cdf-9eea531efcf9-crio-socket\") pod \"insights-runtime-extractor-stbrb\" (UID: \"2ff86f78-bedb-4545-8cdf-9eea531efcf9\") " pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:26:53.436572 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.436564    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2ff86f78-bedb-4545-8cdf-9eea531efcf9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-stbrb\" (UID: \"2ff86f78-bedb-4545-8cdf-9eea531efcf9\") " pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:26:53.436768 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:53.436597    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 17:26:53.436768 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:53.436649    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:26:53.436768 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:53.436690    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2ff86f78-bedb-4545-8cdf-9eea531efcf9-kube-rbac-proxy-cm podName:2ff86f78-bedb-4545-8cdf-9eea531efcf9 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:53.936667857 +0000 UTC m=+137.525057822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2ff86f78-bedb-4545-8cdf-9eea531efcf9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-stbrb" (UID: "2ff86f78-bedb-4545-8cdf-9eea531efcf9") : configmap "kube-rbac-proxy" not found
Sep 16 17:26:53.436768 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:53.436711    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ff86f78-bedb-4545-8cdf-9eea531efcf9-insights-runtime-extractor-tls podName:2ff86f78-bedb-4545-8cdf-9eea531efcf9 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:53.93670064 +0000 UTC m=+137.525090605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2ff86f78-bedb-4545-8cdf-9eea531efcf9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-stbrb" (UID: "2ff86f78-bedb-4545-8cdf-9eea531efcf9") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:26:53.436947 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.436777    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ff86f78-bedb-4545-8cdf-9eea531efcf9-data-volume\") pod \"insights-runtime-extractor-stbrb\" (UID: \"2ff86f78-bedb-4545-8cdf-9eea531efcf9\") " pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:26:53.445241 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.445189    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rqld\" (UniqueName: \"kubernetes.io/projected/2ff86f78-bedb-4545-8cdf-9eea531efcf9-kube-api-access-5rqld\") pod \"insights-runtime-extractor-stbrb\" (UID: \"2ff86f78-bedb-4545-8cdf-9eea531efcf9\") " pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:26:53.466999 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.466977    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-czzhc_a1ff3897-38d1-48bd-be6e-a8a698fcdfee/console-operator/1.log"
Sep 16 17:26:53.467396 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.467382    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-czzhc_a1ff3897-38d1-48bd-be6e-a8a698fcdfee/console-operator/0.log"
Sep 16 17:26:53.467436 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.467414    2562 generic.go:356] "Generic (PLEG): container finished" podID="a1ff3897-38d1-48bd-be6e-a8a698fcdfee" containerID="064c6cc067d3716e16ca62303e4878432855ced1cfd28b6cc8c3ebbf08fc9e48" exitCode=255
Sep 16 17:26:53.467502 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.467488    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-czzhc" event={"ID":"a1ff3897-38d1-48bd-be6e-a8a698fcdfee","Type":"ContainerDied","Data":"064c6cc067d3716e16ca62303e4878432855ced1cfd28b6cc8c3ebbf08fc9e48"}
Sep 16 17:26:53.467568 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.467557    2562 scope.go:117] "RemoveContainer" containerID="b41228c67800b3924dd0bf9b5c15932c105f9fa900cd30af16d19d10ed5ec66a"
Sep 16 17:26:53.467787 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.467765    2562 scope.go:117] "RemoveContainer" containerID="064c6cc067d3716e16ca62303e4878432855ced1cfd28b6cc8c3ebbf08fc9e48"
Sep 16 17:26:53.467997 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:53.467977    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-czzhc_openshift-console-operator(a1ff3897-38d1-48bd-be6e-a8a698fcdfee)\"" pod="openshift-console-operator/console-operator-696884fb9c-czzhc" podUID="a1ff3897-38d1-48bd-be6e-a8a698fcdfee"
Sep 16 17:26:53.633462 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.633431    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-ckczd"]
Sep 16 17:26:53.636465 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.636450    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ckczd"
Sep 16 17:26:53.638915 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.638889    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 16 17:26:53.639059 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.638961    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 16 17:26:53.639059 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.638978    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-r9shc\""
Sep 16 17:26:53.643769 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.643745    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-ckczd"]
Sep 16 17:26:53.739638 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.739566    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d2b67354-93c8-419c-87f0-6d817d914033-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-ckczd\" (UID: \"d2b67354-93c8-419c-87f0-6d817d914033\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ckczd"
Sep 16 17:26:53.739638 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.739620    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d2b67354-93c8-419c-87f0-6d817d914033-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-ckczd\" (UID: \"d2b67354-93c8-419c-87f0-6d817d914033\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ckczd"
Sep 16 17:26:53.840892 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.840850    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d2b67354-93c8-419c-87f0-6d817d914033-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-ckczd\" (UID: \"d2b67354-93c8-419c-87f0-6d817d914033\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ckczd"
Sep 16 17:26:53.841056 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.840928    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d2b67354-93c8-419c-87f0-6d817d914033-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-ckczd\" (UID: \"d2b67354-93c8-419c-87f0-6d817d914033\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ckczd"
Sep 16 17:26:53.841056 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:53.841007    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 17:26:53.841122 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:53.841076    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2b67354-93c8-419c-87f0-6d817d914033-networking-console-plugin-cert podName:d2b67354-93c8-419c-87f0-6d817d914033 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:54.341056496 +0000 UTC m=+137.929446473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d2b67354-93c8-419c-87f0-6d817d914033-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-ckczd" (UID: "d2b67354-93c8-419c-87f0-6d817d914033") : secret "networking-console-plugin-cert" not found
Sep 16 17:26:53.841590 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.841571    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d2b67354-93c8-419c-87f0-6d817d914033-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-ckczd\" (UID: \"d2b67354-93c8-419c-87f0-6d817d914033\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ckczd"
Sep 16 17:26:53.941928 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.941895    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2ff86f78-bedb-4545-8cdf-9eea531efcf9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-stbrb\" (UID: \"2ff86f78-bedb-4545-8cdf-9eea531efcf9\") " pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:26:53.942108 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:53.941964    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2ff86f78-bedb-4545-8cdf-9eea531efcf9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-stbrb\" (UID: \"2ff86f78-bedb-4545-8cdf-9eea531efcf9\") " pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:26:53.942108 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:53.942028    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 17:26:53.942108 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:53.942054    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:26:53.942108 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:53.942073    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2ff86f78-bedb-4545-8cdf-9eea531efcf9-kube-rbac-proxy-cm podName:2ff86f78-bedb-4545-8cdf-9eea531efcf9 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:54.942061554 +0000 UTC m=+138.530451530 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2ff86f78-bedb-4545-8cdf-9eea531efcf9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-stbrb" (UID: "2ff86f78-bedb-4545-8cdf-9eea531efcf9") : configmap "kube-rbac-proxy" not found
Sep 16 17:26:53.942261 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:53.942112    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ff86f78-bedb-4545-8cdf-9eea531efcf9-insights-runtime-extractor-tls podName:2ff86f78-bedb-4545-8cdf-9eea531efcf9 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:54.942095092 +0000 UTC m=+138.530485071 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2ff86f78-bedb-4545-8cdf-9eea531efcf9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-stbrb" (UID: "2ff86f78-bedb-4545-8cdf-9eea531efcf9") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:26:54.346389 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:54.346355    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d2b67354-93c8-419c-87f0-6d817d914033-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-ckczd\" (UID: \"d2b67354-93c8-419c-87f0-6d817d914033\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ckczd"
Sep 16 17:26:54.346576 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:54.346511    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 17:26:54.346627 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:54.346577    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2b67354-93c8-419c-87f0-6d817d914033-networking-console-plugin-cert podName:d2b67354-93c8-419c-87f0-6d817d914033 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:55.346561811 +0000 UTC m=+138.934951791 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d2b67354-93c8-419c-87f0-6d817d914033-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-ckczd" (UID: "d2b67354-93c8-419c-87f0-6d817d914033") : secret "networking-console-plugin-cert" not found
Sep 16 17:26:54.472369 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:54.472345    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-czzhc_a1ff3897-38d1-48bd-be6e-a8a698fcdfee/console-operator/1.log"
Sep 16 17:26:54.472747 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:54.472657    2562 scope.go:117] "RemoveContainer" containerID="064c6cc067d3716e16ca62303e4878432855ced1cfd28b6cc8c3ebbf08fc9e48"
Sep 16 17:26:54.472831 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:54.472795    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-czzhc_openshift-console-operator(a1ff3897-38d1-48bd-be6e-a8a698fcdfee)\"" pod="openshift-console-operator/console-operator-696884fb9c-czzhc" podUID="a1ff3897-38d1-48bd-be6e-a8a698fcdfee"
Sep 16 17:26:54.914729 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:54.914705    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2r4vj_79c16240-59ee-4b5e-98c9-86ef04179efa/dns-node-resolver/0.log"
Sep 16 17:26:54.953197 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:54.953165    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2ff86f78-bedb-4545-8cdf-9eea531efcf9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-stbrb\" (UID: \"2ff86f78-bedb-4545-8cdf-9eea531efcf9\") " pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:26:54.953331 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:54.953264    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2ff86f78-bedb-4545-8cdf-9eea531efcf9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-stbrb\" (UID: \"2ff86f78-bedb-4545-8cdf-9eea531efcf9\") " pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:26:54.953331 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:54.953320    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:26:54.953445 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:54.953339    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 17:26:54.953445 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:54.953383    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2ff86f78-bedb-4545-8cdf-9eea531efcf9-kube-rbac-proxy-cm podName:2ff86f78-bedb-4545-8cdf-9eea531efcf9 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:56.953366508 +0000 UTC m=+140.541756485 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2ff86f78-bedb-4545-8cdf-9eea531efcf9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-stbrb" (UID: "2ff86f78-bedb-4545-8cdf-9eea531efcf9") : configmap "kube-rbac-proxy" not found
Sep 16 17:26:54.953445 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:54.953398    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ff86f78-bedb-4545-8cdf-9eea531efcf9-insights-runtime-extractor-tls podName:2ff86f78-bedb-4545-8cdf-9eea531efcf9 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:56.95339011 +0000 UTC m=+140.541780071 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2ff86f78-bedb-4545-8cdf-9eea531efcf9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-stbrb" (UID: "2ff86f78-bedb-4545-8cdf-9eea531efcf9") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:26:55.355706 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:55.355627    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d2b67354-93c8-419c-87f0-6d817d914033-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-ckczd\" (UID: \"d2b67354-93c8-419c-87f0-6d817d914033\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ckczd"
Sep 16 17:26:55.355858 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:55.355781    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 17:26:55.355908 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:55.355858    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2b67354-93c8-419c-87f0-6d817d914033-networking-console-plugin-cert podName:d2b67354-93c8-419c-87f0-6d817d914033 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:57.355844583 +0000 UTC m=+140.944234549 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d2b67354-93c8-419c-87f0-6d817d914033-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-ckczd" (UID: "d2b67354-93c8-419c-87f0-6d817d914033") : secret "networking-console-plugin-cert" not found
Sep 16 17:26:55.515900 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:55.515865    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-vnd84"]
Sep 16 17:26:55.517827 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:55.517796    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-vnd84"
Sep 16 17:26:55.520385 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:55.520351    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 16 17:26:55.520517 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:55.520474    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 16 17:26:55.521559 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:55.521536    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 16 17:26:55.521683 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:55.521539    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-gpf5b\""
Sep 16 17:26:55.521683 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:55.521543    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 16 17:26:55.529053 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:55.529034    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-vnd84"]
Sep 16 17:26:55.658401 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:55.658311    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5ccd3a30-84b6-4058-95a8-e1e18ce7d50d-signing-key\") pod \"service-ca-568869fdc5-vnd84\" (UID: \"5ccd3a30-84b6-4058-95a8-e1e18ce7d50d\") " pod="openshift-service-ca/service-ca-568869fdc5-vnd84"
Sep 16 17:26:55.658401 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:55.658386    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5ccd3a30-84b6-4058-95a8-e1e18ce7d50d-signing-cabundle\") pod \"service-ca-568869fdc5-vnd84\" (UID: \"5ccd3a30-84b6-4058-95a8-e1e18ce7d50d\") " pod="openshift-service-ca/service-ca-568869fdc5-vnd84"
Sep 16 17:26:55.658584 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:55.658439    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtffl\" (UniqueName: \"kubernetes.io/projected/5ccd3a30-84b6-4058-95a8-e1e18ce7d50d-kube-api-access-gtffl\") pod \"service-ca-568869fdc5-vnd84\" (UID: \"5ccd3a30-84b6-4058-95a8-e1e18ce7d50d\") " pod="openshift-service-ca/service-ca-568869fdc5-vnd84"
Sep 16 17:26:55.759269 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:55.759230    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5ccd3a30-84b6-4058-95a8-e1e18ce7d50d-signing-key\") pod \"service-ca-568869fdc5-vnd84\" (UID: \"5ccd3a30-84b6-4058-95a8-e1e18ce7d50d\") " pod="openshift-service-ca/service-ca-568869fdc5-vnd84"
Sep 16 17:26:55.759425 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:55.759294    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5ccd3a30-84b6-4058-95a8-e1e18ce7d50d-signing-cabundle\") pod \"service-ca-568869fdc5-vnd84\" (UID: \"5ccd3a30-84b6-4058-95a8-e1e18ce7d50d\") " pod="openshift-service-ca/service-ca-568869fdc5-vnd84"
Sep 16 17:26:55.759425 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:55.759328    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtffl\" (UniqueName: \"kubernetes.io/projected/5ccd3a30-84b6-4058-95a8-e1e18ce7d50d-kube-api-access-gtffl\") pod \"service-ca-568869fdc5-vnd84\" (UID: \"5ccd3a30-84b6-4058-95a8-e1e18ce7d50d\") " pod="openshift-service-ca/service-ca-568869fdc5-vnd84"
Sep 16 17:26:55.760001 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:55.759967    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5ccd3a30-84b6-4058-95a8-e1e18ce7d50d-signing-cabundle\") pod \"service-ca-568869fdc5-vnd84\" (UID: \"5ccd3a30-84b6-4058-95a8-e1e18ce7d50d\") " pod="openshift-service-ca/service-ca-568869fdc5-vnd84"
Sep 16 17:26:55.761617 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:55.761594    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5ccd3a30-84b6-4058-95a8-e1e18ce7d50d-signing-key\") pod \"service-ca-568869fdc5-vnd84\" (UID: \"5ccd3a30-84b6-4058-95a8-e1e18ce7d50d\") " pod="openshift-service-ca/service-ca-568869fdc5-vnd84"
Sep 16 17:26:55.767507 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:55.767482    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtffl\" (UniqueName: \"kubernetes.io/projected/5ccd3a30-84b6-4058-95a8-e1e18ce7d50d-kube-api-access-gtffl\") pod \"service-ca-568869fdc5-vnd84\" (UID: \"5ccd3a30-84b6-4058-95a8-e1e18ce7d50d\") " pod="openshift-service-ca/service-ca-568869fdc5-vnd84"
Sep 16 17:26:55.826274 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:55.826239    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-vnd84"
Sep 16 17:26:56.094283 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:56.094243    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-vnd84"]
Sep 16 17:26:56.096903 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:26:56.096876    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ccd3a30_84b6_4058_95a8_e1e18ce7d50d.slice/crio-76bb5c9423e8499cc41df4dd7058f6ed3c1d512b4a3c80674842488fc80b223e WatchSource:0}: Error finding container 76bb5c9423e8499cc41df4dd7058f6ed3c1d512b4a3c80674842488fc80b223e: Status 404 returned error can't find the container with id 76bb5c9423e8499cc41df4dd7058f6ed3c1d512b4a3c80674842488fc80b223e
Sep 16 17:26:56.115069 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:56.115047    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ddqwd_f689aa90-24ff-48a5-8e66-f2e5d749dbd4/node-ca/0.log"
Sep 16 17:26:56.479163 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:56.479124    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-vnd84" event={"ID":"5ccd3a30-84b6-4058-95a8-e1e18ce7d50d","Type":"ContainerStarted","Data":"76bb5c9423e8499cc41df4dd7058f6ed3c1d512b4a3c80674842488fc80b223e"}
Sep 16 17:26:56.566558 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:56.566522    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/036eb2d7-289b-4c7c-8b26-f2697a2460d0-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4nsns\" (UID: \"036eb2d7-289b-4c7c-8b26-f2697a2460d0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4nsns"
Sep 16 17:26:56.566947 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:56.566674    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-registry-tls\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:26:56.566947 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:56.566678    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 17:26:56.566947 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:56.566754    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/036eb2d7-289b-4c7c-8b26-f2697a2460d0-samples-operator-tls podName:036eb2d7-289b-4c7c-8b26-f2697a2460d0 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:04.566738236 +0000 UTC m=+148.155128197 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/036eb2d7-289b-4c7c-8b26-f2697a2460d0-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-4nsns" (UID: "036eb2d7-289b-4c7c-8b26-f2697a2460d0") : secret "samples-operator-tls" not found
Sep 16 17:26:56.566947 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:56.566759    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 17:26:56.566947 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:56.566769    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5565b7c65b-r4644: secret "image-registry-tls" not found
Sep 16 17:26:56.566947 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:56.566828    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-registry-tls podName:7e97796b-c463-41e3-b7f9-d2a4c75b7211 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:04.566816647 +0000 UTC m=+148.155206622 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-registry-tls") pod "image-registry-5565b7c65b-r4644" (UID: "7e97796b-c463-41e3-b7f9-d2a4c75b7211") : secret "image-registry-tls" not found
Sep 16 17:26:56.715256 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:56.715204    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2n9xt_64f7e8df-d87e-426f-acca-39781d05567c/serve-healthcheck-canary/0.log"
Sep 16 17:26:56.970080 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:56.970039    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2ff86f78-bedb-4545-8cdf-9eea531efcf9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-stbrb\" (UID: \"2ff86f78-bedb-4545-8cdf-9eea531efcf9\") " pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:26:56.970262 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:56.970106    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2ff86f78-bedb-4545-8cdf-9eea531efcf9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-stbrb\" (UID: \"2ff86f78-bedb-4545-8cdf-9eea531efcf9\") " pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:26:56.970262 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:56.970186    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 17:26:56.970262 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:56.970230    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:26:56.970382 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:56.970267    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2ff86f78-bedb-4545-8cdf-9eea531efcf9-kube-rbac-proxy-cm podName:2ff86f78-bedb-4545-8cdf-9eea531efcf9 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:00.970245224 +0000 UTC m=+144.558635201 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2ff86f78-bedb-4545-8cdf-9eea531efcf9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-stbrb" (UID: "2ff86f78-bedb-4545-8cdf-9eea531efcf9") : configmap "kube-rbac-proxy" not found
Sep 16 17:26:56.970382 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:56.970286    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ff86f78-bedb-4545-8cdf-9eea531efcf9-insights-runtime-extractor-tls podName:2ff86f78-bedb-4545-8cdf-9eea531efcf9 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:00.970276193 +0000 UTC m=+144.558666158 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2ff86f78-bedb-4545-8cdf-9eea531efcf9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-stbrb" (UID: "2ff86f78-bedb-4545-8cdf-9eea531efcf9") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:26:57.374892 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:57.374791    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d2b67354-93c8-419c-87f0-6d817d914033-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-ckczd\" (UID: \"d2b67354-93c8-419c-87f0-6d817d914033\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ckczd"
Sep 16 17:26:57.375066 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:57.374957    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 17:26:57.375066 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:57.375047    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2b67354-93c8-419c-87f0-6d817d914033-networking-console-plugin-cert podName:d2b67354-93c8-419c-87f0-6d817d914033 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:01.375027932 +0000 UTC m=+144.963417897 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d2b67354-93c8-419c-87f0-6d817d914033-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-ckczd" (UID: "d2b67354-93c8-419c-87f0-6d817d914033") : secret "networking-console-plugin-cert" not found
Sep 16 17:26:58.488046 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:58.488011    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-vnd84" event={"ID":"5ccd3a30-84b6-4058-95a8-e1e18ce7d50d","Type":"ContainerStarted","Data":"fdfa5391b91584a49a8a151aaaf492645d8b758835093edca8026356225b9f8f"}
Sep 16 17:26:58.507220 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:58.507174    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-vnd84" podStartSLOduration=1.500961982 podStartE2EDuration="3.507158075s" totalImagesPullingTime="2.006196093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:26:58.506900643 +0000 UTC m=+142.095290629" watchObservedRunningTime="2026-09-16 17:26:58.507158075 +0000 UTC m=+142.095548058"
Sep 16 17:26:59.047182 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:59.047143    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-czzhc"
Sep 16 17:26:59.047351 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:59.047201    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-czzhc"
Sep 16 17:26:59.047539 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:26:59.047526    2562 scope.go:117] "RemoveContainer" containerID="064c6cc067d3716e16ca62303e4878432855ced1cfd28b6cc8c3ebbf08fc9e48"
Sep 16 17:26:59.047691 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:26:59.047673    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-czzhc_openshift-console-operator(a1ff3897-38d1-48bd-be6e-a8a698fcdfee)\"" pod="openshift-console-operator/console-operator-696884fb9c-czzhc" podUID="a1ff3897-38d1-48bd-be6e-a8a698fcdfee"
Sep 16 17:27:01.008115 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:01.008081    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2ff86f78-bedb-4545-8cdf-9eea531efcf9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-stbrb\" (UID: \"2ff86f78-bedb-4545-8cdf-9eea531efcf9\") " pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:27:01.008508 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:01.008161    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2ff86f78-bedb-4545-8cdf-9eea531efcf9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-stbrb\" (UID: \"2ff86f78-bedb-4545-8cdf-9eea531efcf9\") " pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:27:01.008508 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:27:01.008222    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:01.008508 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:27:01.008251    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 17:27:01.008508 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:27:01.008284    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ff86f78-bedb-4545-8cdf-9eea531efcf9-insights-runtime-extractor-tls podName:2ff86f78-bedb-4545-8cdf-9eea531efcf9 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:09.008268706 +0000 UTC m=+152.596658683 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2ff86f78-bedb-4545-8cdf-9eea531efcf9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-stbrb" (UID: "2ff86f78-bedb-4545-8cdf-9eea531efcf9") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:01.008508 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:27:01.008298    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2ff86f78-bedb-4545-8cdf-9eea531efcf9-kube-rbac-proxy-cm podName:2ff86f78-bedb-4545-8cdf-9eea531efcf9 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:09.008291589 +0000 UTC m=+152.596681550 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2ff86f78-bedb-4545-8cdf-9eea531efcf9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-stbrb" (UID: "2ff86f78-bedb-4545-8cdf-9eea531efcf9") : configmap "kube-rbac-proxy" not found
Sep 16 17:27:01.412533 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:01.412435    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d2b67354-93c8-419c-87f0-6d817d914033-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-ckczd\" (UID: \"d2b67354-93c8-419c-87f0-6d817d914033\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ckczd"
Sep 16 17:27:01.412662 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:27:01.412579    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 17:27:01.412662 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:27:01.412645    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2b67354-93c8-419c-87f0-6d817d914033-networking-console-plugin-cert podName:d2b67354-93c8-419c-87f0-6d817d914033 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:09.412630098 +0000 UTC m=+153.001020064 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d2b67354-93c8-419c-87f0-6d817d914033-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-ckczd" (UID: "d2b67354-93c8-419c-87f0-6d817d914033") : secret "networking-console-plugin-cert" not found
Sep 16 17:27:04.641726 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:04.641686    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-registry-tls\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:27:04.642227 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:04.641751    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/036eb2d7-289b-4c7c-8b26-f2697a2460d0-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4nsns\" (UID: \"036eb2d7-289b-4c7c-8b26-f2697a2460d0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4nsns"
Sep 16 17:27:04.644171 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:04.644138    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/036eb2d7-289b-4c7c-8b26-f2697a2460d0-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4nsns\" (UID: \"036eb2d7-289b-4c7c-8b26-f2697a2460d0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4nsns"
Sep 16 17:27:04.644286 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:04.644229    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-registry-tls\") pod \"image-registry-5565b7c65b-r4644\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") " pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:27:04.756493 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:04.756454    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4nsns"
Sep 16 17:27:04.762458 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:04.762434    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:27:05.031493 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:05.031457    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4nsns"]
Sep 16 17:27:05.038481 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:05.038162    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5565b7c65b-r4644"]
Sep 16 17:27:05.049900 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:27:05.049868    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e97796b_c463_41e3_b7f9_d2a4c75b7211.slice/crio-9fc769786d6b855027df4cdc3d682febc0cbbb65ae0076294084a7be06dc4958 WatchSource:0}: Error finding container 9fc769786d6b855027df4cdc3d682febc0cbbb65ae0076294084a7be06dc4958: Status 404 returned error can't find the container with id 9fc769786d6b855027df4cdc3d682febc0cbbb65ae0076294084a7be06dc4958
Sep 16 17:27:05.512112 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:05.512080    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5565b7c65b-r4644" event={"ID":"7e97796b-c463-41e3-b7f9-d2a4c75b7211","Type":"ContainerStarted","Data":"4ced4e8174970be6c4d91bd232c10ca6d8449e2754cb2de387496d8e67d13618"}
Sep 16 17:27:05.512278 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:05.512170    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5565b7c65b-r4644" event={"ID":"7e97796b-c463-41e3-b7f9-d2a4c75b7211","Type":"ContainerStarted","Data":"9fc769786d6b855027df4cdc3d682febc0cbbb65ae0076294084a7be06dc4958"}
Sep 16 17:27:05.512278 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:05.512193    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:27:05.513285 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:05.513266    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4nsns" event={"ID":"036eb2d7-289b-4c7c-8b26-f2697a2460d0","Type":"ContainerStarted","Data":"6ce854fd449da0dcf13b204257e21d1c47228628ded091624e1ac2c4479d34ea"}
Sep 16 17:27:05.535654 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:05.535613    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5565b7c65b-r4644" podStartSLOduration=17.535601448 podStartE2EDuration="17.535601448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:05.534254398 +0000 UTC m=+149.122644383" watchObservedRunningTime="2026-09-16 17:27:05.535601448 +0000 UTC m=+149.123991432"
Sep 16 17:27:07.522176 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:07.522139    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4nsns" event={"ID":"036eb2d7-289b-4c7c-8b26-f2697a2460d0","Type":"ContainerStarted","Data":"3c192b123a406647033383cb0810674ce070a0de4f6a403031d2efd16a8632b2"}
Sep 16 17:27:07.522614 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:07.522201    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4nsns" event={"ID":"036eb2d7-289b-4c7c-8b26-f2697a2460d0","Type":"ContainerStarted","Data":"4221780cd485d49c332c51a7c3ac5cd2b78e9e6e00553cb403dede7b3bbfc71c"}
Sep 16 17:27:07.543782 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:07.543720    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4nsns" podStartSLOduration=17.467170642 podStartE2EDuration="19.543701867s" totalImagesPullingTime="2.076531225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:07.542962826 +0000 UTC m=+151.131352821" watchObservedRunningTime="2026-09-16 17:27:07.543701867 +0000 UTC m=+151.132091852"
Sep 16 17:27:09.079485 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:09.079436    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2ff86f78-bedb-4545-8cdf-9eea531efcf9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-stbrb\" (UID: \"2ff86f78-bedb-4545-8cdf-9eea531efcf9\") " pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:27:09.079904 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:09.079510    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2ff86f78-bedb-4545-8cdf-9eea531efcf9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-stbrb\" (UID: \"2ff86f78-bedb-4545-8cdf-9eea531efcf9\") " pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:27:09.079904 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:27:09.079654    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:09.079904 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:27:09.079721    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ff86f78-bedb-4545-8cdf-9eea531efcf9-insights-runtime-extractor-tls podName:2ff86f78-bedb-4545-8cdf-9eea531efcf9 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:25.079701748 +0000 UTC m=+168.668091710 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2ff86f78-bedb-4545-8cdf-9eea531efcf9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-stbrb" (UID: "2ff86f78-bedb-4545-8cdf-9eea531efcf9") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:09.080060 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:09.080043    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2ff86f78-bedb-4545-8cdf-9eea531efcf9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-stbrb\" (UID: \"2ff86f78-bedb-4545-8cdf-9eea531efcf9\") " pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:27:09.482537 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:09.482501    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d2b67354-93c8-419c-87f0-6d817d914033-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-ckczd\" (UID: \"d2b67354-93c8-419c-87f0-6d817d914033\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ckczd"
Sep 16 17:27:09.484940 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:09.484916    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d2b67354-93c8-419c-87f0-6d817d914033-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-ckczd\" (UID: \"d2b67354-93c8-419c-87f0-6d817d914033\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ckczd"
Sep 16 17:27:09.545920 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:09.545888    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ckczd"
Sep 16 17:27:09.819118 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:09.819094    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-ckczd"]
Sep 16 17:27:09.821248 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:27:09.821213    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b67354_93c8_419c_87f0_6d817d914033.slice/crio-c0ff528cc8c76815323f5e0a33cda71c48878d295adda985bc1354fff79aefb7 WatchSource:0}: Error finding container c0ff528cc8c76815323f5e0a33cda71c48878d295adda985bc1354fff79aefb7: Status 404 returned error can't find the container with id c0ff528cc8c76815323f5e0a33cda71c48878d295adda985bc1354fff79aefb7
Sep 16 17:27:10.532547 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:10.532509    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ckczd" event={"ID":"d2b67354-93c8-419c-87f0-6d817d914033","Type":"ContainerStarted","Data":"c0ff528cc8c76815323f5e0a33cda71c48878d295adda985bc1354fff79aefb7"}
Sep 16 17:27:11.537270 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:11.537236    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ckczd" event={"ID":"d2b67354-93c8-419c-87f0-6d817d914033","Type":"ContainerStarted","Data":"08f13e13aa0ec1bf2c43586f89f9c35dbb8ef599330db04ce01263a5f1e71ede"}
Sep 16 17:27:11.556385 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:11.556334    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ckczd" podStartSLOduration=17.383635923 podStartE2EDuration="18.556321516s" totalImagesPullingTime="1.172685593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:11.554607291 +0000 UTC m=+155.142997272" watchObservedRunningTime="2026-09-16 17:27:11.556321516 +0000 UTC m=+155.144711501"
Sep 16 17:27:13.996096 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:13.996056    2562 scope.go:117] "RemoveContainer" containerID="064c6cc067d3716e16ca62303e4878432855ced1cfd28b6cc8c3ebbf08fc9e48"
Sep 16 17:27:14.549481 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:14.549455    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-czzhc_a1ff3897-38d1-48bd-be6e-a8a698fcdfee/console-operator/1.log"
Sep 16 17:27:14.549621 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:14.549517    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-czzhc" event={"ID":"a1ff3897-38d1-48bd-be6e-a8a698fcdfee","Type":"ContainerStarted","Data":"bab125b18e6fc464b560403b165ee8736805ce4e471f064afdc95f653133bea4"}
Sep 16 17:27:14.549824 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:14.549790    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-czzhc"
Sep 16 17:27:14.569774 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:14.569726    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-czzhc" podStartSLOduration=24.130126708 podStartE2EDuration="26.569709861s" totalImagesPullingTime="2.439583153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:14.567957936 +0000 UTC m=+158.156347931" watchObservedRunningTime="2026-09-16 17:27:14.569709861 +0000 UTC m=+158.158099859"
Sep 16 17:27:14.633926 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:14.633895    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-czzhc"
Sep 16 17:27:14.866383 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:27:14.866281    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-4lcwz" podUID="afb12787-82e5-40ce-b4f5-b34655ba032f"
Sep 16 17:27:15.552178 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:15.552145    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4lcwz"
Sep 16 17:27:16.015592 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:27:16.015543    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-fw78m" podUID="9d27ec00-06e7-431b-925f-addd45ff0115"
Sep 16 17:27:17.641323 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.641294    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5565b7c65b-r4644"]
Sep 16 17:27:17.684304 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.684269    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-868c457c6b-pgc4m"]
Sep 16 17:27:17.689436 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.689414    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:17.707446 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.707425    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-868c457c6b-pgc4m"]
Sep 16 17:27:17.751469 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.751443    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc3dea1b-13cf-444b-bb84-42d022358828-registry-tls\") pod \"image-registry-868c457c6b-pgc4m\" (UID: \"dc3dea1b-13cf-444b-bb84-42d022358828\") " pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:17.751576 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.751472    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25fpf\" (UniqueName: \"kubernetes.io/projected/dc3dea1b-13cf-444b-bb84-42d022358828-kube-api-access-25fpf\") pod \"image-registry-868c457c6b-pgc4m\" (UID: \"dc3dea1b-13cf-444b-bb84-42d022358828\") " pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:17.751576 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.751496    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dc3dea1b-13cf-444b-bb84-42d022358828-ca-trust-extracted\") pod \"image-registry-868c457c6b-pgc4m\" (UID: \"dc3dea1b-13cf-444b-bb84-42d022358828\") " pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:17.751576 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.751520    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dc3dea1b-13cf-444b-bb84-42d022358828-bound-sa-token\") pod \"image-registry-868c457c6b-pgc4m\" (UID: \"dc3dea1b-13cf-444b-bb84-42d022358828\") " pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:17.751693 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.751588    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dc3dea1b-13cf-444b-bb84-42d022358828-image-registry-private-configuration\") pod \"image-registry-868c457c6b-pgc4m\" (UID: \"dc3dea1b-13cf-444b-bb84-42d022358828\") " pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:17.751693 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.751632    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dc3dea1b-13cf-444b-bb84-42d022358828-installation-pull-secrets\") pod \"image-registry-868c457c6b-pgc4m\" (UID: \"dc3dea1b-13cf-444b-bb84-42d022358828\") " pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:17.751786 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.751692    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dc3dea1b-13cf-444b-bb84-42d022358828-registry-certificates\") pod \"image-registry-868c457c6b-pgc4m\" (UID: \"dc3dea1b-13cf-444b-bb84-42d022358828\") " pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:17.751786 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.751747    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc3dea1b-13cf-444b-bb84-42d022358828-trusted-ca\") pod \"image-registry-868c457c6b-pgc4m\" (UID: \"dc3dea1b-13cf-444b-bb84-42d022358828\") " pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:17.852477 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.852456    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dc3dea1b-13cf-444b-bb84-42d022358828-registry-certificates\") pod \"image-registry-868c457c6b-pgc4m\" (UID: \"dc3dea1b-13cf-444b-bb84-42d022358828\") " pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:17.852589 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.852485    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc3dea1b-13cf-444b-bb84-42d022358828-trusted-ca\") pod \"image-registry-868c457c6b-pgc4m\" (UID: \"dc3dea1b-13cf-444b-bb84-42d022358828\") " pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:17.852589 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.852509    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc3dea1b-13cf-444b-bb84-42d022358828-registry-tls\") pod \"image-registry-868c457c6b-pgc4m\" (UID: \"dc3dea1b-13cf-444b-bb84-42d022358828\") " pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:17.852589 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.852534    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25fpf\" (UniqueName: \"kubernetes.io/projected/dc3dea1b-13cf-444b-bb84-42d022358828-kube-api-access-25fpf\") pod \"image-registry-868c457c6b-pgc4m\" (UID: \"dc3dea1b-13cf-444b-bb84-42d022358828\") " pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:17.852589 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.852560    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dc3dea1b-13cf-444b-bb84-42d022358828-ca-trust-extracted\") pod \"image-registry-868c457c6b-pgc4m\" (UID: \"dc3dea1b-13cf-444b-bb84-42d022358828\") " pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:17.852783 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.852593    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dc3dea1b-13cf-444b-bb84-42d022358828-bound-sa-token\") pod \"image-registry-868c457c6b-pgc4m\" (UID: \"dc3dea1b-13cf-444b-bb84-42d022358828\") " pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:17.852783 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.852627    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dc3dea1b-13cf-444b-bb84-42d022358828-image-registry-private-configuration\") pod \"image-registry-868c457c6b-pgc4m\" (UID: \"dc3dea1b-13cf-444b-bb84-42d022358828\") " pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:17.852783 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.852655    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dc3dea1b-13cf-444b-bb84-42d022358828-installation-pull-secrets\") pod \"image-registry-868c457c6b-pgc4m\" (UID: \"dc3dea1b-13cf-444b-bb84-42d022358828\") " pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:17.853556 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.853525    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dc3dea1b-13cf-444b-bb84-42d022358828-ca-trust-extracted\") pod \"image-registry-868c457c6b-pgc4m\" (UID: \"dc3dea1b-13cf-444b-bb84-42d022358828\") " pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:17.853988 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.853964    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dc3dea1b-13cf-444b-bb84-42d022358828-registry-certificates\") pod \"image-registry-868c457c6b-pgc4m\" (UID: \"dc3dea1b-13cf-444b-bb84-42d022358828\") " pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:17.854358 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.854335    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc3dea1b-13cf-444b-bb84-42d022358828-trusted-ca\") pod \"image-registry-868c457c6b-pgc4m\" (UID: \"dc3dea1b-13cf-444b-bb84-42d022358828\") " pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:17.858473 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.856103    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dc3dea1b-13cf-444b-bb84-42d022358828-installation-pull-secrets\") pod \"image-registry-868c457c6b-pgc4m\" (UID: \"dc3dea1b-13cf-444b-bb84-42d022358828\") " pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:17.858473 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.857586    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dc3dea1b-13cf-444b-bb84-42d022358828-image-registry-private-configuration\") pod \"image-registry-868c457c6b-pgc4m\" (UID: \"dc3dea1b-13cf-444b-bb84-42d022358828\") " pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:17.859149 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.859131    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc3dea1b-13cf-444b-bb84-42d022358828-registry-tls\") pod \"image-registry-868c457c6b-pgc4m\" (UID: \"dc3dea1b-13cf-444b-bb84-42d022358828\") " pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:17.870512 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.870493    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25fpf\" (UniqueName: \"kubernetes.io/projected/dc3dea1b-13cf-444b-bb84-42d022358828-kube-api-access-25fpf\") pod \"image-registry-868c457c6b-pgc4m\" (UID: \"dc3dea1b-13cf-444b-bb84-42d022358828\") " pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:17.870911 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.870895    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dc3dea1b-13cf-444b-bb84-42d022358828-bound-sa-token\") pod \"image-registry-868c457c6b-pgc4m\" (UID: \"dc3dea1b-13cf-444b-bb84-42d022358828\") " pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:17.998122 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:17.998090    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:18.273749 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:18.273677    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-868c457c6b-pgc4m"]
Sep 16 17:27:18.276940 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:27:18.276916    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3dea1b_13cf_444b_bb84_42d022358828.slice/crio-7250f9e46fc5ed85edd3d4ec8a0ce1f8ba786934f4ed3d8c0edcee00649726b5 WatchSource:0}: Error finding container 7250f9e46fc5ed85edd3d4ec8a0ce1f8ba786934f4ed3d8c0edcee00649726b5: Status 404 returned error can't find the container with id 7250f9e46fc5ed85edd3d4ec8a0ce1f8ba786934f4ed3d8c0edcee00649726b5
Sep 16 17:27:18.563964 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:18.563878    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-868c457c6b-pgc4m" event={"ID":"dc3dea1b-13cf-444b-bb84-42d022358828","Type":"ContainerStarted","Data":"0085d0eacc900d71e3839e574617a06c6d8d877b14429426a9e60d7c0e7f84de"}
Sep 16 17:27:18.563964 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:18.563934    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-868c457c6b-pgc4m" event={"ID":"dc3dea1b-13cf-444b-bb84-42d022358828","Type":"ContainerStarted","Data":"7250f9e46fc5ed85edd3d4ec8a0ce1f8ba786934f4ed3d8c0edcee00649726b5"}
Sep 16 17:27:18.564142 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:18.563972    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:18.585903 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:18.585852    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-868c457c6b-pgc4m" podStartSLOduration=1.5858367979999999 podStartE2EDuration="1.585836798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:18.584980401 +0000 UTC m=+162.173370421" watchObservedRunningTime="2026-09-16 17:27:18.585836798 +0000 UTC m=+162.174226782"
Sep 16 17:27:19.768031 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:19.767994    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afb12787-82e5-40ce-b4f5-b34655ba032f-metrics-tls\") pod \"dns-default-4lcwz\" (UID: \"afb12787-82e5-40ce-b4f5-b34655ba032f\") " pod="openshift-dns/dns-default-4lcwz"
Sep 16 17:27:19.770279 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:19.770258    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afb12787-82e5-40ce-b4f5-b34655ba032f-metrics-tls\") pod \"dns-default-4lcwz\" (UID: \"afb12787-82e5-40ce-b4f5-b34655ba032f\") " pod="openshift-dns/dns-default-4lcwz"
Sep 16 17:27:20.055709 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:20.055635    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v8tmk\""
Sep 16 17:27:20.063891 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:20.063871    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4lcwz"
Sep 16 17:27:20.367305 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:20.367242    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4lcwz"]
Sep 16 17:27:20.370092 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:27:20.370065    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb12787_82e5_40ce_b4f5_b34655ba032f.slice/crio-47cbcf7efe3cc14e14547e7a90a599a190f0e3908d6b3ea1fd438c07c78fcd45 WatchSource:0}: Error finding container 47cbcf7efe3cc14e14547e7a90a599a190f0e3908d6b3ea1fd438c07c78fcd45: Status 404 returned error can't find the container with id 47cbcf7efe3cc14e14547e7a90a599a190f0e3908d6b3ea1fd438c07c78fcd45
Sep 16 17:27:20.572329 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:20.572299    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4lcwz" event={"ID":"afb12787-82e5-40ce-b4f5-b34655ba032f","Type":"ContainerStarted","Data":"47cbcf7efe3cc14e14547e7a90a599a190f0e3908d6b3ea1fd438c07c78fcd45"}
Sep 16 17:27:22.581012 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:22.580987    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4lcwz" event={"ID":"afb12787-82e5-40ce-b4f5-b34655ba032f","Type":"ContainerStarted","Data":"7e55c4d410e1039ffb2a349419fe6ca3d92538905ab72dae06dbeca16a4050c6"}
Sep 16 17:27:23.585950 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:23.585913    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4lcwz" event={"ID":"afb12787-82e5-40ce-b4f5-b34655ba032f","Type":"ContainerStarted","Data":"28569764809c430739bdc980ee6ab521bdc9af5e8390854f40fd9ccc49e56de3"}
Sep 16 17:27:23.586432 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:23.586074    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4lcwz"
Sep 16 17:27:23.606310 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:23.606264    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4lcwz" podStartSLOduration=130.632834183 podStartE2EDuration="2m12.606249865s" totalImagesPullingTime="1.973415682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:25:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:23.604928893 +0000 UTC m=+167.193318877" watchObservedRunningTime="2026-09-16 17:27:23.606249865 +0000 UTC m=+167.194639849"
Sep 16 17:27:25.108711 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:25.108670    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2ff86f78-bedb-4545-8cdf-9eea531efcf9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-stbrb\" (UID: \"2ff86f78-bedb-4545-8cdf-9eea531efcf9\") " pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:27:25.111074 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:25.111048    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2ff86f78-bedb-4545-8cdf-9eea531efcf9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-stbrb\" (UID: \"2ff86f78-bedb-4545-8cdf-9eea531efcf9\") " pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:27:25.303354 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:25.303322    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-stbrb"
Sep 16 17:27:25.573377 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:25.573329    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-stbrb"]
Sep 16 17:27:25.577755 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:27:25.577724    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff86f78_bedb_4545_8cdf_9eea531efcf9.slice/crio-7d7483c6883c53036f13d7c563d571e680af04073b4777166386e7d687f3c80d WatchSource:0}: Error finding container 7d7483c6883c53036f13d7c563d571e680af04073b4777166386e7d687f3c80d: Status 404 returned error can't find the container with id 7d7483c6883c53036f13d7c563d571e680af04073b4777166386e7d687f3c80d
Sep 16 17:27:25.593355 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:25.593316    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-stbrb" event={"ID":"2ff86f78-bedb-4545-8cdf-9eea531efcf9","Type":"ContainerStarted","Data":"7d7483c6883c53036f13d7c563d571e680af04073b4777166386e7d687f3c80d"}
Sep 16 17:27:26.598386 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:26.598310    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-stbrb" event={"ID":"2ff86f78-bedb-4545-8cdf-9eea531efcf9","Type":"ContainerStarted","Data":"ef875fba8af309ad8bd86dec69c7175218f2c051f203553979a9e768a8e1aba3"}
Sep 16 17:27:26.598386 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:26.598363    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-stbrb" event={"ID":"2ff86f78-bedb-4545-8cdf-9eea531efcf9","Type":"ContainerStarted","Data":"be345643508f9e389fc72084a2b1f0d4089c922188ddf6b4d1a0973af0312939"}
Sep 16 17:27:27.193636 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:27.193606    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-clv5f"]
Sep 16 17:27:27.195819 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:27.195779    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-clv5f"
Sep 16 17:27:27.203150 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:27.202949    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 17:27:27.203393 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:27.203372    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-2grzh\""
Sep 16 17:27:27.204661 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:27.204544    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 17:27:27.204661 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:27.204562    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 17:27:27.204661 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:27.204578    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 17:27:27.204661 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:27.204637    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 17:27:27.213428 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:27.213405    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-clv5f"]
Sep 16 17:27:27.224771 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:27.224746    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c0e1c10-67d7-4804-a977-829534bf6416-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-clv5f\" (UID: \"3c0e1c10-67d7-4804-a977-829534bf6416\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-clv5f"
Sep 16 17:27:27.224893 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:27.224793    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c0e1c10-67d7-4804-a977-829534bf6416-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-clv5f\" (UID: \"3c0e1c10-67d7-4804-a977-829534bf6416\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-clv5f"
Sep 16 17:27:27.224893 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:27.224864    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrl4x\" (UniqueName: \"kubernetes.io/projected/3c0e1c10-67d7-4804-a977-829534bf6416-kube-api-access-xrl4x\") pod \"prometheus-operator-68bcd7f987-clv5f\" (UID: \"3c0e1c10-67d7-4804-a977-829534bf6416\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-clv5f"
Sep 16 17:27:27.225003 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:27.224897    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c0e1c10-67d7-4804-a977-829534bf6416-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-clv5f\" (UID: \"3c0e1c10-67d7-4804-a977-829534bf6416\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-clv5f"
Sep 16 17:27:27.325686 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:27.325645    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c0e1c10-67d7-4804-a977-829534bf6416-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-clv5f\" (UID: \"3c0e1c10-67d7-4804-a977-829534bf6416\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-clv5f"
Sep 16 17:27:27.326131 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:27.325706    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c0e1c10-67d7-4804-a977-829534bf6416-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-clv5f\" (UID: \"3c0e1c10-67d7-4804-a977-829534bf6416\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-clv5f"
Sep 16 17:27:27.326131 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:27.325738    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrl4x\" (UniqueName: \"kubernetes.io/projected/3c0e1c10-67d7-4804-a977-829534bf6416-kube-api-access-xrl4x\") pod \"prometheus-operator-68bcd7f987-clv5f\" (UID: \"3c0e1c10-67d7-4804-a977-829534bf6416\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-clv5f"
Sep 16 17:27:27.326131 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:27.325777    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c0e1c10-67d7-4804-a977-829534bf6416-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-clv5f\" (UID: \"3c0e1c10-67d7-4804-a977-829534bf6416\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-clv5f"
Sep 16 17:27:27.326131 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:27:27.325837    2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 16 17:27:27.326131 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:27:27.325916    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c0e1c10-67d7-4804-a977-829534bf6416-prometheus-operator-tls podName:3c0e1c10-67d7-4804-a977-829534bf6416 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:27.825895009 +0000 UTC m=+171.414284977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/3c0e1c10-67d7-4804-a977-829534bf6416-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-clv5f" (UID: "3c0e1c10-67d7-4804-a977-829534bf6416") : secret "prometheus-operator-tls" not found
Sep 16 17:27:27.326550 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:27.326530    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c0e1c10-67d7-4804-a977-829534bf6416-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-clv5f\" (UID: \"3c0e1c10-67d7-4804-a977-829534bf6416\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-clv5f"
Sep 16 17:27:27.328560 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:27.328534    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c0e1c10-67d7-4804-a977-829534bf6416-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-clv5f\" (UID: \"3c0e1c10-67d7-4804-a977-829534bf6416\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-clv5f"
Sep 16 17:27:27.336966 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:27.336940    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrl4x\" (UniqueName: \"kubernetes.io/projected/3c0e1c10-67d7-4804-a977-829534bf6416-kube-api-access-xrl4x\") pod \"prometheus-operator-68bcd7f987-clv5f\" (UID: \"3c0e1c10-67d7-4804-a977-829534bf6416\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-clv5f"
Sep 16 17:27:27.647630 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:27.647596    2562 patch_prober.go:28] interesting pod/image-registry-5565b7c65b-r4644 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 17:27:27.648001 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:27.647652    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5565b7c65b-r4644" podUID="7e97796b-c463-41e3-b7f9-d2a4c75b7211" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:27:27.829403 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:27.829319    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c0e1c10-67d7-4804-a977-829534bf6416-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-clv5f\" (UID: \"3c0e1c10-67d7-4804-a977-829534bf6416\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-clv5f"
Sep 16 17:27:27.831846 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:27.831815    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c0e1c10-67d7-4804-a977-829534bf6416-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-clv5f\" (UID: \"3c0e1c10-67d7-4804-a977-829534bf6416\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-clv5f"
Sep 16 17:27:28.106489 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:28.106410    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-clv5f"
Sep 16 17:27:28.399389 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:28.399299    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-clv5f"]
Sep 16 17:27:28.402001 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:27:28.401962    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0e1c10_67d7_4804_a977_829534bf6416.slice/crio-b8e8a20000c574d6fb44d9d6bf4c303d08249705782bf6093440476d06adc2b9 WatchSource:0}: Error finding container b8e8a20000c574d6fb44d9d6bf4c303d08249705782bf6093440476d06adc2b9: Status 404 returned error can't find the container with id b8e8a20000c574d6fb44d9d6bf4c303d08249705782bf6093440476d06adc2b9
Sep 16 17:27:28.606761 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:28.606725    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-stbrb" event={"ID":"2ff86f78-bedb-4545-8cdf-9eea531efcf9","Type":"ContainerStarted","Data":"4fbe43bd30bd80c12c71c32b764ce2445ffccbc1fee0ae297d3122624514f89c"}
Sep 16 17:27:28.607915 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:28.607894    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-clv5f" event={"ID":"3c0e1c10-67d7-4804-a977-829534bf6416","Type":"ContainerStarted","Data":"b8e8a20000c574d6fb44d9d6bf4c303d08249705782bf6093440476d06adc2b9"}
Sep 16 17:27:28.631371 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:28.631320    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-stbrb" podStartSLOduration=33.764728527 podStartE2EDuration="35.631305708s" totalImagesPullingTime="1.866577181s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:26:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:28.63012829 +0000 UTC m=+172.218518273" watchObservedRunningTime="2026-09-16 17:27:28.631305708 +0000 UTC m=+172.219695674"
Sep 16 17:27:30.616816 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:30.616780    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-clv5f" event={"ID":"3c0e1c10-67d7-4804-a977-829534bf6416","Type":"ContainerStarted","Data":"079d07b55f45991e9a2701680dfeaba81f36234bcc1ede26a3a32bc089f2df38"}
Sep 16 17:27:30.617212 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:30.616856    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-clv5f" event={"ID":"3c0e1c10-67d7-4804-a977-829534bf6416","Type":"ContainerStarted","Data":"d413b42d6877b6abdda7d6891ca2c04905f7418cb91dcde6fb00c7c5922ee3c1"}
Sep 16 17:27:30.638494 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:30.638443    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-clv5f" podStartSLOduration=2.244820932 podStartE2EDuration="3.638427598s" totalImagesPullingTime="1.393606666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:30.637638045 +0000 UTC m=+174.226028030" watchObservedRunningTime="2026-09-16 17:27:30.638427598 +0000 UTC m=+174.226817582"
Sep 16 17:27:31.996157 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:31.996115    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:27:32.632898 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.632866    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-26vr9"]
Sep 16 17:27:32.635818 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.635779    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-26vr9"
Sep 16 17:27:32.636029 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.636008    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4"]
Sep 16 17:27:32.641470 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.639673    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 16 17:27:32.641470 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.639707    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-gh9rw\""
Sep 16 17:27:32.641470 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.640053    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 16 17:27:32.642407 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.642387    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4"
Sep 16 17:27:32.648077 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.648054    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 17:27:32.648338 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.648322    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 17:27:32.648437 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.648374    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 17:27:32.648508 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.648477    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4xrjq\""
Sep 16 17:27:32.651010 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.650990    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-26vr9"]
Sep 16 17:27:32.660080 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.660058    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4"]
Sep 16 17:27:32.676753 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.676723    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pkc6m"]
Sep 16 17:27:32.679179 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.679163    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.690686 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.690667    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 17:27:32.690784 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.690675    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rg679\""
Sep 16 17:27:32.690978 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.690965    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 17:27:32.691484 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.691467    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 17:27:32.772967 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.772930    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qddm\" (UniqueName: \"kubernetes.io/projected/8ac71616-9ee5-41ef-8e28-0d4beda37f01-kube-api-access-2qddm\") pod \"openshift-state-metrics-67dcf8c979-26vr9\" (UID: \"8ac71616-9ee5-41ef-8e28-0d4beda37f01\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-26vr9"
Sep 16 17:27:32.772967 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.772971    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1853113c-2c05-4c44-b4a1-cb84d1f7f954-node-exporter-accelerators-collector-config\") pod \"node-exporter-pkc6m\" (UID: \"1853113c-2c05-4c44-b4a1-cb84d1f7f954\") " pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.773182 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.772991    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf8577d8-003e-4084-afe8-f76e6aa93a8d-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-dcsg4\" (UID: \"bf8577d8-003e-4084-afe8-f76e6aa93a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4"
Sep 16 17:27:32.773182 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.773014    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bf8577d8-003e-4084-afe8-f76e6aa93a8d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-dcsg4\" (UID: \"bf8577d8-003e-4084-afe8-f76e6aa93a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4"
Sep 16 17:27:32.773182 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.773048    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf8577d8-003e-4084-afe8-f76e6aa93a8d-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-dcsg4\" (UID: \"bf8577d8-003e-4084-afe8-f76e6aa93a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4"
Sep 16 17:27:32.773182 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.773077    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ac71616-9ee5-41ef-8e28-0d4beda37f01-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-26vr9\" (UID: \"8ac71616-9ee5-41ef-8e28-0d4beda37f01\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-26vr9"
Sep 16 17:27:32.773182 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.773136    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1853113c-2c05-4c44-b4a1-cb84d1f7f954-sys\") pod \"node-exporter-pkc6m\" (UID: \"1853113c-2c05-4c44-b4a1-cb84d1f7f954\") " pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.773182 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.773165    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1853113c-2c05-4c44-b4a1-cb84d1f7f954-root\") pod \"node-exporter-pkc6m\" (UID: \"1853113c-2c05-4c44-b4a1-cb84d1f7f954\") " pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.773381 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.773211    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1853113c-2c05-4c44-b4a1-cb84d1f7f954-metrics-client-ca\") pod \"node-exporter-pkc6m\" (UID: \"1853113c-2c05-4c44-b4a1-cb84d1f7f954\") " pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.773381 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.773233    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1853113c-2c05-4c44-b4a1-cb84d1f7f954-node-exporter-textfile\") pod \"node-exporter-pkc6m\" (UID: \"1853113c-2c05-4c44-b4a1-cb84d1f7f954\") " pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.773381 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.773253    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1853113c-2c05-4c44-b4a1-cb84d1f7f954-node-exporter-tls\") pod \"node-exporter-pkc6m\" (UID: \"1853113c-2c05-4c44-b4a1-cb84d1f7f954\") " pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.773381 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.773266    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1853113c-2c05-4c44-b4a1-cb84d1f7f954-node-exporter-wtmp\") pod \"node-exporter-pkc6m\" (UID: \"1853113c-2c05-4c44-b4a1-cb84d1f7f954\") " pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.773381 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.773283    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bf8577d8-003e-4084-afe8-f76e6aa93a8d-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-dcsg4\" (UID: \"bf8577d8-003e-4084-afe8-f76e6aa93a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4"
Sep 16 17:27:32.773381 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.773300    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf8577d8-003e-4084-afe8-f76e6aa93a8d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-dcsg4\" (UID: \"bf8577d8-003e-4084-afe8-f76e6aa93a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4"
Sep 16 17:27:32.773381 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.773351    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcr87\" (UniqueName: \"kubernetes.io/projected/bf8577d8-003e-4084-afe8-f76e6aa93a8d-kube-api-access-wcr87\") pod \"kube-state-metrics-59fbdd77d-dcsg4\" (UID: \"bf8577d8-003e-4084-afe8-f76e6aa93a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4"
Sep 16 17:27:32.773381 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.773374    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ac71616-9ee5-41ef-8e28-0d4beda37f01-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-26vr9\" (UID: \"8ac71616-9ee5-41ef-8e28-0d4beda37f01\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-26vr9"
Sep 16 17:27:32.773639 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.773392    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1853113c-2c05-4c44-b4a1-cb84d1f7f954-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pkc6m\" (UID: \"1853113c-2c05-4c44-b4a1-cb84d1f7f954\") " pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.773639 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.773424    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ac71616-9ee5-41ef-8e28-0d4beda37f01-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-26vr9\" (UID: \"8ac71616-9ee5-41ef-8e28-0d4beda37f01\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-26vr9"
Sep 16 17:27:32.773639 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.773448    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svslw\" (UniqueName: \"kubernetes.io/projected/1853113c-2c05-4c44-b4a1-cb84d1f7f954-kube-api-access-svslw\") pod \"node-exporter-pkc6m\" (UID: \"1853113c-2c05-4c44-b4a1-cb84d1f7f954\") " pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.874116 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.874072    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1853113c-2c05-4c44-b4a1-cb84d1f7f954-metrics-client-ca\") pod \"node-exporter-pkc6m\" (UID: \"1853113c-2c05-4c44-b4a1-cb84d1f7f954\") " pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.874116 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.874111    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1853113c-2c05-4c44-b4a1-cb84d1f7f954-node-exporter-textfile\") pod \"node-exporter-pkc6m\" (UID: \"1853113c-2c05-4c44-b4a1-cb84d1f7f954\") " pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.874362 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.874132    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1853113c-2c05-4c44-b4a1-cb84d1f7f954-node-exporter-tls\") pod \"node-exporter-pkc6m\" (UID: \"1853113c-2c05-4c44-b4a1-cb84d1f7f954\") " pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.874362 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.874148    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1853113c-2c05-4c44-b4a1-cb84d1f7f954-node-exporter-wtmp\") pod \"node-exporter-pkc6m\" (UID: \"1853113c-2c05-4c44-b4a1-cb84d1f7f954\") " pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.874362 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.874168    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bf8577d8-003e-4084-afe8-f76e6aa93a8d-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-dcsg4\" (UID: \"bf8577d8-003e-4084-afe8-f76e6aa93a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4"
Sep 16 17:27:32.874362 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.874189    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf8577d8-003e-4084-afe8-f76e6aa93a8d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-dcsg4\" (UID: \"bf8577d8-003e-4084-afe8-f76e6aa93a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4"
Sep 16 17:27:32.874362 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.874209    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcr87\" (UniqueName: \"kubernetes.io/projected/bf8577d8-003e-4084-afe8-f76e6aa93a8d-kube-api-access-wcr87\") pod \"kube-state-metrics-59fbdd77d-dcsg4\" (UID: \"bf8577d8-003e-4084-afe8-f76e6aa93a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4"
Sep 16 17:27:32.874362 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.874230    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ac71616-9ee5-41ef-8e28-0d4beda37f01-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-26vr9\" (UID: \"8ac71616-9ee5-41ef-8e28-0d4beda37f01\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-26vr9"
Sep 16 17:27:32.874362 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.874260    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1853113c-2c05-4c44-b4a1-cb84d1f7f954-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pkc6m\" (UID: \"1853113c-2c05-4c44-b4a1-cb84d1f7f954\") " pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.874362 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.874290    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ac71616-9ee5-41ef-8e28-0d4beda37f01-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-26vr9\" (UID: \"8ac71616-9ee5-41ef-8e28-0d4beda37f01\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-26vr9"
Sep 16 17:27:32.874362 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.874316    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svslw\" (UniqueName: \"kubernetes.io/projected/1853113c-2c05-4c44-b4a1-cb84d1f7f954-kube-api-access-svslw\") pod \"node-exporter-pkc6m\" (UID: \"1853113c-2c05-4c44-b4a1-cb84d1f7f954\") " pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.874362 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.874342    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qddm\" (UniqueName: \"kubernetes.io/projected/8ac71616-9ee5-41ef-8e28-0d4beda37f01-kube-api-access-2qddm\") pod \"openshift-state-metrics-67dcf8c979-26vr9\" (UID: \"8ac71616-9ee5-41ef-8e28-0d4beda37f01\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-26vr9"
Sep 16 17:27:32.874362 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.874367    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1853113c-2c05-4c44-b4a1-cb84d1f7f954-node-exporter-accelerators-collector-config\") pod \"node-exporter-pkc6m\" (UID: \"1853113c-2c05-4c44-b4a1-cb84d1f7f954\") " pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.874912 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.874391    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf8577d8-003e-4084-afe8-f76e6aa93a8d-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-dcsg4\" (UID: \"bf8577d8-003e-4084-afe8-f76e6aa93a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4"
Sep 16 17:27:32.874912 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.874418    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1853113c-2c05-4c44-b4a1-cb84d1f7f954-node-exporter-wtmp\") pod \"node-exporter-pkc6m\" (UID: \"1853113c-2c05-4c44-b4a1-cb84d1f7f954\") " pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.874912 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.874422    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bf8577d8-003e-4084-afe8-f76e6aa93a8d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-dcsg4\" (UID: \"bf8577d8-003e-4084-afe8-f76e6aa93a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4"
Sep 16 17:27:32.874912 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.874502    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf8577d8-003e-4084-afe8-f76e6aa93a8d-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-dcsg4\" (UID: \"bf8577d8-003e-4084-afe8-f76e6aa93a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4"
Sep 16 17:27:32.874912 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.874561    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ac71616-9ee5-41ef-8e28-0d4beda37f01-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-26vr9\" (UID: \"8ac71616-9ee5-41ef-8e28-0d4beda37f01\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-26vr9"
Sep 16 17:27:32.874912 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.874594    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1853113c-2c05-4c44-b4a1-cb84d1f7f954-sys\") pod \"node-exporter-pkc6m\" (UID: \"1853113c-2c05-4c44-b4a1-cb84d1f7f954\") " pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.874912 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.874619    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bf8577d8-003e-4084-afe8-f76e6aa93a8d-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-dcsg4\" (UID: \"bf8577d8-003e-4084-afe8-f76e6aa93a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4"
Sep 16 17:27:32.874912 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.874626    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1853113c-2c05-4c44-b4a1-cb84d1f7f954-root\") pod \"node-exporter-pkc6m\" (UID: \"1853113c-2c05-4c44-b4a1-cb84d1f7f954\") " pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.874912 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.874669    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1853113c-2c05-4c44-b4a1-cb84d1f7f954-root\") pod \"node-exporter-pkc6m\" (UID: \"1853113c-2c05-4c44-b4a1-cb84d1f7f954\") " pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.874912 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.874798    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1853113c-2c05-4c44-b4a1-cb84d1f7f954-metrics-client-ca\") pod \"node-exporter-pkc6m\" (UID: \"1853113c-2c05-4c44-b4a1-cb84d1f7f954\") " pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.874912 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.874917    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1853113c-2c05-4c44-b4a1-cb84d1f7f954-node-exporter-textfile\") pod \"node-exporter-pkc6m\" (UID: \"1853113c-2c05-4c44-b4a1-cb84d1f7f954\") " pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.875449 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.875160    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bf8577d8-003e-4084-afe8-f76e6aa93a8d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-dcsg4\" (UID: \"bf8577d8-003e-4084-afe8-f76e6aa93a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4"
Sep 16 17:27:32.875566 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.875537    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1853113c-2c05-4c44-b4a1-cb84d1f7f954-node-exporter-accelerators-collector-config\") pod \"node-exporter-pkc6m\" (UID: \"1853113c-2c05-4c44-b4a1-cb84d1f7f954\") " pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.875767 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.875733    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf8577d8-003e-4084-afe8-f76e6aa93a8d-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-dcsg4\" (UID: \"bf8577d8-003e-4084-afe8-f76e6aa93a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4"
Sep 16 17:27:32.875874 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.875841    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1853113c-2c05-4c44-b4a1-cb84d1f7f954-sys\") pod \"node-exporter-pkc6m\" (UID: \"1853113c-2c05-4c44-b4a1-cb84d1f7f954\") " pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.876053 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.876032    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ac71616-9ee5-41ef-8e28-0d4beda37f01-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-26vr9\" (UID: \"8ac71616-9ee5-41ef-8e28-0d4beda37f01\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-26vr9"
Sep 16 17:27:32.877205 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.877177    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1853113c-2c05-4c44-b4a1-cb84d1f7f954-node-exporter-tls\") pod \"node-exporter-pkc6m\" (UID: \"1853113c-2c05-4c44-b4a1-cb84d1f7f954\") " pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.877325 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.877262    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ac71616-9ee5-41ef-8e28-0d4beda37f01-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-26vr9\" (UID: \"8ac71616-9ee5-41ef-8e28-0d4beda37f01\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-26vr9"
Sep 16 17:27:32.877757 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.877734    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ac71616-9ee5-41ef-8e28-0d4beda37f01-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-26vr9\" (UID: \"8ac71616-9ee5-41ef-8e28-0d4beda37f01\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-26vr9"
Sep 16 17:27:32.877925 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.877909    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf8577d8-003e-4084-afe8-f76e6aa93a8d-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-dcsg4\" (UID: \"bf8577d8-003e-4084-afe8-f76e6aa93a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4"
Sep 16 17:27:32.877970 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.877949    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf8577d8-003e-4084-afe8-f76e6aa93a8d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-dcsg4\" (UID: \"bf8577d8-003e-4084-afe8-f76e6aa93a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4"
Sep 16 17:27:32.878062 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.878042    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1853113c-2c05-4c44-b4a1-cb84d1f7f954-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pkc6m\" (UID: \"1853113c-2c05-4c44-b4a1-cb84d1f7f954\") " pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.896229 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.896173    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svslw\" (UniqueName: \"kubernetes.io/projected/1853113c-2c05-4c44-b4a1-cb84d1f7f954-kube-api-access-svslw\") pod \"node-exporter-pkc6m\" (UID: \"1853113c-2c05-4c44-b4a1-cb84d1f7f954\") " pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:32.897086 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.897068    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcr87\" (UniqueName: \"kubernetes.io/projected/bf8577d8-003e-4084-afe8-f76e6aa93a8d-kube-api-access-wcr87\") pod \"kube-state-metrics-59fbdd77d-dcsg4\" (UID: \"bf8577d8-003e-4084-afe8-f76e6aa93a8d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4"
Sep 16 17:27:32.897336 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.897317    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qddm\" (UniqueName: \"kubernetes.io/projected/8ac71616-9ee5-41ef-8e28-0d4beda37f01-kube-api-access-2qddm\") pod \"openshift-state-metrics-67dcf8c979-26vr9\" (UID: \"8ac71616-9ee5-41ef-8e28-0d4beda37f01\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-26vr9"
Sep 16 17:27:32.950672 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.950645    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-26vr9"
Sep 16 17:27:32.956080 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.956063    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4"
Sep 16 17:27:32.987272 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:32.987234    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pkc6m"
Sep 16 17:27:33.000696 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:27:33.000661    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1853113c_2c05_4c44_b4a1_cb84d1f7f954.slice/crio-d49dc148b2bb72c4b281c78880ad8c8587782bc2ca40f468e68d0be9e682b10a WatchSource:0}: Error finding container d49dc148b2bb72c4b281c78880ad8c8587782bc2ca40f468e68d0be9e682b10a: Status 404 returned error can't find the container with id d49dc148b2bb72c4b281c78880ad8c8587782bc2ca40f468e68d0be9e682b10a
Sep 16 17:27:33.224464 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:33.224440    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-26vr9"]
Sep 16 17:27:33.226191 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:27:33.226152    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac71616_9ee5_41ef_8e28_0d4beda37f01.slice/crio-d8de35322877839701a0e2a86f8c4ae4693cd331eec8693a7a4c1bfad176a5bb WatchSource:0}: Error finding container d8de35322877839701a0e2a86f8c4ae4693cd331eec8693a7a4c1bfad176a5bb: Status 404 returned error can't find the container with id d8de35322877839701a0e2a86f8c4ae4693cd331eec8693a7a4c1bfad176a5bb
Sep 16 17:27:33.229563 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:33.229535    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4"]
Sep 16 17:27:33.231619 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:27:33.231588    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf8577d8_003e_4084_afe8_f76e6aa93a8d.slice/crio-43f22735ac9501dc3bfa0d45db1fc2f3b6c8429b5249b6653dbd1cacd642c534 WatchSource:0}: Error finding container 43f22735ac9501dc3bfa0d45db1fc2f3b6c8429b5249b6653dbd1cacd642c534: Status 404 returned error can't find the container with id 43f22735ac9501dc3bfa0d45db1fc2f3b6c8429b5249b6653dbd1cacd642c534
Sep 16 17:27:33.591821 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:33.591733    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4lcwz"
Sep 16 17:27:33.631788 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:33.631748    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-26vr9" event={"ID":"8ac71616-9ee5-41ef-8e28-0d4beda37f01","Type":"ContainerStarted","Data":"22f1d8595f25471cadb7055556f3c0831a1e1f99ae46544bc530675c2750deb6"}
Sep 16 17:27:33.631968 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:33.631831    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-26vr9" event={"ID":"8ac71616-9ee5-41ef-8e28-0d4beda37f01","Type":"ContainerStarted","Data":"fc935eafd812a9088374f0e93fac7f835ca99154b8b9b83aeb887b0a3ca96612"}
Sep 16 17:27:33.631968 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:33.631863    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-26vr9" event={"ID":"8ac71616-9ee5-41ef-8e28-0d4beda37f01","Type":"ContainerStarted","Data":"d8de35322877839701a0e2a86f8c4ae4693cd331eec8693a7a4c1bfad176a5bb"}
Sep 16 17:27:33.633104 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:33.633070    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pkc6m" event={"ID":"1853113c-2c05-4c44-b4a1-cb84d1f7f954","Type":"ContainerStarted","Data":"d49dc148b2bb72c4b281c78880ad8c8587782bc2ca40f468e68d0be9e682b10a"}
Sep 16 17:27:33.634344 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:33.634319    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4" event={"ID":"bf8577d8-003e-4084-afe8-f76e6aa93a8d","Type":"ContainerStarted","Data":"43f22735ac9501dc3bfa0d45db1fc2f3b6c8429b5249b6653dbd1cacd642c534"}
Sep 16 17:27:34.642276 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:34.642187    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-26vr9" event={"ID":"8ac71616-9ee5-41ef-8e28-0d4beda37f01","Type":"ContainerStarted","Data":"94ebec12fed2a6f4e523735dd8f92148607a8568db1d1b18a61ef625e1a19169"}
Sep 16 17:27:34.643753 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:34.643727    2562 generic.go:356] "Generic (PLEG): container finished" podID="1853113c-2c05-4c44-b4a1-cb84d1f7f954" containerID="9b8ea8fc05edb7d72359dd67c02eef16effa002f7a4418c5c9536fb10127a848" exitCode=0
Sep 16 17:27:34.643880 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:34.643796    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pkc6m" event={"ID":"1853113c-2c05-4c44-b4a1-cb84d1f7f954","Type":"ContainerDied","Data":"9b8ea8fc05edb7d72359dd67c02eef16effa002f7a4418c5c9536fb10127a848"}
Sep 16 17:27:34.672231 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:34.672186    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-26vr9" podStartSLOduration=1.598968792 podStartE2EDuration="2.672170442s" totalImagesPullingTime="1.07320165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:34.671788405 +0000 UTC m=+178.260178389" watchObservedRunningTime="2026-09-16 17:27:34.672170442 +0000 UTC m=+178.260560426"
Sep 16 17:27:35.651038 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:35.650957    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pkc6m" event={"ID":"1853113c-2c05-4c44-b4a1-cb84d1f7f954","Type":"ContainerStarted","Data":"b5ae818a116e4abcb03e31174d8332565c684a5c8a3a5582ece26a71e29db3b9"}
Sep 16 17:27:35.651038 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:35.651023    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pkc6m" event={"ID":"1853113c-2c05-4c44-b4a1-cb84d1f7f954","Type":"ContainerStarted","Data":"4930a013e5073b4b5dfd0124d7cf828f032c9a59ed5eda4c5972d44247f4e245"}
Sep 16 17:27:35.653444 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:35.653414    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4" event={"ID":"bf8577d8-003e-4084-afe8-f76e6aa93a8d","Type":"ContainerStarted","Data":"f4b01baa68fd615cf6a250346bf8d8cc4743b90358c516df414f0aede635b413"}
Sep 16 17:27:35.653554 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:35.653484    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4" event={"ID":"bf8577d8-003e-4084-afe8-f76e6aa93a8d","Type":"ContainerStarted","Data":"855e8154d93b7a0f9a24402c505cef579c9135ef6d25e129916d30573ffd5236"}
Sep 16 17:27:35.653554 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:35.653512    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4" event={"ID":"bf8577d8-003e-4084-afe8-f76e6aa93a8d","Type":"ContainerStarted","Data":"8c1bd1b20ab999f35904f1ba5f03c4720c2619930827a74720f2d63f19882126"}
Sep 16 17:27:35.704211 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:35.704087    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pkc6m" podStartSLOduration=2.2447309 podStartE2EDuration="3.704067441s" totalImagesPullingTime="1.39071437s" totalInitContainerRuntime="68.622171ms" isStatefulPod=true podCreationTimestamp="2026-09-16 17:27:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:35.70208948 +0000 UTC m=+179.290479476" watchObservedRunningTime="2026-09-16 17:27:35.704067441 +0000 UTC m=+179.292457425"
Sep 16 17:27:37.646192 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:37.646163    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:27:37.668395 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:37.668346    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-dcsg4" podStartSLOduration=3.5039445750000002 podStartE2EDuration="5.668330388s" totalImagesPullingTime="2.164385813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:35.783898474 +0000 UTC m=+179.372288457" watchObservedRunningTime="2026-09-16 17:27:37.668330388 +0000 UTC m=+181.256720372"
Sep 16 17:27:38.002639 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:38.002596    2562 patch_prober.go:28] interesting pod/image-registry-868c457c6b-pgc4m container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 17:27:38.002848 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:38.002662    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-868c457c6b-pgc4m" podUID="dc3dea1b-13cf-444b-bb84-42d022358828" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:27:39.292695 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.292658    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 17:27:39.295965 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.295947    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.305256 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.305236    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 17:27:39.305425 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.305407    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 17:27:39.305503 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.305486    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6ohvju49g81o3\""
Sep 16 17:27:39.306359 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.306343    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 17:27:39.306539 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.306508    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 17:27:39.306657 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.306543    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 17:27:39.306657 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.306564    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 17:27:39.306657 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.306600    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 17:27:39.306657 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.306602    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 17:27:39.306993 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.306978    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 17:27:39.307928 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.307914    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 17:27:39.309851 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.309833    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 17:27:39.309916 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.309863    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 17:27:39.309916 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.309903    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 17:27:39.311150 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.311134    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7s7cp\""
Sep 16 17:27:39.326162 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.326141    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 17:27:39.327551 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.327526    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 17:27:39.384495 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.384464    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 17:27:39.436043 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.436011    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.436234 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.436046    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.436234 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.436071    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.436234 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.436090    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-config\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.436234 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.436104    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.436234 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.436120    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.436234 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.436182    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz7m2\" (UniqueName: \"kubernetes.io/projected/b4486e32-1bf7-4a7e-a061-693552991f05-kube-api-access-gz7m2\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.436512 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.436242    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.436512 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.436275    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4486e32-1bf7-4a7e-a061-693552991f05-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.436512 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.436327    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.436512 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.436394    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-web-config\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.436512 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.436485    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.436756 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.436514    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.436756 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.436557    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.436756 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.436588    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.436756 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.436619    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.436756 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.436673    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.436756 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.436707    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.436756 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.436729    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.437010 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.436755    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4486e32-1bf7-4a7e-a061-693552991f05-config-out\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.537634 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.537597    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.537634 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.537636    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.537890 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.537652    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.537890 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.537671    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4486e32-1bf7-4a7e-a061-693552991f05-config-out\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.537890 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.537695    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.537890 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.537720    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.537890 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.537768    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.537890 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.537794    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-config\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.537890 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.537840    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.537890 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.537865    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.537890 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.537885    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz7m2\" (UniqueName: \"kubernetes.io/projected/b4486e32-1bf7-4a7e-a061-693552991f05-kube-api-access-gz7m2\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.538303 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.537919    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.538303 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.537944    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4486e32-1bf7-4a7e-a061-693552991f05-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.538303 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.537965    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.538303 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.538006    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-web-config\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.538303 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.538084    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.538303 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.538111    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.538303 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.538134    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.538864 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.538836    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.538973 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.538836    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.538973 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.538141    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.538973 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.538931    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.539116 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.538985    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.539116 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.539001    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.539531 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.539502    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.540364 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.539752    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.540957 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.540879    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.541613 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.541583    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.541790 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.541766    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.541879 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.541797    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4486e32-1bf7-4a7e-a061-693552991f05-config-out\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.541879 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.541768    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.542252 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.542231    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.542642 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.542619    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-config\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.542742 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.542641    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4486e32-1bf7-4a7e-a061-693552991f05-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.542952 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.542933    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.543406 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.543381    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.543881 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.543858    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.544210 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.544187    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.544512 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.544495    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-web-config\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.558177 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.558152    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz7m2\" (UniqueName: \"kubernetes.io/projected/b4486e32-1bf7-4a7e-a061-693552991f05-kube-api-access-gz7m2\") pod \"prometheus-k8s-0\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.572656 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.572629    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-868c457c6b-pgc4m"
Sep 16 17:27:39.604549 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.604518    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:27:39.912339 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:39.912203    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 17:27:39.914834 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:27:39.914793    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4486e32_1bf7_4a7e_a061_693552991f05.slice/crio-c16ecb8bc7cbfcaa272fa5b8362bdbfa83e1a8648aab553c9db75e175d00a751 WatchSource:0}: Error finding container c16ecb8bc7cbfcaa272fa5b8362bdbfa83e1a8648aab553c9db75e175d00a751: Status 404 returned error can't find the container with id c16ecb8bc7cbfcaa272fa5b8362bdbfa83e1a8648aab553c9db75e175d00a751
Sep 16 17:27:40.675144 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:40.675108    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4486e32-1bf7-4a7e-a061-693552991f05","Type":"ContainerStarted","Data":"c16ecb8bc7cbfcaa272fa5b8362bdbfa83e1a8648aab553c9db75e175d00a751"}
Sep 16 17:27:41.680585 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:41.680550    2562 generic.go:356] "Generic (PLEG): container finished" podID="b4486e32-1bf7-4a7e-a061-693552991f05" containerID="45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5" exitCode=0
Sep 16 17:27:41.680970 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:41.680616    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4486e32-1bf7-4a7e-a061-693552991f05","Type":"ContainerDied","Data":"45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5"}
Sep 16 17:27:42.663332 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:42.663256    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5565b7c65b-r4644" podUID="7e97796b-c463-41e3-b7f9-d2a4c75b7211" containerName="registry" containerID="cri-o://4ced4e8174970be6c4d91bd232c10ca6d8449e2754cb2de387496d8e67d13618" gracePeriod=30
Sep 16 17:27:42.919647 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:42.919621    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:27:43.068501 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.068472    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7e97796b-c463-41e3-b7f9-d2a4c75b7211-image-registry-private-configuration\") pod \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") "
Sep 16 17:27:43.068678 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.068539    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7e97796b-c463-41e3-b7f9-d2a4c75b7211-registry-certificates\") pod \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") "
Sep 16 17:27:43.068678 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.068566    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7e97796b-c463-41e3-b7f9-d2a4c75b7211-trusted-ca\") pod \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") "
Sep 16 17:27:43.068791 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.068692    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-bound-sa-token\") pod \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") "
Sep 16 17:27:43.068791 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.068740    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-registry-tls\") pod \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") "
Sep 16 17:27:43.068918 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.068788    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7e97796b-c463-41e3-b7f9-d2a4c75b7211-installation-pull-secrets\") pod \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") "
Sep 16 17:27:43.068918 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.068857    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bld2r\" (UniqueName: \"kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-kube-api-access-bld2r\") pod \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") "
Sep 16 17:27:43.068918 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.068889    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7e97796b-c463-41e3-b7f9-d2a4c75b7211-ca-trust-extracted\") pod \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\" (UID: \"7e97796b-c463-41e3-b7f9-d2a4c75b7211\") "
Sep 16 17:27:43.069051 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.069005    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e97796b-c463-41e3-b7f9-d2a4c75b7211-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "7e97796b-c463-41e3-b7f9-d2a4c75b7211" (UID: "7e97796b-c463-41e3-b7f9-d2a4c75b7211"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:27:43.069051 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.069017    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e97796b-c463-41e3-b7f9-d2a4c75b7211-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "7e97796b-c463-41e3-b7f9-d2a4c75b7211" (UID: "7e97796b-c463-41e3-b7f9-d2a4c75b7211"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:27:43.069356 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.069308    2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7e97796b-c463-41e3-b7f9-d2a4c75b7211-registry-certificates\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:27:43.069356 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.069332    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7e97796b-c463-41e3-b7f9-d2a4c75b7211-trusted-ca\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:27:43.071100 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.071045    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "7e97796b-c463-41e3-b7f9-d2a4c75b7211" (UID: "7e97796b-c463-41e3-b7f9-d2a4c75b7211"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:27:43.071259 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.071216    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-kube-api-access-bld2r" (OuterVolumeSpecName: "kube-api-access-bld2r") pod "7e97796b-c463-41e3-b7f9-d2a4c75b7211" (UID: "7e97796b-c463-41e3-b7f9-d2a4c75b7211"). InnerVolumeSpecName "kube-api-access-bld2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:27:43.071344 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.071327    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e97796b-c463-41e3-b7f9-d2a4c75b7211-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "7e97796b-c463-41e3-b7f9-d2a4c75b7211" (UID: "7e97796b-c463-41e3-b7f9-d2a4c75b7211"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:27:43.071426 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.071405    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e97796b-c463-41e3-b7f9-d2a4c75b7211-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "7e97796b-c463-41e3-b7f9-d2a4c75b7211" (UID: "7e97796b-c463-41e3-b7f9-d2a4c75b7211"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:27:43.071524 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.071439    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7e97796b-c463-41e3-b7f9-d2a4c75b7211" (UID: "7e97796b-c463-41e3-b7f9-d2a4c75b7211"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:27:43.078818 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.078785    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e97796b-c463-41e3-b7f9-d2a4c75b7211-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "7e97796b-c463-41e3-b7f9-d2a4c75b7211" (UID: "7e97796b-c463-41e3-b7f9-d2a4c75b7211"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:27:43.170891 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.170787    2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-bound-sa-token\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:27:43.170891 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.170838    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-registry-tls\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:27:43.170891 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.170852    2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7e97796b-c463-41e3-b7f9-d2a4c75b7211-installation-pull-secrets\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:27:43.170891 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.170865    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bld2r\" (UniqueName: \"kubernetes.io/projected/7e97796b-c463-41e3-b7f9-d2a4c75b7211-kube-api-access-bld2r\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:27:43.170891 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.170879    2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7e97796b-c463-41e3-b7f9-d2a4c75b7211-ca-trust-extracted\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:27:43.170891 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.170893    2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7e97796b-c463-41e3-b7f9-d2a4c75b7211-image-registry-private-configuration\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:27:43.689841 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.689780    2562 generic.go:356] "Generic (PLEG): container finished" podID="7e97796b-c463-41e3-b7f9-d2a4c75b7211" containerID="4ced4e8174970be6c4d91bd232c10ca6d8449e2754cb2de387496d8e67d13618" exitCode=0
Sep 16 17:27:43.690075 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.689849    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5565b7c65b-r4644"
Sep 16 17:27:43.690201 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.689843    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5565b7c65b-r4644" event={"ID":"7e97796b-c463-41e3-b7f9-d2a4c75b7211","Type":"ContainerDied","Data":"4ced4e8174970be6c4d91bd232c10ca6d8449e2754cb2de387496d8e67d13618"}
Sep 16 17:27:43.690261 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.690203    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5565b7c65b-r4644" event={"ID":"7e97796b-c463-41e3-b7f9-d2a4c75b7211","Type":"ContainerDied","Data":"9fc769786d6b855027df4cdc3d682febc0cbbb65ae0076294084a7be06dc4958"}
Sep 16 17:27:43.690261 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.690234    2562 scope.go:117] "RemoveContainer" containerID="4ced4e8174970be6c4d91bd232c10ca6d8449e2754cb2de387496d8e67d13618"
Sep 16 17:27:43.699820 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.699776    2562 scope.go:117] "RemoveContainer" containerID="4ced4e8174970be6c4d91bd232c10ca6d8449e2754cb2de387496d8e67d13618"
Sep 16 17:27:43.700118 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:27:43.700087    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ced4e8174970be6c4d91bd232c10ca6d8449e2754cb2de387496d8e67d13618\": container with ID starting with 4ced4e8174970be6c4d91bd232c10ca6d8449e2754cb2de387496d8e67d13618 not found: ID does not exist" containerID="4ced4e8174970be6c4d91bd232c10ca6d8449e2754cb2de387496d8e67d13618"
Sep 16 17:27:43.700204 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.700129    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ced4e8174970be6c4d91bd232c10ca6d8449e2754cb2de387496d8e67d13618"} err="failed to get container status \"4ced4e8174970be6c4d91bd232c10ca6d8449e2754cb2de387496d8e67d13618\": rpc error: code = NotFound desc = could not find container \"4ced4e8174970be6c4d91bd232c10ca6d8449e2754cb2de387496d8e67d13618\": container with ID starting with 4ced4e8174970be6c4d91bd232c10ca6d8449e2754cb2de387496d8e67d13618 not found: ID does not exist"
Sep 16 17:27:43.716093 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.716066    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5565b7c65b-r4644"]
Sep 16 17:27:43.720455 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:43.720435    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5565b7c65b-r4644"]
Sep 16 17:27:45.001102 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:45.001067    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e97796b-c463-41e3-b7f9-d2a4c75b7211" path="/var/lib/kubelet/pods/7e97796b-c463-41e3-b7f9-d2a4c75b7211/volumes"
Sep 16 17:27:45.703530 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:45.703495    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4486e32-1bf7-4a7e-a061-693552991f05","Type":"ContainerStarted","Data":"d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6"}
Sep 16 17:27:45.703700 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:45.703569    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4486e32-1bf7-4a7e-a061-693552991f05","Type":"ContainerStarted","Data":"9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab"}
Sep 16 17:27:47.714961 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:47.714927    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4486e32-1bf7-4a7e-a061-693552991f05","Type":"ContainerStarted","Data":"73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439"}
Sep 16 17:27:47.715353 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:47.714991    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4486e32-1bf7-4a7e-a061-693552991f05","Type":"ContainerStarted","Data":"dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c"}
Sep 16 17:27:47.715353 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:47.715021    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4486e32-1bf7-4a7e-a061-693552991f05","Type":"ContainerStarted","Data":"78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8"}
Sep 16 17:27:47.715353 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:47.715046    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4486e32-1bf7-4a7e-a061-693552991f05","Type":"ContainerStarted","Data":"217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39"}
Sep 16 17:27:47.752101 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:47.752050    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.587421338 podStartE2EDuration="8.752032601s" totalImagesPullingTime="6.143130492s" totalInitContainerRuntime="21.480771ms" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:47.748986998 +0000 UTC m=+191.337376981" watchObservedRunningTime="2026-09-16 17:27:47.752032601 +0000 UTC m=+191.340422585"
Sep 16 17:27:49.604639 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:27:49.604612    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:02.777457 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:02.777420    2562 generic.go:356] "Generic (PLEG): container finished" podID="6ad7439f-6922-4550-84fa-5ea0e6ff8081" containerID="800048d6cd9a241cf75c8363f05b3bad39c769414ba7274511bc741e06a77bcb" exitCode=0
Sep 16 17:28:02.777899 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:02.777492    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-gghkt" event={"ID":"6ad7439f-6922-4550-84fa-5ea0e6ff8081","Type":"ContainerDied","Data":"800048d6cd9a241cf75c8363f05b3bad39c769414ba7274511bc741e06a77bcb"}
Sep 16 17:28:02.777899 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:02.777847    2562 scope.go:117] "RemoveContainer" containerID="800048d6cd9a241cf75c8363f05b3bad39c769414ba7274511bc741e06a77bcb"
Sep 16 17:28:03.549464 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:03.549436    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4lcwz_afb12787-82e5-40ce-b4f5-b34655ba032f/dns/0.log"
Sep 16 17:28:03.749779 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:03.749747    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4lcwz_afb12787-82e5-40ce-b4f5-b34655ba032f/kube-rbac-proxy/0.log"
Sep 16 17:28:03.782708 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:03.782679    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-gghkt" event={"ID":"6ad7439f-6922-4550-84fa-5ea0e6ff8081","Type":"ContainerStarted","Data":"1961f3b89c1bd9faa4b564c4e2e2c01421b68e901fa5923f2ccfe5d5e5dc4e1b"}
Sep 16 17:28:04.748697 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:04.748671    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2r4vj_79c16240-59ee-4b5e-98c9-86ef04179efa/dns-node-resolver/0.log"
Sep 16 17:28:05.551256 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:05.551232    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2n9xt_64f7e8df-d87e-426f-acca-39781d05567c/serve-healthcheck-canary/0.log"
Sep 16 17:28:39.605328 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:39.605295    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:39.625398 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:39.625375    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:39.945486 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:39.945461    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:48.737152 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:48.737111    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs\") pod \"network-metrics-daemon-fw78m\" (UID: \"9d27ec00-06e7-431b-925f-addd45ff0115\") " pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:28:48.739452 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:48.739427    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d27ec00-06e7-431b-925f-addd45ff0115-metrics-certs\") pod \"network-metrics-daemon-fw78m\" (UID: \"9d27ec00-06e7-431b-925f-addd45ff0115\") " pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:28:48.800341 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:48.800299    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q452j\""
Sep 16 17:28:48.807868 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:48.807849    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fw78m"
Sep 16 17:28:49.078531 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:49.078504    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fw78m"]
Sep 16 17:28:49.080654 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:28:49.080629    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d27ec00_06e7_431b_925f_addd45ff0115.slice/crio-0d511c463826e3eba387cc21cfe665be2e46540c42eb6c66b9e3d2b5a6ed3174 WatchSource:0}: Error finding container 0d511c463826e3eba387cc21cfe665be2e46540c42eb6c66b9e3d2b5a6ed3174: Status 404 returned error can't find the container with id 0d511c463826e3eba387cc21cfe665be2e46540c42eb6c66b9e3d2b5a6ed3174
Sep 16 17:28:49.968663 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:49.968632    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fw78m" event={"ID":"9d27ec00-06e7-431b-925f-addd45ff0115","Type":"ContainerStarted","Data":"0d511c463826e3eba387cc21cfe665be2e46540c42eb6c66b9e3d2b5a6ed3174"}
Sep 16 17:28:50.974511 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:50.974472    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fw78m" event={"ID":"9d27ec00-06e7-431b-925f-addd45ff0115","Type":"ContainerStarted","Data":"4fcd35ec643a5fc1e9c10c44108377cacbc260b291efd6dcc2f05b2edf083db1"}
Sep 16 17:28:50.974882 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:50.974525    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fw78m" event={"ID":"9d27ec00-06e7-431b-925f-addd45ff0115","Type":"ContainerStarted","Data":"adda9c6ec034558d7c55882ec1f45a5465ec5ac5f505f729bde7e7c8cf7226a2"}
Sep 16 17:28:50.993616 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:50.993565    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fw78m" podStartSLOduration=253.144547355 podStartE2EDuration="4m13.993545481s" totalImagesPullingTime="848.998126ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:28:50.991572611 +0000 UTC m=+254.579962596" watchObservedRunningTime="2026-09-16 17:28:50.993545481 +0000 UTC m=+254.581935464"
Sep 16 17:28:57.455706 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.455667    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 17:28:57.456205 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.456083    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b4486e32-1bf7-4a7e-a061-693552991f05" containerName="prometheus" containerID="cri-o://9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab" gracePeriod=600
Sep 16 17:28:57.456205 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.456131    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b4486e32-1bf7-4a7e-a061-693552991f05" containerName="kube-rbac-proxy" containerID="cri-o://dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c" gracePeriod=600
Sep 16 17:28:57.456205 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.456156    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b4486e32-1bf7-4a7e-a061-693552991f05" containerName="kube-rbac-proxy-thanos" containerID="cri-o://73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439" gracePeriod=600
Sep 16 17:28:57.456205 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.456157    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b4486e32-1bf7-4a7e-a061-693552991f05" containerName="thanos-sidecar" containerID="cri-o://217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39" gracePeriod=600
Sep 16 17:28:57.456422 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.456281    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b4486e32-1bf7-4a7e-a061-693552991f05" containerName="kube-rbac-proxy-web" containerID="cri-o://78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8" gracePeriod=600
Sep 16 17:28:57.456422 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.456286    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b4486e32-1bf7-4a7e-a061-693552991f05" containerName="config-reloader" containerID="cri-o://d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6" gracePeriod=600
Sep 16 17:28:57.723755 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.723572    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:57.810461 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.810415    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-prometheus-k8s-tls\") pod \"b4486e32-1bf7-4a7e-a061-693552991f05\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") "
Sep 16 17:28:57.810461 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.810465    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4486e32-1bf7-4a7e-a061-693552991f05-tls-assets\") pod \"b4486e32-1bf7-4a7e-a061-693552991f05\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") "
Sep 16 17:28:57.810706 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.810502    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-k8s-rulefiles-0\") pod \"b4486e32-1bf7-4a7e-a061-693552991f05\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") "
Sep 16 17:28:57.810706 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.810526    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-k8s-rulefiles-1\") pod \"b4486e32-1bf7-4a7e-a061-693552991f05\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") "
Sep 16 17:28:57.810706 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.810552    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-configmap-kubelet-serving-ca-bundle\") pod \"b4486e32-1bf7-4a7e-a061-693552991f05\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") "
Sep 16 17:28:57.810706 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.810587    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-trusted-ca-bundle\") pod \"b4486e32-1bf7-4a7e-a061-693552991f05\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") "
Sep 16 17:28:57.810706 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.810613    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-k8s-db\") pod \"b4486e32-1bf7-4a7e-a061-693552991f05\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") "
Sep 16 17:28:57.810706 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.810651    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b4486e32-1bf7-4a7e-a061-693552991f05\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") "
Sep 16 17:28:57.810706 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.810681    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4486e32-1bf7-4a7e-a061-693552991f05-config-out\") pod \"b4486e32-1bf7-4a7e-a061-693552991f05\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") "
Sep 16 17:28:57.811087 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.810708    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-k8s-rulefiles-2\") pod \"b4486e32-1bf7-4a7e-a061-693552991f05\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") "
Sep 16 17:28:57.811087 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.810749    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-metrics-client-certs\") pod \"b4486e32-1bf7-4a7e-a061-693552991f05\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") "
Sep 16 17:28:57.811087 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.810780    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-web-config\") pod \"b4486e32-1bf7-4a7e-a061-693552991f05\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") "
Sep 16 17:28:57.811087 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.810856    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz7m2\" (UniqueName: \"kubernetes.io/projected/b4486e32-1bf7-4a7e-a061-693552991f05-kube-api-access-gz7m2\") pod \"b4486e32-1bf7-4a7e-a061-693552991f05\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") "
Sep 16 17:28:57.811087 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.810883    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-kube-rbac-proxy\") pod \"b4486e32-1bf7-4a7e-a061-693552991f05\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") "
Sep 16 17:28:57.811087 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.810934    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-configmap-serving-certs-ca-bundle\") pod \"b4486e32-1bf7-4a7e-a061-693552991f05\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") "
Sep 16 17:28:57.811087 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.810959    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-grpc-tls\") pod \"b4486e32-1bf7-4a7e-a061-693552991f05\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") "
Sep 16 17:28:57.811087 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.810963    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "b4486e32-1bf7-4a7e-a061-693552991f05" (UID: "b4486e32-1bf7-4a7e-a061-693552991f05"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:28:57.811087 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.810984    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-config\") pod \"b4486e32-1bf7-4a7e-a061-693552991f05\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") "
Sep 16 17:28:57.811087 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.811021    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b4486e32-1bf7-4a7e-a061-693552991f05\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") "
Sep 16 17:28:57.811087 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.811056    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-thanos-prometheus-http-client-file\") pod \"b4486e32-1bf7-4a7e-a061-693552991f05\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") "
Sep 16 17:28:57.811605 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.811094    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-configmap-metrics-client-ca\") pod \"b4486e32-1bf7-4a7e-a061-693552991f05\" (UID: \"b4486e32-1bf7-4a7e-a061-693552991f05\") "
Sep 16 17:28:57.811605 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.811268    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "b4486e32-1bf7-4a7e-a061-693552991f05" (UID: "b4486e32-1bf7-4a7e-a061-693552991f05"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:28:57.811605 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.811379    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:28:57.811605 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.811400    2562 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:28:57.811791 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.811603    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "b4486e32-1bf7-4a7e-a061-693552991f05" (UID: "b4486e32-1bf7-4a7e-a061-693552991f05"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:28:57.811791 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.811669    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "b4486e32-1bf7-4a7e-a061-693552991f05" (UID: "b4486e32-1bf7-4a7e-a061-693552991f05"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:28:57.812750 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.811935    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "b4486e32-1bf7-4a7e-a061-693552991f05" (UID: "b4486e32-1bf7-4a7e-a061-693552991f05"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:28:57.812750 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.812402    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "b4486e32-1bf7-4a7e-a061-693552991f05" (UID: "b4486e32-1bf7-4a7e-a061-693552991f05"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:28:57.813784 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.813352    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "b4486e32-1bf7-4a7e-a061-693552991f05" (UID: "b4486e32-1bf7-4a7e-a061-693552991f05"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:28:57.813784 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.813434    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "b4486e32-1bf7-4a7e-a061-693552991f05" (UID: "b4486e32-1bf7-4a7e-a061-693552991f05"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:28:57.813784 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.813736    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "b4486e32-1bf7-4a7e-a061-693552991f05" (UID: "b4486e32-1bf7-4a7e-a061-693552991f05"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:28:57.814513 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.814488    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4486e32-1bf7-4a7e-a061-693552991f05-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b4486e32-1bf7-4a7e-a061-693552991f05" (UID: "b4486e32-1bf7-4a7e-a061-693552991f05"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:28:57.814611 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.814509    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4486e32-1bf7-4a7e-a061-693552991f05-kube-api-access-gz7m2" (OuterVolumeSpecName: "kube-api-access-gz7m2") pod "b4486e32-1bf7-4a7e-a061-693552991f05" (UID: "b4486e32-1bf7-4a7e-a061-693552991f05"). InnerVolumeSpecName "kube-api-access-gz7m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:28:57.815467 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.815438    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "b4486e32-1bf7-4a7e-a061-693552991f05" (UID: "b4486e32-1bf7-4a7e-a061-693552991f05"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:28:57.816007 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.815977    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "b4486e32-1bf7-4a7e-a061-693552991f05" (UID: "b4486e32-1bf7-4a7e-a061-693552991f05"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:28:57.816083 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.815998    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "b4486e32-1bf7-4a7e-a061-693552991f05" (UID: "b4486e32-1bf7-4a7e-a061-693552991f05"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:28:57.816129 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.816115    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-config" (OuterVolumeSpecName: "config") pod "b4486e32-1bf7-4a7e-a061-693552991f05" (UID: "b4486e32-1bf7-4a7e-a061-693552991f05"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:28:57.816236 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.816218    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4486e32-1bf7-4a7e-a061-693552991f05-config-out" (OuterVolumeSpecName: "config-out") pod "b4486e32-1bf7-4a7e-a061-693552991f05" (UID: "b4486e32-1bf7-4a7e-a061-693552991f05"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:28:57.816301 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.816286    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "b4486e32-1bf7-4a7e-a061-693552991f05" (UID: "b4486e32-1bf7-4a7e-a061-693552991f05"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:28:57.816772 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.816746    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "b4486e32-1bf7-4a7e-a061-693552991f05" (UID: "b4486e32-1bf7-4a7e-a061-693552991f05"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:28:57.817225 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.817207    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "b4486e32-1bf7-4a7e-a061-693552991f05" (UID: "b4486e32-1bf7-4a7e-a061-693552991f05"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:28:57.827729 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.827704    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-web-config" (OuterVolumeSpecName: "web-config") pod "b4486e32-1bf7-4a7e-a061-693552991f05" (UID: "b4486e32-1bf7-4a7e-a061-693552991f05"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:28:57.911913 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.911874    2562 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-metrics-client-certs\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:28:57.911913 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.911908    2562 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-web-config\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:28:57.911913 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.911920    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz7m2\" (UniqueName: \"kubernetes.io/projected/b4486e32-1bf7-4a7e-a061-693552991f05-kube-api-access-gz7m2\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:28:57.912200 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.911933    2562 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-kube-rbac-proxy\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:28:57.912200 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.911947    2562 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:28:57.912200 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.911960    2562 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-grpc-tls\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:28:57.912200 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.911973    2562 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-config\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:28:57.912200 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.911985    2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:28:57.912200 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.911998    2562 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:28:57.912200 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.912010    2562 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-configmap-metrics-client-ca\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:28:57.912200 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.912023    2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:28:57.912200 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.912034    2562 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4486e32-1bf7-4a7e-a061-693552991f05-tls-assets\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:28:57.912200 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.912047    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:28:57.912200 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.912061    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:28:57.912200 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.912074    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-k8s-db\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:28:57.912200 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.912089    2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4486e32-1bf7-4a7e-a061-693552991f05-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:28:57.912200 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.912102    2562 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4486e32-1bf7-4a7e-a061-693552991f05-config-out\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:28:57.912200 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:57.912115    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4486e32-1bf7-4a7e-a061-693552991f05-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:28:58.005926 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.005847    2562 generic.go:356] "Generic (PLEG): container finished" podID="b4486e32-1bf7-4a7e-a061-693552991f05" containerID="73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439" exitCode=0
Sep 16 17:28:58.005926 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.005873    2562 generic.go:356] "Generic (PLEG): container finished" podID="b4486e32-1bf7-4a7e-a061-693552991f05" containerID="dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c" exitCode=0
Sep 16 17:28:58.005926 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.005879    2562 generic.go:356] "Generic (PLEG): container finished" podID="b4486e32-1bf7-4a7e-a061-693552991f05" containerID="78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8" exitCode=0
Sep 16 17:28:58.005926 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.005885    2562 generic.go:356] "Generic (PLEG): container finished" podID="b4486e32-1bf7-4a7e-a061-693552991f05" containerID="217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39" exitCode=0
Sep 16 17:28:58.005926 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.005891    2562 generic.go:356] "Generic (PLEG): container finished" podID="b4486e32-1bf7-4a7e-a061-693552991f05" containerID="d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6" exitCode=0
Sep 16 17:28:58.005926 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.005897    2562 generic.go:356] "Generic (PLEG): container finished" podID="b4486e32-1bf7-4a7e-a061-693552991f05" containerID="9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab" exitCode=0
Sep 16 17:28:58.005926 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.005922    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.006246 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.005920    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4486e32-1bf7-4a7e-a061-693552991f05","Type":"ContainerDied","Data":"73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439"}
Sep 16 17:28:58.006246 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.006051    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4486e32-1bf7-4a7e-a061-693552991f05","Type":"ContainerDied","Data":"dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c"}
Sep 16 17:28:58.006246 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.006080    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4486e32-1bf7-4a7e-a061-693552991f05","Type":"ContainerDied","Data":"78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8"}
Sep 16 17:28:58.006246 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.006104    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4486e32-1bf7-4a7e-a061-693552991f05","Type":"ContainerDied","Data":"217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39"}
Sep 16 17:28:58.006246 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.006116    2562 scope.go:117] "RemoveContainer" containerID="73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439"
Sep 16 17:28:58.006246 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.006128    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4486e32-1bf7-4a7e-a061-693552991f05","Type":"ContainerDied","Data":"d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6"}
Sep 16 17:28:58.006246 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.006155    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4486e32-1bf7-4a7e-a061-693552991f05","Type":"ContainerDied","Data":"9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab"}
Sep 16 17:28:58.006246 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.006177    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4486e32-1bf7-4a7e-a061-693552991f05","Type":"ContainerDied","Data":"c16ecb8bc7cbfcaa272fa5b8362bdbfa83e1a8648aab553c9db75e175d00a751"}
Sep 16 17:28:58.014261 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.014244    2562 scope.go:117] "RemoveContainer" containerID="dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c"
Sep 16 17:28:58.021309 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.021292    2562 scope.go:117] "RemoveContainer" containerID="78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8"
Sep 16 17:28:58.027767 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.027747    2562 scope.go:117] "RemoveContainer" containerID="217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39"
Sep 16 17:28:58.033982 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.033961    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 17:28:58.034837 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.034820    2562 scope.go:117] "RemoveContainer" containerID="d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6"
Sep 16 17:28:58.040443 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.040420    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 17:28:58.042415 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.042393    2562 scope.go:117] "RemoveContainer" containerID="9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab"
Sep 16 17:28:58.049911 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.049892    2562 scope.go:117] "RemoveContainer" containerID="45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5"
Sep 16 17:28:58.056530 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.056510    2562 scope.go:117] "RemoveContainer" containerID="73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439"
Sep 16 17:28:58.056828 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:28:58.056793    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439\": container with ID starting with 73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439 not found: ID does not exist" containerID="73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439"
Sep 16 17:28:58.056905 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.056835    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439"} err="failed to get container status \"73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439\": rpc error: code = NotFound desc = could not find container \"73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439\": container with ID starting with 73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439 not found: ID does not exist"
Sep 16 17:28:58.056905 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.056855    2562 scope.go:117] "RemoveContainer" containerID="dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c"
Sep 16 17:28:58.057084 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:28:58.057069    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c\": container with ID starting with dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c not found: ID does not exist" containerID="dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c"
Sep 16 17:28:58.057128 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.057088    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c"} err="failed to get container status \"dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c\": rpc error: code = NotFound desc = could not find container \"dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c\": container with ID starting with dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c not found: ID does not exist"
Sep 16 17:28:58.057128 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.057101    2562 scope.go:117] "RemoveContainer" containerID="78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8"
Sep 16 17:28:58.057313 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:28:58.057294    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8\": container with ID starting with 78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8 not found: ID does not exist" containerID="78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8"
Sep 16 17:28:58.057377 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.057322    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8"} err="failed to get container status \"78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8\": rpc error: code = NotFound desc = could not find container \"78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8\": container with ID starting with 78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8 not found: ID does not exist"
Sep 16 17:28:58.057377 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.057347    2562 scope.go:117] "RemoveContainer" containerID="217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39"
Sep 16 17:28:58.057576 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:28:58.057560    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39\": container with ID starting with 217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39 not found: ID does not exist" containerID="217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39"
Sep 16 17:28:58.057610 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.057580    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39"} err="failed to get container status \"217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39\": rpc error: code = NotFound desc = could not find container \"217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39\": container with ID starting with 217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39 not found: ID does not exist"
Sep 16 17:28:58.057610 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.057593    2562 scope.go:117] "RemoveContainer" containerID="d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6"
Sep 16 17:28:58.057781 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:28:58.057767    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6\": container with ID starting with d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6 not found: ID does not exist" containerID="d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6"
Sep 16 17:28:58.057895 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.057783    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6"} err="failed to get container status \"d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6\": rpc error: code = NotFound desc = could not find container \"d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6\": container with ID starting with d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6 not found: ID does not exist"
Sep 16 17:28:58.057895 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.057794    2562 scope.go:117] "RemoveContainer" containerID="9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab"
Sep 16 17:28:58.058044 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:28:58.058027    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab\": container with ID starting with 9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab not found: ID does not exist" containerID="9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab"
Sep 16 17:28:58.058082 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.058050    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab"} err="failed to get container status \"9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab\": rpc error: code = NotFound desc = could not find container \"9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab\": container with ID starting with 9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab not found: ID does not exist"
Sep 16 17:28:58.058082 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.058065    2562 scope.go:117] "RemoveContainer" containerID="45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5"
Sep 16 17:28:58.058264 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:28:58.058248    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5\": container with ID starting with 45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5 not found: ID does not exist" containerID="45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5"
Sep 16 17:28:58.058300 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.058267    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5"} err="failed to get container status \"45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5\": rpc error: code = NotFound desc = could not find container \"45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5\": container with ID starting with 45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5 not found: ID does not exist"
Sep 16 17:28:58.058300 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.058279    2562 scope.go:117] "RemoveContainer" containerID="73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439"
Sep 16 17:28:58.058469 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.058454    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439"} err="failed to get container status \"73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439\": rpc error: code = NotFound desc = could not find container \"73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439\": container with ID starting with 73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439 not found: ID does not exist"
Sep 16 17:28:58.058514 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.058469    2562 scope.go:117] "RemoveContainer" containerID="dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c"
Sep 16 17:28:58.058660 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.058645    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c"} err="failed to get container status \"dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c\": rpc error: code = NotFound desc = could not find container \"dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c\": container with ID starting with dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c not found: ID does not exist"
Sep 16 17:28:58.058705 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.058660    2562 scope.go:117] "RemoveContainer" containerID="78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8"
Sep 16 17:28:58.058866 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.058850    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8"} err="failed to get container status \"78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8\": rpc error: code = NotFound desc = could not find container \"78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8\": container with ID starting with 78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8 not found: ID does not exist"
Sep 16 17:28:58.058914 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.058866    2562 scope.go:117] "RemoveContainer" containerID="217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39"
Sep 16 17:28:58.059058 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.059044    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39"} err="failed to get container status \"217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39\": rpc error: code = NotFound desc = could not find container \"217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39\": container with ID starting with 217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39 not found: ID does not exist"
Sep 16 17:28:58.059108 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.059058    2562 scope.go:117] "RemoveContainer" containerID="d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6"
Sep 16 17:28:58.059244 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.059224    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6"} err="failed to get container status \"d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6\": rpc error: code = NotFound desc = could not find container \"d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6\": container with ID starting with d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6 not found: ID does not exist"
Sep 16 17:28:58.059244 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.059241    2562 scope.go:117] "RemoveContainer" containerID="9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab"
Sep 16 17:28:58.059491 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.059465    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab"} err="failed to get container status \"9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab\": rpc error: code = NotFound desc = could not find container \"9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab\": container with ID starting with 9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab not found: ID does not exist"
Sep 16 17:28:58.059538 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.059493    2562 scope.go:117] "RemoveContainer" containerID="45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5"
Sep 16 17:28:58.059782 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.059758    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5"} err="failed to get container status \"45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5\": rpc error: code = NotFound desc = could not find container \"45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5\": container with ID starting with 45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5 not found: ID does not exist"
Sep 16 17:28:58.059782 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.059783    2562 scope.go:117] "RemoveContainer" containerID="73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439"
Sep 16 17:28:58.060121 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.060068    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439"} err="failed to get container status \"73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439\": rpc error: code = NotFound desc = could not find container \"73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439\": container with ID starting with 73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439 not found: ID does not exist"
Sep 16 17:28:58.060195 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.060123    2562 scope.go:117] "RemoveContainer" containerID="dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c"
Sep 16 17:28:58.060378 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.060359    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c"} err="failed to get container status \"dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c\": rpc error: code = NotFound desc = could not find container \"dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c\": container with ID starting with dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c not found: ID does not exist"
Sep 16 17:28:58.060420 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.060378    2562 scope.go:117] "RemoveContainer" containerID="78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8"
Sep 16 17:28:58.060607 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.060594    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8"} err="failed to get container status \"78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8\": rpc error: code = NotFound desc = could not find container \"78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8\": container with ID starting with 78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8 not found: ID does not exist"
Sep 16 17:28:58.060655 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.060607    2562 scope.go:117] "RemoveContainer" containerID="217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39"
Sep 16 17:28:58.060799 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.060783    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39"} err="failed to get container status \"217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39\": rpc error: code = NotFound desc = could not find container \"217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39\": container with ID starting with 217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39 not found: ID does not exist"
Sep 16 17:28:58.060799 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.060797    2562 scope.go:117] "RemoveContainer" containerID="d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6"
Sep 16 17:28:58.061031 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.061014    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6"} err="failed to get container status \"d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6\": rpc error: code = NotFound desc = could not find container \"d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6\": container with ID starting with d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6 not found: ID does not exist"
Sep 16 17:28:58.061075 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.061032    2562 scope.go:117] "RemoveContainer" containerID="9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab"
Sep 16 17:28:58.061228 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.061213    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab"} err="failed to get container status \"9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab\": rpc error: code = NotFound desc = could not find container \"9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab\": container with ID starting with 9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab not found: ID does not exist"
Sep 16 17:28:58.061266 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.061228    2562 scope.go:117] "RemoveContainer" containerID="45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5"
Sep 16 17:28:58.061416 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.061400    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5"} err="failed to get container status \"45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5\": rpc error: code = NotFound desc = could not find container \"45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5\": container with ID starting with 45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5 not found: ID does not exist"
Sep 16 17:28:58.061462 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.061416    2562 scope.go:117] "RemoveContainer" containerID="73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439"
Sep 16 17:28:58.061594 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.061580    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439"} err="failed to get container status \"73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439\": rpc error: code = NotFound desc = could not find container \"73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439\": container with ID starting with 73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439 not found: ID does not exist"
Sep 16 17:28:58.061636 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.061594    2562 scope.go:117] "RemoveContainer" containerID="dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c"
Sep 16 17:28:58.061766 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.061750    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c"} err="failed to get container status \"dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c\": rpc error: code = NotFound desc = could not find container \"dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c\": container with ID starting with dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c not found: ID does not exist"
Sep 16 17:28:58.061766 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.061765    2562 scope.go:117] "RemoveContainer" containerID="78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8"
Sep 16 17:28:58.062020 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.062004    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8"} err="failed to get container status \"78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8\": rpc error: code = NotFound desc = could not find container \"78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8\": container with ID starting with 78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8 not found: ID does not exist"
Sep 16 17:28:58.062063 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.062020    2562 scope.go:117] "RemoveContainer" containerID="217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39"
Sep 16 17:28:58.062200 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.062186    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39"} err="failed to get container status \"217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39\": rpc error: code = NotFound desc = could not find container \"217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39\": container with ID starting with 217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39 not found: ID does not exist"
Sep 16 17:28:58.062246 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.062201    2562 scope.go:117] "RemoveContainer" containerID="d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6"
Sep 16 17:28:58.062396 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.062380    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6"} err="failed to get container status \"d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6\": rpc error: code = NotFound desc = could not find container \"d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6\": container with ID starting with d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6 not found: ID does not exist"
Sep 16 17:28:58.062434 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.062396    2562 scope.go:117] "RemoveContainer" containerID="9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab"
Sep 16 17:28:58.062605 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.062590    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab"} err="failed to get container status \"9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab\": rpc error: code = NotFound desc = could not find container \"9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab\": container with ID starting with 9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab not found: ID does not exist"
Sep 16 17:28:58.062605 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.062604    2562 scope.go:117] "RemoveContainer" containerID="45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5"
Sep 16 17:28:58.062791 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.062776    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5"} err="failed to get container status \"45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5\": rpc error: code = NotFound desc = could not find container \"45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5\": container with ID starting with 45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5 not found: ID does not exist"
Sep 16 17:28:58.062860 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.062791    2562 scope.go:117] "RemoveContainer" containerID="73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439"
Sep 16 17:28:58.063009 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.062994    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439"} err="failed to get container status \"73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439\": rpc error: code = NotFound desc = could not find container \"73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439\": container with ID starting with 73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439 not found: ID does not exist"
Sep 16 17:28:58.063054 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.063010    2562 scope.go:117] "RemoveContainer" containerID="dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c"
Sep 16 17:28:58.063225 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.063203    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c"} err="failed to get container status \"dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c\": rpc error: code = NotFound desc = could not find container \"dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c\": container with ID starting with dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c not found: ID does not exist"
Sep 16 17:28:58.063269 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.063226    2562 scope.go:117] "RemoveContainer" containerID="78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8"
Sep 16 17:28:58.063422 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.063406    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8"} err="failed to get container status \"78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8\": rpc error: code = NotFound desc = could not find container \"78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8\": container with ID starting with 78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8 not found: ID does not exist"
Sep 16 17:28:58.063576 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.063422    2562 scope.go:117] "RemoveContainer" containerID="217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39"
Sep 16 17:28:58.063660 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.063611    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39"} err="failed to get container status \"217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39\": rpc error: code = NotFound desc = could not find container \"217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39\": container with ID starting with 217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39 not found: ID does not exist"
Sep 16 17:28:58.063660 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.063623    2562 scope.go:117] "RemoveContainer" containerID="d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6"
Sep 16 17:28:58.063829 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.063813    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6"} err="failed to get container status \"d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6\": rpc error: code = NotFound desc = could not find container \"d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6\": container with ID starting with d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6 not found: ID does not exist"
Sep 16 17:28:58.063884 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.063831    2562 scope.go:117] "RemoveContainer" containerID="9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab"
Sep 16 17:28:58.064097 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.064079    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab"} err="failed to get container status \"9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab\": rpc error: code = NotFound desc = could not find container \"9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab\": container with ID starting with 9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab not found: ID does not exist"
Sep 16 17:28:58.064162 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.064097    2562 scope.go:117] "RemoveContainer" containerID="45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5"
Sep 16 17:28:58.064295 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.064280    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5"} err="failed to get container status \"45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5\": rpc error: code = NotFound desc = could not find container \"45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5\": container with ID starting with 45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5 not found: ID does not exist"
Sep 16 17:28:58.064342 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.064294    2562 scope.go:117] "RemoveContainer" containerID="73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439"
Sep 16 17:28:58.064521 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.064503    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439"} err="failed to get container status \"73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439\": rpc error: code = NotFound desc = could not find container \"73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439\": container with ID starting with 73bc2f849eb12be2417f059cf4636201a923e7b186cf521d966c6a0aad357439 not found: ID does not exist"
Sep 16 17:28:58.064566 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.064524    2562 scope.go:117] "RemoveContainer" containerID="dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c"
Sep 16 17:28:58.064736 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.064716    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c"} err="failed to get container status \"dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c\": rpc error: code = NotFound desc = could not find container \"dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c\": container with ID starting with dec1d1d4eaf354721ab3ea0f89ec442a1e260815203166116ca6c804b9c4736c not found: ID does not exist"
Sep 16 17:28:58.064774 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.064738    2562 scope.go:117] "RemoveContainer" containerID="78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8"
Sep 16 17:28:58.064976 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.064959    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8"} err="failed to get container status \"78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8\": rpc error: code = NotFound desc = could not find container \"78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8\": container with ID starting with 78514edd73bf5259b83fd5929ca4ed0e77fddd1c676496b648a578e9c966c3d8 not found: ID does not exist"
Sep 16 17:28:58.065016 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.064977    2562 scope.go:117] "RemoveContainer" containerID="217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39"
Sep 16 17:28:58.065212 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.065195    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39"} err="failed to get container status \"217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39\": rpc error: code = NotFound desc = could not find container \"217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39\": container with ID starting with 217e3db9640bd91c469633166a10a85b332c0c6237225f6b3d1c1956a55e6f39 not found: ID does not exist"
Sep 16 17:28:58.065252 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.065214    2562 scope.go:117] "RemoveContainer" containerID="d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6"
Sep 16 17:28:58.065467 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.065452    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6"} err="failed to get container status \"d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6\": rpc error: code = NotFound desc = could not find container \"d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6\": container with ID starting with d9ff9f0edb82dd2e381e246462c7868067b766eff8816acd9ab4b43e56427ec6 not found: ID does not exist"
Sep 16 17:28:58.065512 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.065468    2562 scope.go:117] "RemoveContainer" containerID="9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab"
Sep 16 17:28:58.065668 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.065655    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab"} err="failed to get container status \"9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab\": rpc error: code = NotFound desc = could not find container \"9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab\": container with ID starting with 9173e8067d76bf6040362d6bc1bc46410ef1160d635d2a62a1c31d8ad532a2ab not found: ID does not exist"
Sep 16 17:28:58.065708 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.065670    2562 scope.go:117] "RemoveContainer" containerID="45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5"
Sep 16 17:28:58.065919 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.065903    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5"} err="failed to get container status \"45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5\": rpc error: code = NotFound desc = could not find container \"45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5\": container with ID starting with 45b0ff22956631191275b0098f7b06a5e99d10d95c6e6044a28b3f03e375c4d5 not found: ID does not exist"
Sep 16 17:28:58.074260 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.074241    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 17:28:58.074564 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.074551    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4486e32-1bf7-4a7e-a061-693552991f05" containerName="kube-rbac-proxy-thanos"
Sep 16 17:28:58.074604 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.074566    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4486e32-1bf7-4a7e-a061-693552991f05" containerName="kube-rbac-proxy-thanos"
Sep 16 17:28:58.074604 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.074581    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4486e32-1bf7-4a7e-a061-693552991f05" containerName="config-reloader"
Sep 16 17:28:58.074604 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.074587    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4486e32-1bf7-4a7e-a061-693552991f05" containerName="config-reloader"
Sep 16 17:28:58.074604 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.074594    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4486e32-1bf7-4a7e-a061-693552991f05" containerName="kube-rbac-proxy-web"
Sep 16 17:28:58.074604 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.074599    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4486e32-1bf7-4a7e-a061-693552991f05" containerName="kube-rbac-proxy-web"
Sep 16 17:28:58.074739 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.074608    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e97796b-c463-41e3-b7f9-d2a4c75b7211" containerName="registry"
Sep 16 17:28:58.074739 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.074614    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e97796b-c463-41e3-b7f9-d2a4c75b7211" containerName="registry"
Sep 16 17:28:58.074739 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.074621    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4486e32-1bf7-4a7e-a061-693552991f05" containerName="init-config-reloader"
Sep 16 17:28:58.074739 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.074626    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4486e32-1bf7-4a7e-a061-693552991f05" containerName="init-config-reloader"
Sep 16 17:28:58.074739 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.074633    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4486e32-1bf7-4a7e-a061-693552991f05" containerName="prometheus"
Sep 16 17:28:58.074739 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.074639    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4486e32-1bf7-4a7e-a061-693552991f05" containerName="prometheus"
Sep 16 17:28:58.074739 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.074644    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4486e32-1bf7-4a7e-a061-693552991f05" containerName="kube-rbac-proxy"
Sep 16 17:28:58.074739 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.074648    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4486e32-1bf7-4a7e-a061-693552991f05" containerName="kube-rbac-proxy"
Sep 16 17:28:58.074739 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.074656    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4486e32-1bf7-4a7e-a061-693552991f05" containerName="thanos-sidecar"
Sep 16 17:28:58.074739 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.074662    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4486e32-1bf7-4a7e-a061-693552991f05" containerName="thanos-sidecar"
Sep 16 17:28:58.074739 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.074703    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4486e32-1bf7-4a7e-a061-693552991f05" containerName="kube-rbac-proxy-web"
Sep 16 17:28:58.074739 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.074710    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4486e32-1bf7-4a7e-a061-693552991f05" containerName="prometheus"
Sep 16 17:28:58.074739 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.074716    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4486e32-1bf7-4a7e-a061-693552991f05" containerName="kube-rbac-proxy-thanos"
Sep 16 17:28:58.074739 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.074721    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e97796b-c463-41e3-b7f9-d2a4c75b7211" containerName="registry"
Sep 16 17:28:58.074739 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.074727    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4486e32-1bf7-4a7e-a061-693552991f05" containerName="config-reloader"
Sep 16 17:28:58.074739 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.074733    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4486e32-1bf7-4a7e-a061-693552991f05" containerName="thanos-sidecar"
Sep 16 17:28:58.074739 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.074738    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4486e32-1bf7-4a7e-a061-693552991f05" containerName="kube-rbac-proxy"
Sep 16 17:28:58.080183 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.080169    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.103989 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.103968    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 17:28:58.105007 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.104993    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 17:28:58.106888 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.106876    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 17:28:58.107868 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.107856    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 17:28:58.107917 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.107904    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7s7cp\""
Sep 16 17:28:58.108428 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.108415    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 17:28:58.108467 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.108415    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 17:28:58.111222 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.111209    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 17:28:58.113455 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.113439    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-config\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.113495 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.113461    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.113495 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.113481    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.113562 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.113513    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.113592 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.113560    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.113626 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.113590    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.113626 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.113612    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.113711 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.113654    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.113711 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.113674    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.113711 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.113691    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.113711 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.113708    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-web-config\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.113844 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.113755    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.113844 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.113784    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.113844 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.113835    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.113934 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.113851    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.113934 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.113865    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.113934 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.113879    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.113934 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.113904    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmc89\" (UniqueName: \"kubernetes.io/projected/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-kube-api-access-nmc89\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.113934 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.113927    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-config-out\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.114082 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.113953    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.117466 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.117446    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 17:28:58.117466 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.117451    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 17:28:58.117706 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.117692    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 17:28:58.118389 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.118376    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 17:28:58.123447 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.123433    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 17:28:58.134379 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.134366    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 17:28:58.147031 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.146988    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6ohvju49g81o3\""
Sep 16 17:28:58.151879 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.151863    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 17:28:58.183846 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.183828    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 17:28:58.214787 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.214763    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.214922 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.214795    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.214922 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.214831    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.214922 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.214847    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.214922 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.214868    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-web-config\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.214922 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.214887    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.214922 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.214904    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.214922 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.214921    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.215248 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.214947    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.215248 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.214973    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.215248 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.215221    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.216123 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.215652    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.216123 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.215700    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.216123 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.215704    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.216123 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.215737    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmc89\" (UniqueName: \"kubernetes.io/projected/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-kube-api-access-nmc89\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.216123 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.215782    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-config-out\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.216123 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.215831    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.216123 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.215909    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-config\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.216123 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.215965    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.216123 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.215981    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.216123 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.216000    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.216123 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.216045    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.216123 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.216084    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.216123 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.216127    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.216913 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.216665    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.217773 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.217744    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.218047 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.217986    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.218162 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.218140    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.218542 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.218493    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.220963 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.220691    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.221504 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.221480    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.221911 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.221532    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.221985 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.221946    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-config-out\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.222036 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.221981    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-web-config\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.222174 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.222152    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.222174 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.222169    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.222268 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.222245    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-config\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.222601 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.222575    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.223116 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.223102    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.240886 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.240865    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 17:28:58.263268 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.263217    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmc89\" (UniqueName: \"kubernetes.io/projected/17bbe9b1-d647-4c0a-a30e-23fabb8d0075-kube-api-access-nmc89\") pod \"prometheus-k8s-0\" (UID: \"17bbe9b1-d647-4c0a-a30e-23fabb8d0075\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.388379 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.388352    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:28:58.680623 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:58.680594    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 17:28:58.683826 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:28:58.683782    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17bbe9b1_d647_4c0a_a30e_23fabb8d0075.slice/crio-53d706eeabad205990222a54cf13e2afaf28a180af1cf35390c780c78bd7b3fb WatchSource:0}: Error finding container 53d706eeabad205990222a54cf13e2afaf28a180af1cf35390c780c78bd7b3fb: Status 404 returned error can't find the container with id 53d706eeabad205990222a54cf13e2afaf28a180af1cf35390c780c78bd7b3fb
Sep 16 17:28:59.000347 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:59.000263    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4486e32-1bf7-4a7e-a061-693552991f05" path="/var/lib/kubelet/pods/b4486e32-1bf7-4a7e-a061-693552991f05/volumes"
Sep 16 17:28:59.011599 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:59.011563    2562 generic.go:356] "Generic (PLEG): container finished" podID="17bbe9b1-d647-4c0a-a30e-23fabb8d0075" containerID="f5e4c85fd805f3df731697e5e56fab4bc80a97dd6136f9e213edab61371acafc" exitCode=0
Sep 16 17:28:59.011716 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:59.011645    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17bbe9b1-d647-4c0a-a30e-23fabb8d0075","Type":"ContainerDied","Data":"f5e4c85fd805f3df731697e5e56fab4bc80a97dd6136f9e213edab61371acafc"}
Sep 16 17:28:59.011716 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:28:59.011712    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17bbe9b1-d647-4c0a-a30e-23fabb8d0075","Type":"ContainerStarted","Data":"53d706eeabad205990222a54cf13e2afaf28a180af1cf35390c780c78bd7b3fb"}
Sep 16 17:29:00.020356 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:29:00.020322    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17bbe9b1-d647-4c0a-a30e-23fabb8d0075","Type":"ContainerStarted","Data":"1bce78181aea78afdbbf13cc9d3b2cc8267acd0e1959599e71456251ef8d4902"}
Sep 16 17:29:00.020702 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:29:00.020401    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17bbe9b1-d647-4c0a-a30e-23fabb8d0075","Type":"ContainerStarted","Data":"6370d44c139d5dd9f489f65cc508c31072b3aefe8b838b9a9feaa137234fd61b"}
Sep 16 17:29:00.020702 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:29:00.020443    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17bbe9b1-d647-4c0a-a30e-23fabb8d0075","Type":"ContainerStarted","Data":"6a831ec277b90d4bcd8a057ca614ff30a8a1f34b30a7da1fd302021c8268b60c"}
Sep 16 17:29:00.020702 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:29:00.020479    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17bbe9b1-d647-4c0a-a30e-23fabb8d0075","Type":"ContainerStarted","Data":"edad79aa6539730fb8233cf70b8df490e04acf266545ce2f4d947f3f72ea322b"}
Sep 16 17:29:00.020702 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:29:00.020520    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17bbe9b1-d647-4c0a-a30e-23fabb8d0075","Type":"ContainerStarted","Data":"3eb6708bb943b5977489c0b896e2495120c40f1c31566fe117fcd4b9c2c6c3c2"}
Sep 16 17:29:00.020702 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:29:00.020556    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17bbe9b1-d647-4c0a-a30e-23fabb8d0075","Type":"ContainerStarted","Data":"8d1c2760fb92e49b1cf3104117bc351738da13194bf825917113e00ffedb3f49"}
Sep 16 17:29:00.068771 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:29:00.068719    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.049741432 podStartE2EDuration="2.06870375s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.962318ms" isStatefulPod=false podCreationTimestamp="2026-09-16 17:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:29:00.068028283 +0000 UTC m=+263.656418270" watchObservedRunningTime="2026-09-16 17:29:00.06870375 +0000 UTC m=+263.657093780"
Sep 16 17:29:03.388698 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:29:03.388663    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:36.783429 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:29:36.783401    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-czzhc_a1ff3897-38d1-48bd-be6e-a8a698fcdfee/console-operator/1.log"
Sep 16 17:29:36.785607 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:29:36.785583    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-czzhc_a1ff3897-38d1-48bd-be6e-a8a698fcdfee/console-operator/1.log"
Sep 16 17:29:36.787695 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:29:36.787672    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6p26_1d970e5c-29fc-44af-b138-0e60309f2ac0/ovn-acl-logging/0.log"
Sep 16 17:29:36.789934 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:29:36.789911    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6p26_1d970e5c-29fc-44af-b138-0e60309f2ac0/ovn-acl-logging/0.log"
Sep 16 17:29:36.795974 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:29:36.795955    2562 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 17:29:58.389224 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:29:58.389189    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:58.404818 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:29:58.404786    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:29:59.261542 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:29:59.261512    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 17:34:36.812035 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:34:36.812008    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-czzhc_a1ff3897-38d1-48bd-be6e-a8a698fcdfee/console-operator/1.log"
Sep 16 17:34:36.812515 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:34:36.812082    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-czzhc_a1ff3897-38d1-48bd-be6e-a8a698fcdfee/console-operator/1.log"
Sep 16 17:34:36.815602 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:34:36.815572    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6p26_1d970e5c-29fc-44af-b138-0e60309f2ac0/ovn-acl-logging/0.log"
Sep 16 17:34:36.815721 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:34:36.815707    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6p26_1d970e5c-29fc-44af-b138-0e60309f2ac0/ovn-acl-logging/0.log"
Sep 16 17:36:30.042473 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:36:30.042399    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-9rm2h"]
Sep 16 17:36:30.045458 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:36:30.045442    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9rm2h"
Sep 16 17:36:30.048244 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:36:30.048222    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 17:36:30.048361 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:36:30.048295    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-4xwd9\""
Sep 16 17:36:30.048361 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:36:30.048306    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 17:36:30.048480 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:36:30.048368    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 17:36:30.049305 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:36:30.049290    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 17:36:30.054232 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:36:30.054211    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-9rm2h"]
Sep 16 17:36:30.176163 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:36:30.176115    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbhkv\" (UniqueName: \"kubernetes.io/projected/2aa7f4d5-ad0a-40cd-8395-7752814f1525-kube-api-access-mbhkv\") pod \"s3-init-9rm2h\" (UID: \"2aa7f4d5-ad0a-40cd-8395-7752814f1525\") " pod="kserve/s3-init-9rm2h"
Sep 16 17:36:30.176628 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:36:30.176413    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aa7f4d5-ad0a-40cd-8395-7752814f1525-model-cache\") pod \"s3-init-9rm2h\" (UID: \"2aa7f4d5-ad0a-40cd-8395-7752814f1525\") " pod="kserve/s3-init-9rm2h"
Sep 16 17:36:30.278070 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:36:30.278036    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aa7f4d5-ad0a-40cd-8395-7752814f1525-model-cache\") pod \"s3-init-9rm2h\" (UID: \"2aa7f4d5-ad0a-40cd-8395-7752814f1525\") " pod="kserve/s3-init-9rm2h"
Sep 16 17:36:30.278227 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:36:30.278093    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbhkv\" (UniqueName: \"kubernetes.io/projected/2aa7f4d5-ad0a-40cd-8395-7752814f1525-kube-api-access-mbhkv\") pod \"s3-init-9rm2h\" (UID: \"2aa7f4d5-ad0a-40cd-8395-7752814f1525\") " pod="kserve/s3-init-9rm2h"
Sep 16 17:36:30.278554 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:36:30.278530    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aa7f4d5-ad0a-40cd-8395-7752814f1525-model-cache\") pod \"s3-init-9rm2h\" (UID: \"2aa7f4d5-ad0a-40cd-8395-7752814f1525\") " pod="kserve/s3-init-9rm2h"
Sep 16 17:36:30.286786 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:36:30.286761    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbhkv\" (UniqueName: \"kubernetes.io/projected/2aa7f4d5-ad0a-40cd-8395-7752814f1525-kube-api-access-mbhkv\") pod \"s3-init-9rm2h\" (UID: \"2aa7f4d5-ad0a-40cd-8395-7752814f1525\") " pod="kserve/s3-init-9rm2h"
Sep 16 17:36:30.362900 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:36:30.362826    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9rm2h"
Sep 16 17:36:30.636573 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:36:30.636505    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-9rm2h"]
Sep 16 17:36:30.639528 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:36:30.639499    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2aa7f4d5_ad0a_40cd_8395_7752814f1525.slice/crio-b501f5d86e72bcb2a77e98a091a4ec4f519896da7f21cbe3ed18feda73ec4d65 WatchSource:0}: Error finding container b501f5d86e72bcb2a77e98a091a4ec4f519896da7f21cbe3ed18feda73ec4d65: Status 404 returned error can't find the container with id b501f5d86e72bcb2a77e98a091a4ec4f519896da7f21cbe3ed18feda73ec4d65
Sep 16 17:36:30.641733 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:36:30.641715    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:36:30.714307 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:36:30.714269    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9rm2h" event={"ID":"2aa7f4d5-ad0a-40cd-8395-7752814f1525","Type":"ContainerStarted","Data":"b501f5d86e72bcb2a77e98a091a4ec4f519896da7f21cbe3ed18feda73ec4d65"}
Sep 16 17:36:34.736379 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:36:34.736344    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9rm2h" event={"ID":"2aa7f4d5-ad0a-40cd-8395-7752814f1525","Type":"ContainerStarted","Data":"97558b53d195595c9cb4a755e1aa4b58b8c79c269e653c38b9998682ff286d1c"}
Sep 16 17:36:35.741951 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:36:35.741844    2562 generic.go:356] "Generic (PLEG): container finished" podID="2aa7f4d5-ad0a-40cd-8395-7752814f1525" containerID="97558b53d195595c9cb4a755e1aa4b58b8c79c269e653c38b9998682ff286d1c" exitCode=0
Sep 16 17:36:35.742474 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:36:35.741996    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9rm2h" event={"ID":"2aa7f4d5-ad0a-40cd-8395-7752814f1525","Type":"ContainerDied","Data":"97558b53d195595c9cb4a755e1aa4b58b8c79c269e653c38b9998682ff286d1c"}
Sep 16 17:36:40.131189 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:36:40.131164    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 17:36:40.767887 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:36:40.767841    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9rm2h" event={"ID":"2aa7f4d5-ad0a-40cd-8395-7752814f1525","Type":"ContainerStarted","Data":"df8b4ebb2ca5bc002ea332e8e484855b47891a4de8791d16fe7a52530069b9a4"}
Sep 16 17:36:40.785355 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:36:40.785310    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-9rm2h" podStartSLOduration=1.699023613 podStartE2EDuration="10.785296628s" totalImagesPullingTime="7.698306352s" totalInitContainerRuntime="1.387966663s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:36:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:36:40.784461605 +0000 UTC m=+724.372851591" watchObservedRunningTime="2026-09-16 17:36:40.785296628 +0000 UTC m=+724.373686611"
Sep 16 17:37:45.022560 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:37:45.022527    2562 generic.go:356] "Generic (PLEG): container finished" podID="2aa7f4d5-ad0a-40cd-8395-7752814f1525" containerID="df8b4ebb2ca5bc002ea332e8e484855b47891a4de8791d16fe7a52530069b9a4" exitCode=0
Sep 16 17:37:45.022967 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:37:45.022594    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9rm2h" event={"ID":"2aa7f4d5-ad0a-40cd-8395-7752814f1525","Type":"ContainerDied","Data":"df8b4ebb2ca5bc002ea332e8e484855b47891a4de8791d16fe7a52530069b9a4"}
Sep 16 17:37:46.149381 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:37:46.149351    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9rm2h"
Sep 16 17:37:46.307925 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:37:46.307833    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbhkv\" (UniqueName: \"kubernetes.io/projected/2aa7f4d5-ad0a-40cd-8395-7752814f1525-kube-api-access-mbhkv\") pod \"2aa7f4d5-ad0a-40cd-8395-7752814f1525\" (UID: \"2aa7f4d5-ad0a-40cd-8395-7752814f1525\") "
Sep 16 17:37:46.307925 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:37:46.307891    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aa7f4d5-ad0a-40cd-8395-7752814f1525-model-cache\") pod \"2aa7f4d5-ad0a-40cd-8395-7752814f1525\" (UID: \"2aa7f4d5-ad0a-40cd-8395-7752814f1525\") "
Sep 16 17:37:46.310020 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:37:46.309991    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa7f4d5-ad0a-40cd-8395-7752814f1525-kube-api-access-mbhkv" (OuterVolumeSpecName: "kube-api-access-mbhkv") pod "2aa7f4d5-ad0a-40cd-8395-7752814f1525" (UID: "2aa7f4d5-ad0a-40cd-8395-7752814f1525"). InnerVolumeSpecName "kube-api-access-mbhkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:37:46.326566 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:37:46.326534    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aa7f4d5-ad0a-40cd-8395-7752814f1525-model-cache" (OuterVolumeSpecName: "model-cache") pod "2aa7f4d5-ad0a-40cd-8395-7752814f1525" (UID: "2aa7f4d5-ad0a-40cd-8395-7752814f1525"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:37:46.409229 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:37:46.409193    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbhkv\" (UniqueName: \"kubernetes.io/projected/2aa7f4d5-ad0a-40cd-8395-7752814f1525-kube-api-access-mbhkv\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:37:46.409229 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:37:46.409221    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aa7f4d5-ad0a-40cd-8395-7752814f1525-model-cache\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:37:47.031124 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:37:47.031090    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9rm2h" event={"ID":"2aa7f4d5-ad0a-40cd-8395-7752814f1525","Type":"ContainerDied","Data":"b501f5d86e72bcb2a77e98a091a4ec4f519896da7f21cbe3ed18feda73ec4d65"}
Sep 16 17:37:47.031124 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:37:47.031124    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9rm2h"
Sep 16 17:37:47.031313 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:37:47.031130    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b501f5d86e72bcb2a77e98a091a4ec4f519896da7f21cbe3ed18feda73ec4d65"
Sep 16 17:38:01.373272 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.373190    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"]
Sep 16 17:38:01.373642 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.373483    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aa7f4d5-ad0a-40cd-8395-7752814f1525" containerName="download-hf-model"
Sep 16 17:38:01.373642 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.373494    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aa7f4d5-ad0a-40cd-8395-7752814f1525" containerName="download-hf-model"
Sep 16 17:38:01.373642 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.373514    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aa7f4d5-ad0a-40cd-8395-7752814f1525" containerName="s3-init"
Sep 16 17:38:01.373642 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.373520    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aa7f4d5-ad0a-40cd-8395-7752814f1525" containerName="s3-init"
Sep 16 17:38:01.373642 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.373575    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2aa7f4d5-ad0a-40cd-8395-7752814f1525" containerName="s3-init"
Sep 16 17:38:01.376902 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.376883    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"
Sep 16 17:38:01.381218 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.381193    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 17:38:01.381363 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.381266    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9bc9t\""
Sep 16 17:38:01.381363 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.381266    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 17:38:01.382317 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.382300    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 16 17:38:01.382413 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.382402    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 16 17:38:01.382469 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.382412    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 17:38:01.382515 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.382467    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:38:01.395832 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.395795    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"]
Sep 16 17:38:01.399356 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.399336    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"
Sep 16 17:38:01.402560 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.402543    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"]
Sep 16 17:38:01.402691 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.402679    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 17:38:01.439619 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.439583    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c513488b-74b8-475d-8994-004055fd2056-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm\" (UID: \"c513488b-74b8-475d-8994-004055fd2056\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"
Sep 16 17:38:01.439828 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.439636    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hrv4\" (UniqueName: \"kubernetes.io/projected/797456a3-9568-4dd4-9e73-0a8f55411436-kube-api-access-9hrv4\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mkchq\" (UID: \"797456a3-9568-4dd4-9e73-0a8f55411436\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"
Sep 16 17:38:01.439828 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.439731    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c513488b-74b8-475d-8994-004055fd2056-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm\" (UID: \"c513488b-74b8-475d-8994-004055fd2056\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"
Sep 16 17:38:01.439828 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.439773    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm8rl\" (UniqueName: \"kubernetes.io/projected/c513488b-74b8-475d-8994-004055fd2056-kube-api-access-bm8rl\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm\" (UID: \"c513488b-74b8-475d-8994-004055fd2056\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"
Sep 16 17:38:01.440014 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.439842    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/797456a3-9568-4dd4-9e73-0a8f55411436-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mkchq\" (UID: \"797456a3-9568-4dd4-9e73-0a8f55411436\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"
Sep 16 17:38:01.440014 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.439871    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/797456a3-9568-4dd4-9e73-0a8f55411436-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mkchq\" (UID: \"797456a3-9568-4dd4-9e73-0a8f55411436\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"
Sep 16 17:38:01.440014 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.439906    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/797456a3-9568-4dd4-9e73-0a8f55411436-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mkchq\" (UID: \"797456a3-9568-4dd4-9e73-0a8f55411436\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"
Sep 16 17:38:01.440014 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.439959    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/797456a3-9568-4dd4-9e73-0a8f55411436-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mkchq\" (UID: \"797456a3-9568-4dd4-9e73-0a8f55411436\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"
Sep 16 17:38:01.440014 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.439987    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c513488b-74b8-475d-8994-004055fd2056-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm\" (UID: \"c513488b-74b8-475d-8994-004055fd2056\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"
Sep 16 17:38:01.440254 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.440015    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c513488b-74b8-475d-8994-004055fd2056-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm\" (UID: \"c513488b-74b8-475d-8994-004055fd2056\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"
Sep 16 17:38:01.441921 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.441771    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"]
Sep 16 17:38:01.540390 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.540354    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm8rl\" (UniqueName: \"kubernetes.io/projected/c513488b-74b8-475d-8994-004055fd2056-kube-api-access-bm8rl\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm\" (UID: \"c513488b-74b8-475d-8994-004055fd2056\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"
Sep 16 17:38:01.540588 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.540409    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/797456a3-9568-4dd4-9e73-0a8f55411436-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mkchq\" (UID: \"797456a3-9568-4dd4-9e73-0a8f55411436\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"
Sep 16 17:38:01.540588 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.540433    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/797456a3-9568-4dd4-9e73-0a8f55411436-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mkchq\" (UID: \"797456a3-9568-4dd4-9e73-0a8f55411436\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"
Sep 16 17:38:01.540588 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.540454    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/797456a3-9568-4dd4-9e73-0a8f55411436-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mkchq\" (UID: \"797456a3-9568-4dd4-9e73-0a8f55411436\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"
Sep 16 17:38:01.540588 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.540493    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/797456a3-9568-4dd4-9e73-0a8f55411436-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mkchq\" (UID: \"797456a3-9568-4dd4-9e73-0a8f55411436\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"
Sep 16 17:38:01.540588 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.540522    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c513488b-74b8-475d-8994-004055fd2056-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm\" (UID: \"c513488b-74b8-475d-8994-004055fd2056\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"
Sep 16 17:38:01.540588 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.540565    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c513488b-74b8-475d-8994-004055fd2056-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm\" (UID: \"c513488b-74b8-475d-8994-004055fd2056\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"
Sep 16 17:38:01.540952 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.540926    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/797456a3-9568-4dd4-9e73-0a8f55411436-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mkchq\" (UID: \"797456a3-9568-4dd4-9e73-0a8f55411436\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"
Sep 16 17:38:01.541099 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.541070    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c513488b-74b8-475d-8994-004055fd2056-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm\" (UID: \"c513488b-74b8-475d-8994-004055fd2056\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"
Sep 16 17:38:01.541302 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.541279    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c513488b-74b8-475d-8994-004055fd2056-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm\" (UID: \"c513488b-74b8-475d-8994-004055fd2056\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"
Sep 16 17:38:01.541466 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.541329    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/797456a3-9568-4dd4-9e73-0a8f55411436-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mkchq\" (UID: \"797456a3-9568-4dd4-9e73-0a8f55411436\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"
Sep 16 17:38:01.541536 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.541336    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c513488b-74b8-475d-8994-004055fd2056-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm\" (UID: \"c513488b-74b8-475d-8994-004055fd2056\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"
Sep 16 17:38:01.541536 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.541397    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/797456a3-9568-4dd4-9e73-0a8f55411436-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mkchq\" (UID: \"797456a3-9568-4dd4-9e73-0a8f55411436\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"
Sep 16 17:38:01.541536 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.541452    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hrv4\" (UniqueName: \"kubernetes.io/projected/797456a3-9568-4dd4-9e73-0a8f55411436-kube-api-access-9hrv4\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mkchq\" (UID: \"797456a3-9568-4dd4-9e73-0a8f55411436\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"
Sep 16 17:38:01.541536 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.541532    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c513488b-74b8-475d-8994-004055fd2056-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm\" (UID: \"c513488b-74b8-475d-8994-004055fd2056\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"
Sep 16 17:38:01.542032 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.542014    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c513488b-74b8-475d-8994-004055fd2056-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm\" (UID: \"c513488b-74b8-475d-8994-004055fd2056\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"
Sep 16 17:38:01.543830 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.543788    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/797456a3-9568-4dd4-9e73-0a8f55411436-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mkchq\" (UID: \"797456a3-9568-4dd4-9e73-0a8f55411436\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"
Sep 16 17:38:01.543904 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.543836    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c513488b-74b8-475d-8994-004055fd2056-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm\" (UID: \"c513488b-74b8-475d-8994-004055fd2056\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"
Sep 16 17:38:01.579141 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.579112    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm8rl\" (UniqueName: \"kubernetes.io/projected/c513488b-74b8-475d-8994-004055fd2056-kube-api-access-bm8rl\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm\" (UID: \"c513488b-74b8-475d-8994-004055fd2056\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"
Sep 16 17:38:01.585696 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.585671    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hrv4\" (UniqueName: \"kubernetes.io/projected/797456a3-9568-4dd4-9e73-0a8f55411436-kube-api-access-9hrv4\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mkchq\" (UID: \"797456a3-9568-4dd4-9e73-0a8f55411436\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"
Sep 16 17:38:01.686823 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.686777    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"
Sep 16 17:38:01.709051 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.709027    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"
Sep 16 17:38:01.970074 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.970046    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"]
Sep 16 17:38:01.971564 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:38:01.971537    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc513488b_74b8_475d_8994_004055fd2056.slice/crio-b5b509214326aa182d918abf5d03735a3f33a0093d862997c69a9f5d17f25837 WatchSource:0}: Error finding container b5b509214326aa182d918abf5d03735a3f33a0093d862997c69a9f5d17f25837: Status 404 returned error can't find the container with id b5b509214326aa182d918abf5d03735a3f33a0093d862997c69a9f5d17f25837
Sep 16 17:38:01.997614 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:01.997587    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"]
Sep 16 17:38:02.005967 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:38:02.005940    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod797456a3_9568_4dd4_9e73_0a8f55411436.slice/crio-9715899354b3fb50bdc40fbb9d2a08a98e178feb0e4214d348c450732f1b0bbe WatchSource:0}: Error finding container 9715899354b3fb50bdc40fbb9d2a08a98e178feb0e4214d348c450732f1b0bbe: Status 404 returned error can't find the container with id 9715899354b3fb50bdc40fbb9d2a08a98e178feb0e4214d348c450732f1b0bbe
Sep 16 17:38:02.092092 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:02.092049    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm" event={"ID":"c513488b-74b8-475d-8994-004055fd2056","Type":"ContainerStarted","Data":"dfbac27d48b98da5d76a4590fcd763c975c692079e8ef7a04443b52ab0c2e4df"}
Sep 16 17:38:02.092289 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:02.092135    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm" event={"ID":"c513488b-74b8-475d-8994-004055fd2056","Type":"ContainerStarted","Data":"b5b509214326aa182d918abf5d03735a3f33a0093d862997c69a9f5d17f25837"}
Sep 16 17:38:02.093887 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:02.093860    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq" event={"ID":"797456a3-9568-4dd4-9e73-0a8f55411436","Type":"ContainerStarted","Data":"9ae3dfaf3333586fd7d492b8aa7172c25d07ea488246b1ac572b25a324542903"}
Sep 16 17:38:02.094066 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:02.093924    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq" event={"ID":"797456a3-9568-4dd4-9e73-0a8f55411436","Type":"ContainerStarted","Data":"9715899354b3fb50bdc40fbb9d2a08a98e178feb0e4214d348c450732f1b0bbe"}
Sep 16 17:38:05.996485 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:38:05.996455    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod797456a3_9568_4dd4_9e73_0a8f55411436.slice/crio-9ae3dfaf3333586fd7d492b8aa7172c25d07ea488246b1ac572b25a324542903.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 17:38:05.996783 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:38:05.996558    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod797456a3_9568_4dd4_9e73_0a8f55411436.slice/crio-9ae3dfaf3333586fd7d492b8aa7172c25d07ea488246b1ac572b25a324542903.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 17:38:06.112344 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:06.112307    2562 generic.go:356] "Generic (PLEG): container finished" podID="797456a3-9568-4dd4-9e73-0a8f55411436" containerID="9ae3dfaf3333586fd7d492b8aa7172c25d07ea488246b1ac572b25a324542903" exitCode=0
Sep 16 17:38:06.112492 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:06.112380    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq" event={"ID":"797456a3-9568-4dd4-9e73-0a8f55411436","Type":"ContainerDied","Data":"9ae3dfaf3333586fd7d492b8aa7172c25d07ea488246b1ac572b25a324542903"}
Sep 16 17:38:06.114085 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:06.114039    2562 generic.go:356] "Generic (PLEG): container finished" podID="c513488b-74b8-475d-8994-004055fd2056" containerID="dfbac27d48b98da5d76a4590fcd763c975c692079e8ef7a04443b52ab0c2e4df" exitCode=0
Sep 16 17:38:06.114147 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:06.114079    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm" event={"ID":"c513488b-74b8-475d-8994-004055fd2056","Type":"ContainerDied","Data":"dfbac27d48b98da5d76a4590fcd763c975c692079e8ef7a04443b52ab0c2e4df"}
Sep 16 17:38:22.216552 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:22.216513    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm" event={"ID":"c513488b-74b8-475d-8994-004055fd2056","Type":"ContainerStarted","Data":"90226c810ecac90526f3c5537882076faa463961615d7a2786abd154a2762bd5"}
Sep 16 17:38:22.218288 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:22.218263    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq" event={"ID":"797456a3-9568-4dd4-9e73-0a8f55411436","Type":"ContainerStarted","Data":"95b969019565ca4f99b1ba3b2e962dd02f8f40761a4986aa90665c7e72990251"}
Sep 16 17:38:25.234944 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:25.234899    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm" event={"ID":"c513488b-74b8-475d-8994-004055fd2056","Type":"ContainerStarted","Data":"e08a8439edbf9f612dbe1acbd06741d97d12c31c028bd0b17e15cdfee6a5bf49"}
Sep 16 17:38:25.235462 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:25.234970    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"
Sep 16 17:38:25.235462 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:25.235072    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"
Sep 16 17:38:25.236691 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:25.236662    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm" podUID="c513488b-74b8-475d-8994-004055fd2056" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 16 17:38:25.237286 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:25.237270    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq" event={"ID":"797456a3-9568-4dd4-9e73-0a8f55411436","Type":"ContainerStarted","Data":"877356f1eca5c2e6f84303a307a4f87a1ebcbe7cfea7fb2b92f4413a4464609d"}
Sep 16 17:38:25.237481 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:25.237467    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"
Sep 16 17:38:25.237551 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:25.237516    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"
Sep 16 17:38:25.238439 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:25.238416    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq" podUID="797456a3-9568-4dd4-9e73-0a8f55411436" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 16 17:38:25.282556 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:25.282497    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm" podStartSLOduration=2.299736993 podStartE2EDuration="24.282480947s" totalImagesPullingTime="18.338716365s" totalInitContainerRuntime="3.644027589s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:38:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:38:25.281383422 +0000 UTC m=+828.869773405" watchObservedRunningTime="2026-09-16 17:38:25.282480947 +0000 UTC m=+828.870871010"
Sep 16 17:38:25.320988 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:25.320930    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq" podStartSLOduration=2.032225672 podStartE2EDuration="24.320911849s" totalImagesPullingTime="18.359861933s" totalInitContainerRuntime="3.928824244s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:38:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:38:25.320433612 +0000 UTC m=+828.908823597" watchObservedRunningTime="2026-09-16 17:38:25.320911849 +0000 UTC m=+828.909301832"
Sep 16 17:38:26.241384 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:26.241330    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm" podUID="c513488b-74b8-475d-8994-004055fd2056" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 16 17:38:26.241384 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:26.241363    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq" podUID="797456a3-9568-4dd4-9e73-0a8f55411436" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 16 17:38:31.246181 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:31.246149    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"
Sep 16 17:38:31.246751 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:31.246416    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"
Sep 16 17:38:31.246864 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:31.246833    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq" podUID="797456a3-9568-4dd4-9e73-0a8f55411436" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 16 17:38:31.247207 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:31.247182    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm" podUID="c513488b-74b8-475d-8994-004055fd2056" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 16 17:38:41.247210 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:41.247168    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm" podUID="c513488b-74b8-475d-8994-004055fd2056" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 16 17:38:41.248067 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:41.247159    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq" podUID="797456a3-9568-4dd4-9e73-0a8f55411436" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 16 17:38:51.247231 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:51.247193    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm" podUID="c513488b-74b8-475d-8994-004055fd2056" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 16 17:38:51.247615 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:38:51.247193    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq" podUID="797456a3-9568-4dd4-9e73-0a8f55411436" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 16 17:39:01.247585 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:01.247544    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm" podUID="c513488b-74b8-475d-8994-004055fd2056" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 16 17:39:01.248047 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:01.247554    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq" podUID="797456a3-9568-4dd4-9e73-0a8f55411436" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 16 17:39:11.246800 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:11.246761    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq" podUID="797456a3-9568-4dd4-9e73-0a8f55411436" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 16 17:39:11.247263 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:11.247000    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm" podUID="c513488b-74b8-475d-8994-004055fd2056" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 16 17:39:21.247455 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:21.247416    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm" podUID="c513488b-74b8-475d-8994-004055fd2056" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 16 17:39:21.247853 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:21.247416    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq" podUID="797456a3-9568-4dd4-9e73-0a8f55411436" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 16 17:39:31.247698 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:31.247653    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm" podUID="c513488b-74b8-475d-8994-004055fd2056" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 16 17:39:31.248123 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:31.247653    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq" podUID="797456a3-9568-4dd4-9e73-0a8f55411436" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 16 17:39:36.844351 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:36.844326    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-czzhc_a1ff3897-38d1-48bd-be6e-a8a698fcdfee/console-operator/1.log"
Sep 16 17:39:36.845533 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:36.845512    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-czzhc_a1ff3897-38d1-48bd-be6e-a8a698fcdfee/console-operator/1.log"
Sep 16 17:39:36.848301 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:36.848281    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6p26_1d970e5c-29fc-44af-b138-0e60309f2ac0/ovn-acl-logging/0.log"
Sep 16 17:39:36.849542 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:36.849525    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6p26_1d970e5c-29fc-44af-b138-0e60309f2ac0/ovn-acl-logging/0.log"
Sep 16 17:39:41.247905 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:41.247875    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"
Sep 16 17:39:41.248272 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:41.247968    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"
Sep 16 17:39:50.818572 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:50.818537    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"]
Sep 16 17:39:50.819037 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:50.818853    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm" podUID="c513488b-74b8-475d-8994-004055fd2056" containerName="kserve-container" containerID="cri-o://90226c810ecac90526f3c5537882076faa463961615d7a2786abd154a2762bd5" gracePeriod=30
Sep 16 17:39:50.819037 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:50.818876    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm" podUID="c513488b-74b8-475d-8994-004055fd2056" containerName="kube-rbac-proxy" containerID="cri-o://e08a8439edbf9f612dbe1acbd06741d97d12c31c028bd0b17e15cdfee6a5bf49" gracePeriod=30
Sep 16 17:39:51.242060 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:51.242006    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm" podUID="c513488b-74b8-475d-8994-004055fd2056" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Sep 16 17:39:51.247327 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:51.247293    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm" podUID="c513488b-74b8-475d-8994-004055fd2056" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 16 17:39:51.594927 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:51.594844    2562 generic.go:356] "Generic (PLEG): container finished" podID="c513488b-74b8-475d-8994-004055fd2056" containerID="e08a8439edbf9f612dbe1acbd06741d97d12c31c028bd0b17e15cdfee6a5bf49" exitCode=2
Sep 16 17:39:51.594927 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:51.594902    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm" event={"ID":"c513488b-74b8-475d-8994-004055fd2056","Type":"ContainerDied","Data":"e08a8439edbf9f612dbe1acbd06741d97d12c31c028bd0b17e15cdfee6a5bf49"}
Sep 16 17:39:55.563037 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.563005    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"
Sep 16 17:39:55.611654 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.611566    2562 generic.go:356] "Generic (PLEG): container finished" podID="c513488b-74b8-475d-8994-004055fd2056" containerID="90226c810ecac90526f3c5537882076faa463961615d7a2786abd154a2762bd5" exitCode=0
Sep 16 17:39:55.611654 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.611604    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm" event={"ID":"c513488b-74b8-475d-8994-004055fd2056","Type":"ContainerDied","Data":"90226c810ecac90526f3c5537882076faa463961615d7a2786abd154a2762bd5"}
Sep 16 17:39:55.611654 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.611625    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"
Sep 16 17:39:55.611654 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.611651    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm" event={"ID":"c513488b-74b8-475d-8994-004055fd2056","Type":"ContainerDied","Data":"b5b509214326aa182d918abf5d03735a3f33a0093d862997c69a9f5d17f25837"}
Sep 16 17:39:55.612021 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.611676    2562 scope.go:117] "RemoveContainer" containerID="e08a8439edbf9f612dbe1acbd06741d97d12c31c028bd0b17e15cdfee6a5bf49"
Sep 16 17:39:55.619160 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.619141    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c513488b-74b8-475d-8994-004055fd2056-proxy-tls\") pod \"c513488b-74b8-475d-8994-004055fd2056\" (UID: \"c513488b-74b8-475d-8994-004055fd2056\") "
Sep 16 17:39:55.619269 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.619170    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bm8rl\" (UniqueName: \"kubernetes.io/projected/c513488b-74b8-475d-8994-004055fd2056-kube-api-access-bm8rl\") pod \"c513488b-74b8-475d-8994-004055fd2056\" (UID: \"c513488b-74b8-475d-8994-004055fd2056\") "
Sep 16 17:39:55.619269 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.619195    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c513488b-74b8-475d-8994-004055fd2056-cabundle-cert\") pod \"c513488b-74b8-475d-8994-004055fd2056\" (UID: \"c513488b-74b8-475d-8994-004055fd2056\") "
Sep 16 17:39:55.619269 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.619211    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c513488b-74b8-475d-8994-004055fd2056-kube-rbac-proxy-sar-config\") pod \"c513488b-74b8-475d-8994-004055fd2056\" (UID: \"c513488b-74b8-475d-8994-004055fd2056\") "
Sep 16 17:39:55.619380 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.619273    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c513488b-74b8-475d-8994-004055fd2056-kserve-provision-location\") pod \"c513488b-74b8-475d-8994-004055fd2056\" (UID: \"c513488b-74b8-475d-8994-004055fd2056\") "
Sep 16 17:39:55.619598 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.619570    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c513488b-74b8-475d-8994-004055fd2056-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c513488b-74b8-475d-8994-004055fd2056" (UID: "c513488b-74b8-475d-8994-004055fd2056"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:39:55.619736 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.619711    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c513488b-74b8-475d-8994-004055fd2056-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c513488b-74b8-475d-8994-004055fd2056" (UID: "c513488b-74b8-475d-8994-004055fd2056"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:39:55.619736 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.619729    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c513488b-74b8-475d-8994-004055fd2056-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c513488b-74b8-475d-8994-004055fd2056" (UID: "c513488b-74b8-475d-8994-004055fd2056"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:39:55.619985 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.619970    2562 scope.go:117] "RemoveContainer" containerID="90226c810ecac90526f3c5537882076faa463961615d7a2786abd154a2762bd5"
Sep 16 17:39:55.621901 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.621873    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c513488b-74b8-475d-8994-004055fd2056-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c513488b-74b8-475d-8994-004055fd2056" (UID: "c513488b-74b8-475d-8994-004055fd2056"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:39:55.621982 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.621903    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c513488b-74b8-475d-8994-004055fd2056-kube-api-access-bm8rl" (OuterVolumeSpecName: "kube-api-access-bm8rl") pod "c513488b-74b8-475d-8994-004055fd2056" (UID: "c513488b-74b8-475d-8994-004055fd2056"). InnerVolumeSpecName "kube-api-access-bm8rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:39:55.636336 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.636314    2562 scope.go:117] "RemoveContainer" containerID="dfbac27d48b98da5d76a4590fcd763c975c692079e8ef7a04443b52ab0c2e4df"
Sep 16 17:39:55.645490 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.645472    2562 scope.go:117] "RemoveContainer" containerID="e08a8439edbf9f612dbe1acbd06741d97d12c31c028bd0b17e15cdfee6a5bf49"
Sep 16 17:39:55.645758 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:39:55.645741    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e08a8439edbf9f612dbe1acbd06741d97d12c31c028bd0b17e15cdfee6a5bf49\": container with ID starting with e08a8439edbf9f612dbe1acbd06741d97d12c31c028bd0b17e15cdfee6a5bf49 not found: ID does not exist" containerID="e08a8439edbf9f612dbe1acbd06741d97d12c31c028bd0b17e15cdfee6a5bf49"
Sep 16 17:39:55.645836 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.645766    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e08a8439edbf9f612dbe1acbd06741d97d12c31c028bd0b17e15cdfee6a5bf49"} err="failed to get container status \"e08a8439edbf9f612dbe1acbd06741d97d12c31c028bd0b17e15cdfee6a5bf49\": rpc error: code = NotFound desc = could not find container \"e08a8439edbf9f612dbe1acbd06741d97d12c31c028bd0b17e15cdfee6a5bf49\": container with ID starting with e08a8439edbf9f612dbe1acbd06741d97d12c31c028bd0b17e15cdfee6a5bf49 not found: ID does not exist"
Sep 16 17:39:55.645836 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.645784    2562 scope.go:117] "RemoveContainer" containerID="90226c810ecac90526f3c5537882076faa463961615d7a2786abd154a2762bd5"
Sep 16 17:39:55.646072 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:39:55.646053    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90226c810ecac90526f3c5537882076faa463961615d7a2786abd154a2762bd5\": container with ID starting with 90226c810ecac90526f3c5537882076faa463961615d7a2786abd154a2762bd5 not found: ID does not exist" containerID="90226c810ecac90526f3c5537882076faa463961615d7a2786abd154a2762bd5"
Sep 16 17:39:55.646137 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.646081    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90226c810ecac90526f3c5537882076faa463961615d7a2786abd154a2762bd5"} err="failed to get container status \"90226c810ecac90526f3c5537882076faa463961615d7a2786abd154a2762bd5\": rpc error: code = NotFound desc = could not find container \"90226c810ecac90526f3c5537882076faa463961615d7a2786abd154a2762bd5\": container with ID starting with 90226c810ecac90526f3c5537882076faa463961615d7a2786abd154a2762bd5 not found: ID does not exist"
Sep 16 17:39:55.646137 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.646105    2562 scope.go:117] "RemoveContainer" containerID="dfbac27d48b98da5d76a4590fcd763c975c692079e8ef7a04443b52ab0c2e4df"
Sep 16 17:39:55.646371 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:39:55.646355    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfbac27d48b98da5d76a4590fcd763c975c692079e8ef7a04443b52ab0c2e4df\": container with ID starting with dfbac27d48b98da5d76a4590fcd763c975c692079e8ef7a04443b52ab0c2e4df not found: ID does not exist" containerID="dfbac27d48b98da5d76a4590fcd763c975c692079e8ef7a04443b52ab0c2e4df"
Sep 16 17:39:55.646410 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.646375    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfbac27d48b98da5d76a4590fcd763c975c692079e8ef7a04443b52ab0c2e4df"} err="failed to get container status \"dfbac27d48b98da5d76a4590fcd763c975c692079e8ef7a04443b52ab0c2e4df\": rpc error: code = NotFound desc = could not find container \"dfbac27d48b98da5d76a4590fcd763c975c692079e8ef7a04443b52ab0c2e4df\": container with ID starting with dfbac27d48b98da5d76a4590fcd763c975c692079e8ef7a04443b52ab0c2e4df not found: ID does not exist"
Sep 16 17:39:55.720578 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.720538    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c513488b-74b8-475d-8994-004055fd2056-proxy-tls\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:39:55.720578 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.720578    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm8rl\" (UniqueName: \"kubernetes.io/projected/c513488b-74b8-475d-8994-004055fd2056-kube-api-access-bm8rl\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:39:55.720789 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.720592    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c513488b-74b8-475d-8994-004055fd2056-cabundle-cert\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:39:55.720789 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.720602    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c513488b-74b8-475d-8994-004055fd2056-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:39:55.720789 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.720610    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c513488b-74b8-475d-8994-004055fd2056-kserve-provision-location\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:39:55.943980 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.943951    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"]
Sep 16 17:39:55.950235 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:55.950208    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-m4prm"]
Sep 16 17:39:57.000228 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:39:57.000198    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c513488b-74b8-475d-8994-004055fd2056" path="/var/lib/kubelet/pods/c513488b-74b8-475d-8994-004055fd2056/volumes"
Sep 16 17:41:20.996182 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:20.996093    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"]
Sep 16 17:41:20.996737 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:20.996518    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq" podUID="797456a3-9568-4dd4-9e73-0a8f55411436" containerName="kserve-container" containerID="cri-o://95b969019565ca4f99b1ba3b2e962dd02f8f40761a4986aa90665c7e72990251" gracePeriod=30
Sep 16 17:41:20.996737 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:20.996550    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq" podUID="797456a3-9568-4dd4-9e73-0a8f55411436" containerName="kube-rbac-proxy" containerID="cri-o://877356f1eca5c2e6f84303a307a4f87a1ebcbe7cfea7fb2b92f4413a4464609d" gracePeriod=30
Sep 16 17:41:21.242042 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:21.241997    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq" podUID="797456a3-9568-4dd4-9e73-0a8f55411436" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Sep 16 17:41:21.247386 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:21.247333    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq" podUID="797456a3-9568-4dd4-9e73-0a8f55411436" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 16 17:41:21.960327 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:21.960294    2562 generic.go:356] "Generic (PLEG): container finished" podID="797456a3-9568-4dd4-9e73-0a8f55411436" containerID="877356f1eca5c2e6f84303a307a4f87a1ebcbe7cfea7fb2b92f4413a4464609d" exitCode=2
Sep 16 17:41:21.960327 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:21.960333    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq" event={"ID":"797456a3-9568-4dd4-9e73-0a8f55411436","Type":"ContainerDied","Data":"877356f1eca5c2e6f84303a307a4f87a1ebcbe7cfea7fb2b92f4413a4464609d"}
Sep 16 17:41:25.744721 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:25.744691    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"
Sep 16 17:41:25.871482 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:25.871404    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/797456a3-9568-4dd4-9e73-0a8f55411436-proxy-tls\") pod \"797456a3-9568-4dd4-9e73-0a8f55411436\" (UID: \"797456a3-9568-4dd4-9e73-0a8f55411436\") "
Sep 16 17:41:25.871482 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:25.871447    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/797456a3-9568-4dd4-9e73-0a8f55411436-cabundle-cert\") pod \"797456a3-9568-4dd4-9e73-0a8f55411436\" (UID: \"797456a3-9568-4dd4-9e73-0a8f55411436\") "
Sep 16 17:41:25.871482 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:25.871481    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/797456a3-9568-4dd4-9e73-0a8f55411436-kube-rbac-proxy-sar-config\") pod \"797456a3-9568-4dd4-9e73-0a8f55411436\" (UID: \"797456a3-9568-4dd4-9e73-0a8f55411436\") "
Sep 16 17:41:25.871758 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:25.871521    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hrv4\" (UniqueName: \"kubernetes.io/projected/797456a3-9568-4dd4-9e73-0a8f55411436-kube-api-access-9hrv4\") pod \"797456a3-9568-4dd4-9e73-0a8f55411436\" (UID: \"797456a3-9568-4dd4-9e73-0a8f55411436\") "
Sep 16 17:41:25.871758 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:25.871559    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/797456a3-9568-4dd4-9e73-0a8f55411436-kserve-provision-location\") pod \"797456a3-9568-4dd4-9e73-0a8f55411436\" (UID: \"797456a3-9568-4dd4-9e73-0a8f55411436\") "
Sep 16 17:41:25.871932 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:25.871904    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/797456a3-9568-4dd4-9e73-0a8f55411436-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "797456a3-9568-4dd4-9e73-0a8f55411436" (UID: "797456a3-9568-4dd4-9e73-0a8f55411436"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:41:25.871984 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:25.871923    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/797456a3-9568-4dd4-9e73-0a8f55411436-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "797456a3-9568-4dd4-9e73-0a8f55411436" (UID: "797456a3-9568-4dd4-9e73-0a8f55411436"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:41:25.871984 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:25.871961    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/797456a3-9568-4dd4-9e73-0a8f55411436-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "797456a3-9568-4dd4-9e73-0a8f55411436" (UID: "797456a3-9568-4dd4-9e73-0a8f55411436"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:41:25.873596 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:25.873574    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/797456a3-9568-4dd4-9e73-0a8f55411436-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "797456a3-9568-4dd4-9e73-0a8f55411436" (UID: "797456a3-9568-4dd4-9e73-0a8f55411436"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:41:25.873646 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:25.873617    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/797456a3-9568-4dd4-9e73-0a8f55411436-kube-api-access-9hrv4" (OuterVolumeSpecName: "kube-api-access-9hrv4") pod "797456a3-9568-4dd4-9e73-0a8f55411436" (UID: "797456a3-9568-4dd4-9e73-0a8f55411436"). InnerVolumeSpecName "kube-api-access-9hrv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:41:25.972138 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:25.972109    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hrv4\" (UniqueName: \"kubernetes.io/projected/797456a3-9568-4dd4-9e73-0a8f55411436-kube-api-access-9hrv4\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:41:25.972138 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:25.972135    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/797456a3-9568-4dd4-9e73-0a8f55411436-kserve-provision-location\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:41:25.972332 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:25.972146    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/797456a3-9568-4dd4-9e73-0a8f55411436-proxy-tls\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:41:25.972332 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:25.972156    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/797456a3-9568-4dd4-9e73-0a8f55411436-cabundle-cert\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:41:25.972332 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:25.972165    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/797456a3-9568-4dd4-9e73-0a8f55411436-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:41:25.976316 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:25.976281    2562 generic.go:356] "Generic (PLEG): container finished" podID="797456a3-9568-4dd4-9e73-0a8f55411436" containerID="95b969019565ca4f99b1ba3b2e962dd02f8f40761a4986aa90665c7e72990251" exitCode=0
Sep 16 17:41:25.976432 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:25.976340    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"
Sep 16 17:41:25.976432 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:25.976346    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq" event={"ID":"797456a3-9568-4dd4-9e73-0a8f55411436","Type":"ContainerDied","Data":"95b969019565ca4f99b1ba3b2e962dd02f8f40761a4986aa90665c7e72990251"}
Sep 16 17:41:25.976432 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:25.976396    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq" event={"ID":"797456a3-9568-4dd4-9e73-0a8f55411436","Type":"ContainerDied","Data":"9715899354b3fb50bdc40fbb9d2a08a98e178feb0e4214d348c450732f1b0bbe"}
Sep 16 17:41:25.977119 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:25.976422    2562 scope.go:117] "RemoveContainer" containerID="877356f1eca5c2e6f84303a307a4f87a1ebcbe7cfea7fb2b92f4413a4464609d"
Sep 16 17:41:25.988267 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:25.988247    2562 scope.go:117] "RemoveContainer" containerID="95b969019565ca4f99b1ba3b2e962dd02f8f40761a4986aa90665c7e72990251"
Sep 16 17:41:25.996223 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:25.996200    2562 scope.go:117] "RemoveContainer" containerID="9ae3dfaf3333586fd7d492b8aa7172c25d07ea488246b1ac572b25a324542903"
Sep 16 17:41:26.002641 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.002616    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"]
Sep 16 17:41:26.004359 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.004338    2562 scope.go:117] "RemoveContainer" containerID="877356f1eca5c2e6f84303a307a4f87a1ebcbe7cfea7fb2b92f4413a4464609d"
Sep 16 17:41:26.004698 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:41:26.004678    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"877356f1eca5c2e6f84303a307a4f87a1ebcbe7cfea7fb2b92f4413a4464609d\": container with ID starting with 877356f1eca5c2e6f84303a307a4f87a1ebcbe7cfea7fb2b92f4413a4464609d not found: ID does not exist" containerID="877356f1eca5c2e6f84303a307a4f87a1ebcbe7cfea7fb2b92f4413a4464609d"
Sep 16 17:41:26.004765 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.004704    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"877356f1eca5c2e6f84303a307a4f87a1ebcbe7cfea7fb2b92f4413a4464609d"} err="failed to get container status \"877356f1eca5c2e6f84303a307a4f87a1ebcbe7cfea7fb2b92f4413a4464609d\": rpc error: code = NotFound desc = could not find container \"877356f1eca5c2e6f84303a307a4f87a1ebcbe7cfea7fb2b92f4413a4464609d\": container with ID starting with 877356f1eca5c2e6f84303a307a4f87a1ebcbe7cfea7fb2b92f4413a4464609d not found: ID does not exist"
Sep 16 17:41:26.004765 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.004722    2562 scope.go:117] "RemoveContainer" containerID="95b969019565ca4f99b1ba3b2e962dd02f8f40761a4986aa90665c7e72990251"
Sep 16 17:41:26.005033 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:41:26.005017    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b969019565ca4f99b1ba3b2e962dd02f8f40761a4986aa90665c7e72990251\": container with ID starting with 95b969019565ca4f99b1ba3b2e962dd02f8f40761a4986aa90665c7e72990251 not found: ID does not exist" containerID="95b969019565ca4f99b1ba3b2e962dd02f8f40761a4986aa90665c7e72990251"
Sep 16 17:41:26.005081 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.005038    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b969019565ca4f99b1ba3b2e962dd02f8f40761a4986aa90665c7e72990251"} err="failed to get container status \"95b969019565ca4f99b1ba3b2e962dd02f8f40761a4986aa90665c7e72990251\": rpc error: code = NotFound desc = could not find container \"95b969019565ca4f99b1ba3b2e962dd02f8f40761a4986aa90665c7e72990251\": container with ID starting with 95b969019565ca4f99b1ba3b2e962dd02f8f40761a4986aa90665c7e72990251 not found: ID does not exist"
Sep 16 17:41:26.005081 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.005053    2562 scope.go:117] "RemoveContainer" containerID="9ae3dfaf3333586fd7d492b8aa7172c25d07ea488246b1ac572b25a324542903"
Sep 16 17:41:26.005308 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:41:26.005289    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ae3dfaf3333586fd7d492b8aa7172c25d07ea488246b1ac572b25a324542903\": container with ID starting with 9ae3dfaf3333586fd7d492b8aa7172c25d07ea488246b1ac572b25a324542903 not found: ID does not exist" containerID="9ae3dfaf3333586fd7d492b8aa7172c25d07ea488246b1ac572b25a324542903"
Sep 16 17:41:26.005358 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.005314    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ae3dfaf3333586fd7d492b8aa7172c25d07ea488246b1ac572b25a324542903"} err="failed to get container status \"9ae3dfaf3333586fd7d492b8aa7172c25d07ea488246b1ac572b25a324542903\": rpc error: code = NotFound desc = could not find container \"9ae3dfaf3333586fd7d492b8aa7172c25d07ea488246b1ac572b25a324542903\": container with ID starting with 9ae3dfaf3333586fd7d492b8aa7172c25d07ea488246b1ac572b25a324542903 not found: ID does not exist"
Sep 16 17:41:26.005980 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.005963    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mkchq"]
Sep 16 17:41:26.232513 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.232477    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"]
Sep 16 17:41:26.232787 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.232775    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c513488b-74b8-475d-8994-004055fd2056" containerName="storage-initializer"
Sep 16 17:41:26.232856 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.232789    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c513488b-74b8-475d-8994-004055fd2056" containerName="storage-initializer"
Sep 16 17:41:26.232856 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.232815    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="797456a3-9568-4dd4-9e73-0a8f55411436" containerName="kube-rbac-proxy"
Sep 16 17:41:26.232856 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.232822    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="797456a3-9568-4dd4-9e73-0a8f55411436" containerName="kube-rbac-proxy"
Sep 16 17:41:26.232856 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.232830    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="797456a3-9568-4dd4-9e73-0a8f55411436" containerName="storage-initializer"
Sep 16 17:41:26.232856 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.232836    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="797456a3-9568-4dd4-9e73-0a8f55411436" containerName="storage-initializer"
Sep 16 17:41:26.232856 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.232844    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="797456a3-9568-4dd4-9e73-0a8f55411436" containerName="kserve-container"
Sep 16 17:41:26.232856 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.232848    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="797456a3-9568-4dd4-9e73-0a8f55411436" containerName="kserve-container"
Sep 16 17:41:26.233068 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.232859    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c513488b-74b8-475d-8994-004055fd2056" containerName="kserve-container"
Sep 16 17:41:26.233068 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.232865    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c513488b-74b8-475d-8994-004055fd2056" containerName="kserve-container"
Sep 16 17:41:26.233068 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.232871    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c513488b-74b8-475d-8994-004055fd2056" containerName="kube-rbac-proxy"
Sep 16 17:41:26.233068 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.232876    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c513488b-74b8-475d-8994-004055fd2056" containerName="kube-rbac-proxy"
Sep 16 17:41:26.233068 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.232925    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="797456a3-9568-4dd4-9e73-0a8f55411436" containerName="kserve-container"
Sep 16 17:41:26.233068 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.232932    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c513488b-74b8-475d-8994-004055fd2056" containerName="kserve-container"
Sep 16 17:41:26.233068 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.232937    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c513488b-74b8-475d-8994-004055fd2056" containerName="kube-rbac-proxy"
Sep 16 17:41:26.233068 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.232944    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="797456a3-9568-4dd4-9e73-0a8f55411436" containerName="kube-rbac-proxy"
Sep 16 17:41:26.237978 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.237955    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"
Sep 16 17:41:26.240692 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.240667    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 17:41:26.240919 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.240883    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 17:41:26.241053 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.240930    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 16 17:41:26.241053 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.240957    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 17:41:26.241053 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.240960    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9bc9t\""
Sep 16 17:41:26.241053 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.241013    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 16 17:41:26.241380 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.241362    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:41:26.248421 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.248402    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"]
Sep 16 17:41:26.375335 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.375298    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abc69283-ea0b-495f-813f-115a9b8203b7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59fbfd5bc-fgvhz\" (UID: \"abc69283-ea0b-495f-813f-115a9b8203b7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"
Sep 16 17:41:26.375335 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.375337    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7cjg\" (UniqueName: \"kubernetes.io/projected/abc69283-ea0b-495f-813f-115a9b8203b7-kube-api-access-q7cjg\") pod \"isvc-canary-stop-predictor-59fbfd5bc-fgvhz\" (UID: \"abc69283-ea0b-495f-813f-115a9b8203b7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"
Sep 16 17:41:26.375548 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.375358    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abc69283-ea0b-495f-813f-115a9b8203b7-proxy-tls\") pod \"isvc-canary-stop-predictor-59fbfd5bc-fgvhz\" (UID: \"abc69283-ea0b-495f-813f-115a9b8203b7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"
Sep 16 17:41:26.375548 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.375483    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abc69283-ea0b-495f-813f-115a9b8203b7-cabundle-cert\") pod \"isvc-canary-stop-predictor-59fbfd5bc-fgvhz\" (UID: \"abc69283-ea0b-495f-813f-115a9b8203b7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"
Sep 16 17:41:26.375548 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.375528    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abc69283-ea0b-495f-813f-115a9b8203b7-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59fbfd5bc-fgvhz\" (UID: \"abc69283-ea0b-495f-813f-115a9b8203b7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"
Sep 16 17:41:26.476864 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.476795    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abc69283-ea0b-495f-813f-115a9b8203b7-proxy-tls\") pod \"isvc-canary-stop-predictor-59fbfd5bc-fgvhz\" (UID: \"abc69283-ea0b-495f-813f-115a9b8203b7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"
Sep 16 17:41:26.477038 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.476938    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abc69283-ea0b-495f-813f-115a9b8203b7-cabundle-cert\") pod \"isvc-canary-stop-predictor-59fbfd5bc-fgvhz\" (UID: \"abc69283-ea0b-495f-813f-115a9b8203b7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"
Sep 16 17:41:26.477038 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.476981    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abc69283-ea0b-495f-813f-115a9b8203b7-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59fbfd5bc-fgvhz\" (UID: \"abc69283-ea0b-495f-813f-115a9b8203b7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"
Sep 16 17:41:26.477038 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.477030    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abc69283-ea0b-495f-813f-115a9b8203b7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59fbfd5bc-fgvhz\" (UID: \"abc69283-ea0b-495f-813f-115a9b8203b7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"
Sep 16 17:41:26.477292 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.477060    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7cjg\" (UniqueName: \"kubernetes.io/projected/abc69283-ea0b-495f-813f-115a9b8203b7-kube-api-access-q7cjg\") pod \"isvc-canary-stop-predictor-59fbfd5bc-fgvhz\" (UID: \"abc69283-ea0b-495f-813f-115a9b8203b7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"
Sep 16 17:41:26.477476 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.477452    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abc69283-ea0b-495f-813f-115a9b8203b7-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59fbfd5bc-fgvhz\" (UID: \"abc69283-ea0b-495f-813f-115a9b8203b7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"
Sep 16 17:41:26.477687 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.477661    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abc69283-ea0b-495f-813f-115a9b8203b7-cabundle-cert\") pod \"isvc-canary-stop-predictor-59fbfd5bc-fgvhz\" (UID: \"abc69283-ea0b-495f-813f-115a9b8203b7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"
Sep 16 17:41:26.477838 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.477689    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abc69283-ea0b-495f-813f-115a9b8203b7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59fbfd5bc-fgvhz\" (UID: \"abc69283-ea0b-495f-813f-115a9b8203b7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"
Sep 16 17:41:26.479446 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.479426    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abc69283-ea0b-495f-813f-115a9b8203b7-proxy-tls\") pod \"isvc-canary-stop-predictor-59fbfd5bc-fgvhz\" (UID: \"abc69283-ea0b-495f-813f-115a9b8203b7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"
Sep 16 17:41:26.485727 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.485676    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7cjg\" (UniqueName: \"kubernetes.io/projected/abc69283-ea0b-495f-813f-115a9b8203b7-kube-api-access-q7cjg\") pod \"isvc-canary-stop-predictor-59fbfd5bc-fgvhz\" (UID: \"abc69283-ea0b-495f-813f-115a9b8203b7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"
Sep 16 17:41:26.548206 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.548168    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"
Sep 16 17:41:26.818523 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.818499    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"]
Sep 16 17:41:26.820586 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:41:26.820553    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc69283_ea0b_495f_813f_115a9b8203b7.slice/crio-d443b6e7cba9ed976990111f256a75085467807548e1d5b29a9af75c5d8bbb3c WatchSource:0}: Error finding container d443b6e7cba9ed976990111f256a75085467807548e1d5b29a9af75c5d8bbb3c: Status 404 returned error can't find the container with id d443b6e7cba9ed976990111f256a75085467807548e1d5b29a9af75c5d8bbb3c
Sep 16 17:41:26.983097 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.983065    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" event={"ID":"abc69283-ea0b-495f-813f-115a9b8203b7","Type":"ContainerStarted","Data":"0a74dc011f5c0517abb74a737d095637e2d0a5544e69c5b54bebc1026a722453"}
Sep 16 17:41:26.983263 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:26.983125    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" event={"ID":"abc69283-ea0b-495f-813f-115a9b8203b7","Type":"ContainerStarted","Data":"d443b6e7cba9ed976990111f256a75085467807548e1d5b29a9af75c5d8bbb3c"}
Sep 16 17:41:27.001798 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:27.001578    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="797456a3-9568-4dd4-9e73-0a8f55411436" path="/var/lib/kubelet/pods/797456a3-9568-4dd4-9e73-0a8f55411436/volumes"
Sep 16 17:41:31.001385 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:31.001349    2562 generic.go:356] "Generic (PLEG): container finished" podID="abc69283-ea0b-495f-813f-115a9b8203b7" containerID="0a74dc011f5c0517abb74a737d095637e2d0a5544e69c5b54bebc1026a722453" exitCode=0
Sep 16 17:41:31.001743 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:31.001423    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" event={"ID":"abc69283-ea0b-495f-813f-115a9b8203b7","Type":"ContainerDied","Data":"0a74dc011f5c0517abb74a737d095637e2d0a5544e69c5b54bebc1026a722453"}
Sep 16 17:41:32.007186 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:32.007152    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" event={"ID":"abc69283-ea0b-495f-813f-115a9b8203b7","Type":"ContainerStarted","Data":"bcad3c5bbf3337eb5a5c9c1dd180e4cf61b86c1337b352fd9cce1ab8d96aa13a"}
Sep 16 17:41:32.007596 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:32.007211    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" event={"ID":"abc69283-ea0b-495f-813f-115a9b8203b7","Type":"ContainerStarted","Data":"b4d2e3925da68ebaf9f84edcccd8b3314c781e0735bc95182c54c369d922876d"}
Sep 16 17:41:32.007596 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:32.007479    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"
Sep 16 17:41:32.007703 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:32.007617    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"
Sep 16 17:41:32.008769 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:32.008748    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 16 17:41:32.030966 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:32.030918    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" podStartSLOduration=2.099075647 podStartE2EDuration="6.030904986s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.931829339s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:41:32.02852844 +0000 UTC m=+1015.616918423" watchObservedRunningTime="2026-09-16 17:41:32.030904986 +0000 UTC m=+1015.619295022"
Sep 16 17:41:33.010918 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:33.010878    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 16 17:41:38.015958 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:38.015929    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"
Sep 16 17:41:38.016502 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:38.016474    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 16 17:41:46.448538 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.448502    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"]
Sep 16 17:41:46.453293 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.453263    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"
Sep 16 17:41:46.455833 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.455786    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-c0e2d-predictor-serving-cert\""
Sep 16 17:41:46.455939 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.455862    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-c0e2d-kube-rbac-proxy-sar-config\""
Sep 16 17:41:46.463010 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.462983    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"]
Sep 16 17:41:46.543651 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.543616    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5f347d3-d6a5-4942-ac50-3205c8459da3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv\" (UID: \"f5f347d3-d6a5-4942-ac50-3205c8459da3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"
Sep 16 17:41:46.543651 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.543654    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm8ql\" (UniqueName: \"kubernetes.io/projected/f5f347d3-d6a5-4942-ac50-3205c8459da3-kube-api-access-zm8ql\") pod \"isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv\" (UID: \"f5f347d3-d6a5-4942-ac50-3205c8459da3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"
Sep 16 17:41:46.543883 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.543689    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5f347d3-d6a5-4942-ac50-3205c8459da3-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv\" (UID: \"f5f347d3-d6a5-4942-ac50-3205c8459da3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"
Sep 16 17:41:46.543883 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.543712    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5f347d3-d6a5-4942-ac50-3205c8459da3-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv\" (UID: \"f5f347d3-d6a5-4942-ac50-3205c8459da3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"
Sep 16 17:41:46.543883 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.543732    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5f347d3-d6a5-4942-ac50-3205c8459da3-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv\" (UID: \"f5f347d3-d6a5-4942-ac50-3205c8459da3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"
Sep 16 17:41:46.568785 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.568746    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"]
Sep 16 17:41:46.572751 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.572734    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"
Sep 16 17:41:46.577206 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.577180    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-c0e2d-predictor-serving-cert\""
Sep 16 17:41:46.577656 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.577638    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-c0e2d-kube-rbac-proxy-sar-config\""
Sep 16 17:41:46.593628 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.593604    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"]
Sep 16 17:41:46.644551 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.644513    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5f347d3-d6a5-4942-ac50-3205c8459da3-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv\" (UID: \"f5f347d3-d6a5-4942-ac50-3205c8459da3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"
Sep 16 17:41:46.644551 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.644551    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5f347d3-d6a5-4942-ac50-3205c8459da3-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv\" (UID: \"f5f347d3-d6a5-4942-ac50-3205c8459da3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"
Sep 16 17:41:46.644819 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.644578    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5f347d3-d6a5-4942-ac50-3205c8459da3-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv\" (UID: \"f5f347d3-d6a5-4942-ac50-3205c8459da3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"
Sep 16 17:41:46.644819 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.644619    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8\" (UID: \"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"
Sep 16 17:41:46.644819 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.644656    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnm6d\" (UniqueName: \"kubernetes.io/projected/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-kube-api-access-qnm6d\") pod \"isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8\" (UID: \"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"
Sep 16 17:41:46.644819 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.644696    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8\" (UID: \"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"
Sep 16 17:41:46.644819 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.644724    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5f347d3-d6a5-4942-ac50-3205c8459da3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv\" (UID: \"f5f347d3-d6a5-4942-ac50-3205c8459da3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"
Sep 16 17:41:46.644819 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.644751    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8\" (UID: \"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"
Sep 16 17:41:46.644819 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:41:46.644761    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-c0e2d-predictor-serving-cert" not found
Sep 16 17:41:46.644819 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.644787    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8\" (UID: \"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"
Sep 16 17:41:46.645202 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:41:46.644847    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5f347d3-d6a5-4942-ac50-3205c8459da3-proxy-tls podName:f5f347d3-d6a5-4942-ac50-3205c8459da3 nodeName:}" failed. No retries permitted until 2026-09-16 17:41:47.14482539 +0000 UTC m=+1030.733215372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f5f347d3-d6a5-4942-ac50-3205c8459da3-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv" (UID: "f5f347d3-d6a5-4942-ac50-3205c8459da3") : secret "isvc-sklearn-graph-raw-hpa-c0e2d-predictor-serving-cert" not found
Sep 16 17:41:46.645202 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.644872    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm8ql\" (UniqueName: \"kubernetes.io/projected/f5f347d3-d6a5-4942-ac50-3205c8459da3-kube-api-access-zm8ql\") pod \"isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv\" (UID: \"f5f347d3-d6a5-4942-ac50-3205c8459da3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"
Sep 16 17:41:46.645202 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.644976    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5f347d3-d6a5-4942-ac50-3205c8459da3-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv\" (UID: \"f5f347d3-d6a5-4942-ac50-3205c8459da3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"
Sep 16 17:41:46.645450 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.645429    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5f347d3-d6a5-4942-ac50-3205c8459da3-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv\" (UID: \"f5f347d3-d6a5-4942-ac50-3205c8459da3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"
Sep 16 17:41:46.645487 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.645442    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5f347d3-d6a5-4942-ac50-3205c8459da3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv\" (UID: \"f5f347d3-d6a5-4942-ac50-3205c8459da3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"
Sep 16 17:41:46.658866 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.658795    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm8ql\" (UniqueName: \"kubernetes.io/projected/f5f347d3-d6a5-4942-ac50-3205c8459da3-kube-api-access-zm8ql\") pod \"isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv\" (UID: \"f5f347d3-d6a5-4942-ac50-3205c8459da3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"
Sep 16 17:41:46.746167 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.746083    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8\" (UID: \"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"
Sep 16 17:41:46.746167 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.746121    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnm6d\" (UniqueName: \"kubernetes.io/projected/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-kube-api-access-qnm6d\") pod \"isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8\" (UID: \"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"
Sep 16 17:41:46.746167 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.746153    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8\" (UID: \"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"
Sep 16 17:41:46.746560 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.746280    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8\" (UID: \"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"
Sep 16 17:41:46.746560 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.746330    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8\" (UID: \"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"
Sep 16 17:41:46.746560 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:41:46.746462    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-c0e2d-predictor-serving-cert" not found
Sep 16 17:41:46.746560 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:41:46.746525    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-proxy-tls podName:9fd22c7d-30d0-4dd7-b7e9-da14faace1e1 nodeName:}" failed. No retries permitted until 2026-09-16 17:41:47.246506955 +0000 UTC m=+1030.834896936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8" (UID: "9fd22c7d-30d0-4dd7-b7e9-da14faace1e1") : secret "isvc-xgboost-graph-raw-hpa-c0e2d-predictor-serving-cert" not found
Sep 16 17:41:46.746825 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.746784    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8\" (UID: \"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"
Sep 16 17:41:46.746924 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.746904    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8\" (UID: \"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"
Sep 16 17:41:46.746962 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.746904    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8\" (UID: \"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"
Sep 16 17:41:46.762285 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:46.762255    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnm6d\" (UniqueName: \"kubernetes.io/projected/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-kube-api-access-qnm6d\") pod \"isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8\" (UID: \"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"
Sep 16 17:41:47.150485 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:47.150385    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5f347d3-d6a5-4942-ac50-3205c8459da3-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv\" (UID: \"f5f347d3-d6a5-4942-ac50-3205c8459da3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"
Sep 16 17:41:47.150662 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:41:47.150534    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-c0e2d-predictor-serving-cert" not found
Sep 16 17:41:47.150662 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:41:47.150600    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5f347d3-d6a5-4942-ac50-3205c8459da3-proxy-tls podName:f5f347d3-d6a5-4942-ac50-3205c8459da3 nodeName:}" failed. No retries permitted until 2026-09-16 17:41:48.150584492 +0000 UTC m=+1031.738974459 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f5f347d3-d6a5-4942-ac50-3205c8459da3-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv" (UID: "f5f347d3-d6a5-4942-ac50-3205c8459da3") : secret "isvc-sklearn-graph-raw-hpa-c0e2d-predictor-serving-cert" not found
Sep 16 17:41:47.251343 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:47.251306    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8\" (UID: \"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"
Sep 16 17:41:47.253925 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:47.253892    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8\" (UID: \"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"
Sep 16 17:41:47.482329 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:47.482282    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"
Sep 16 17:41:47.765769 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:47.765746    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"]
Sep 16 17:41:47.767774 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:41:47.767740    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd22c7d_30d0_4dd7_b7e9_da14faace1e1.slice/crio-f4b8e989629a57654b6b18f2bb9a4eadbb9af9b48718adeceee24a283037f4ef WatchSource:0}: Error finding container f4b8e989629a57654b6b18f2bb9a4eadbb9af9b48718adeceee24a283037f4ef: Status 404 returned error can't find the container with id f4b8e989629a57654b6b18f2bb9a4eadbb9af9b48718adeceee24a283037f4ef
Sep 16 17:41:48.017183 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:48.017101    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 16 17:41:48.071454 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:48.071417    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8" event={"ID":"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1","Type":"ContainerStarted","Data":"ba99e4bd4354691f464553c740556275727ef859b78e2874f63a1c9de6d17539"}
Sep 16 17:41:48.071647 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:48.071476    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8" event={"ID":"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1","Type":"ContainerStarted","Data":"f4b8e989629a57654b6b18f2bb9a4eadbb9af9b48718adeceee24a283037f4ef"}
Sep 16 17:41:48.161149 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:48.161113    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5f347d3-d6a5-4942-ac50-3205c8459da3-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv\" (UID: \"f5f347d3-d6a5-4942-ac50-3205c8459da3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"
Sep 16 17:41:48.163640 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:48.163604    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5f347d3-d6a5-4942-ac50-3205c8459da3-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv\" (UID: \"f5f347d3-d6a5-4942-ac50-3205c8459da3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"
Sep 16 17:41:48.264165 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:48.264125    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"
Sep 16 17:41:48.546238 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:48.546194    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"]
Sep 16 17:41:48.548371 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:41:48.548334    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f347d3_d6a5_4942_ac50_3205c8459da3.slice/crio-40079be846a526bd5183288bfc31e32b075f31c11fa4a480421e7b8b26c2b04a WatchSource:0}: Error finding container 40079be846a526bd5183288bfc31e32b075f31c11fa4a480421e7b8b26c2b04a: Status 404 returned error can't find the container with id 40079be846a526bd5183288bfc31e32b075f31c11fa4a480421e7b8b26c2b04a
Sep 16 17:41:49.076797 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:49.076763    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv" event={"ID":"f5f347d3-d6a5-4942-ac50-3205c8459da3","Type":"ContainerStarted","Data":"137d128fe2a5e8ac693cee3202e4c6f4afb37b6fa3e17991c2764b0158ed3252"}
Sep 16 17:41:49.077007 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:49.076856    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv" event={"ID":"f5f347d3-d6a5-4942-ac50-3205c8459da3","Type":"ContainerStarted","Data":"40079be846a526bd5183288bfc31e32b075f31c11fa4a480421e7b8b26c2b04a"}
Sep 16 17:41:52.090070 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:52.090030    2562 generic.go:356] "Generic (PLEG): container finished" podID="9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" containerID="ba99e4bd4354691f464553c740556275727ef859b78e2874f63a1c9de6d17539" exitCode=0
Sep 16 17:41:52.090457 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:52.090103    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8" event={"ID":"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1","Type":"ContainerDied","Data":"ba99e4bd4354691f464553c740556275727ef859b78e2874f63a1c9de6d17539"}
Sep 16 17:41:52.091471 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:52.091453    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:41:53.096829 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:53.096727    2562 generic.go:356] "Generic (PLEG): container finished" podID="f5f347d3-d6a5-4942-ac50-3205c8459da3" containerID="137d128fe2a5e8ac693cee3202e4c6f4afb37b6fa3e17991c2764b0158ed3252" exitCode=0
Sep 16 17:41:53.097318 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:53.096819    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv" event={"ID":"f5f347d3-d6a5-4942-ac50-3205c8459da3","Type":"ContainerDied","Data":"137d128fe2a5e8ac693cee3202e4c6f4afb37b6fa3e17991c2764b0158ed3252"}
Sep 16 17:41:55.110631 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:55.110586    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv" event={"ID":"f5f347d3-d6a5-4942-ac50-3205c8459da3","Type":"ContainerStarted","Data":"c5d08cb4c8a26b3b766b8b7855f201f6b1d7231c52a2b2767bb85e94e6973aef"}
Sep 16 17:41:55.111119 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:55.110660    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv" event={"ID":"f5f347d3-d6a5-4942-ac50-3205c8459da3","Type":"ContainerStarted","Data":"24b4f91ad28fd1b550717ea2e781acb25020c286b2a5ba6664d737225a990f03"}
Sep 16 17:41:55.111119 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:55.110998    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"
Sep 16 17:41:55.111231 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:55.111135    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"
Sep 16 17:41:55.112413 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:55.112388    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv" podUID="f5f347d3-d6a5-4942-ac50-3205c8459da3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 17:41:55.135137 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:55.135092    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv" podStartSLOduration=5.038050094 podStartE2EDuration="9.135077002s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.097026908s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:41:55.133131107 +0000 UTC m=+1038.721521094" watchObservedRunningTime="2026-09-16 17:41:55.135077002 +0000 UTC m=+1038.723467045"
Sep 16 17:41:56.118858 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:56.118791    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv" podUID="f5f347d3-d6a5-4942-ac50-3205c8459da3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 17:41:58.017138 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:41:58.017099    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 16 17:42:01.123713 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:01.123680    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"
Sep 16 17:42:01.124382 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:01.124351    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv" podUID="f5f347d3-d6a5-4942-ac50-3205c8459da3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 17:42:08.016997 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:08.016955    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 16 17:42:10.184229 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:10.184192    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8" event={"ID":"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1","Type":"ContainerStarted","Data":"beefc1e20d3788224a44580a6ea87f72cf38a67545c3dd1110aafce536af7234"}
Sep 16 17:42:10.184705 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:10.184300    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8" event={"ID":"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1","Type":"ContainerStarted","Data":"f7d38ccf096ccfe0b1a40a6fb7ccb43718999f83c03da46a7cd79b53da06d718"}
Sep 16 17:42:10.184705 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:10.184524    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"
Sep 16 17:42:10.208353 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:10.208302    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8" podStartSLOduration=3.173607883 podStartE2EDuration="24.208289339s" totalImagesPullingTime="17.289490033s" totalInitContainerRuntime="3.745191423s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:42:10.208029882 +0000 UTC m=+1053.796419866" watchObservedRunningTime="2026-09-16 17:42:10.208289339 +0000 UTC m=+1053.796679323"
Sep 16 17:42:11.125195 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:11.125150    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv" podUID="f5f347d3-d6a5-4942-ac50-3205c8459da3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 17:42:11.188627 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:11.188587    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"
Sep 16 17:42:11.189837 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:11.189798    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8" podUID="9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 16 17:42:12.194787 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:12.194742    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8" podUID="9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 16 17:42:17.199466 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:17.199433    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"
Sep 16 17:42:17.200149 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:17.200122    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8" podUID="9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 16 17:42:18.016955 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:18.016921    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 16 17:42:21.124686 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:21.124648    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv" podUID="f5f347d3-d6a5-4942-ac50-3205c8459da3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 17:42:27.200316 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:27.200276    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8" podUID="9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 16 17:42:28.016766 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:28.016729    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 16 17:42:31.124680 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:31.124635    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv" podUID="f5f347d3-d6a5-4942-ac50-3205c8459da3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 17:42:37.200342 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:37.200301    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8" podUID="9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 16 17:42:38.016414 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:38.016366    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 16 17:42:41.124939 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:41.124897    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv" podUID="f5f347d3-d6a5-4942-ac50-3205c8459da3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 17:42:47.199975 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:47.199939    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8" podUID="9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 16 17:42:48.016962 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:48.016922    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 16 17:42:48.812559 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:48.812524    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"]
Sep 16 17:42:48.812968 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:48.812755    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" containerName="kserve-container" containerID="cri-o://b4d2e3925da68ebaf9f84edcccd8b3314c781e0735bc95182c54c369d922876d" gracePeriod=30
Sep 16 17:42:48.812968 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:48.812858    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" containerName="kube-rbac-proxy" containerID="cri-o://bcad3c5bbf3337eb5a5c9c1dd180e4cf61b86c1337b352fd9cce1ab8d96aa13a" gracePeriod=30
Sep 16 17:42:48.813052 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:48.813011    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 16 17:42:49.347894 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:49.347856    2562 generic.go:356] "Generic (PLEG): container finished" podID="abc69283-ea0b-495f-813f-115a9b8203b7" containerID="bcad3c5bbf3337eb5a5c9c1dd180e4cf61b86c1337b352fd9cce1ab8d96aa13a" exitCode=2
Sep 16 17:42:49.348080 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:49.347922    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" event={"ID":"abc69283-ea0b-495f-813f-115a9b8203b7","Type":"ContainerDied","Data":"bcad3c5bbf3337eb5a5c9c1dd180e4cf61b86c1337b352fd9cce1ab8d96aa13a"}
Sep 16 17:42:51.124679 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:51.124641    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv" podUID="f5f347d3-d6a5-4942-ac50-3205c8459da3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 17:42:53.011192 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:53.011150    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 16 17:42:57.200060 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:57.200020    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8" podUID="9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 16 17:42:58.011459 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:58.011413    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 16 17:42:58.815017 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:42:58.814989    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"
Sep 16 17:43:01.125132 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:01.125090    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv" podUID="f5f347d3-d6a5-4942-ac50-3205c8459da3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 17:43:03.011495 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:03.011454    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 16 17:43:03.011903 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:03.011576    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"
Sep 16 17:43:07.203728 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:07.203674    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8" podUID="9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 16 17:43:08.012016 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:08.011974    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 16 17:43:10.996348 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:10.996304    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv" podUID="f5f347d3-d6a5-4942-ac50-3205c8459da3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 17:43:13.011303 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:13.011268    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 16 17:43:17.200620 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:17.200590    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"
Sep 16 17:43:18.011280 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:18.011240    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 16 17:43:19.456522 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.456498    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"
Sep 16 17:43:19.472846 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.472822    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abc69283-ea0b-495f-813f-115a9b8203b7-kserve-provision-location\") pod \"abc69283-ea0b-495f-813f-115a9b8203b7\" (UID: \"abc69283-ea0b-495f-813f-115a9b8203b7\") "
Sep 16 17:43:19.472951 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.472907    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abc69283-ea0b-495f-813f-115a9b8203b7-kube-rbac-proxy-sar-config\") pod \"abc69283-ea0b-495f-813f-115a9b8203b7\" (UID: \"abc69283-ea0b-495f-813f-115a9b8203b7\") "
Sep 16 17:43:19.472951 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.472933    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abc69283-ea0b-495f-813f-115a9b8203b7-proxy-tls\") pod \"abc69283-ea0b-495f-813f-115a9b8203b7\" (UID: \"abc69283-ea0b-495f-813f-115a9b8203b7\") "
Sep 16 17:43:19.473060 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.472959    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7cjg\" (UniqueName: \"kubernetes.io/projected/abc69283-ea0b-495f-813f-115a9b8203b7-kube-api-access-q7cjg\") pod \"abc69283-ea0b-495f-813f-115a9b8203b7\" (UID: \"abc69283-ea0b-495f-813f-115a9b8203b7\") "
Sep 16 17:43:19.473060 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.472990    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abc69283-ea0b-495f-813f-115a9b8203b7-cabundle-cert\") pod \"abc69283-ea0b-495f-813f-115a9b8203b7\" (UID: \"abc69283-ea0b-495f-813f-115a9b8203b7\") "
Sep 16 17:43:19.473158 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.473086    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc69283-ea0b-495f-813f-115a9b8203b7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "abc69283-ea0b-495f-813f-115a9b8203b7" (UID: "abc69283-ea0b-495f-813f-115a9b8203b7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:43:19.473299 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.473282    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abc69283-ea0b-495f-813f-115a9b8203b7-kserve-provision-location\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:43:19.473365 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.473321    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc69283-ea0b-495f-813f-115a9b8203b7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "abc69283-ea0b-495f-813f-115a9b8203b7" (UID: "abc69283-ea0b-495f-813f-115a9b8203b7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:43:19.473365 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.473334    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc69283-ea0b-495f-813f-115a9b8203b7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "abc69283-ea0b-495f-813f-115a9b8203b7" (UID: "abc69283-ea0b-495f-813f-115a9b8203b7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:43:19.475147 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.475124    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc69283-ea0b-495f-813f-115a9b8203b7-kube-api-access-q7cjg" (OuterVolumeSpecName: "kube-api-access-q7cjg") pod "abc69283-ea0b-495f-813f-115a9b8203b7" (UID: "abc69283-ea0b-495f-813f-115a9b8203b7"). InnerVolumeSpecName "kube-api-access-q7cjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:43:19.475402 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.475385    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abc69283-ea0b-495f-813f-115a9b8203b7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "abc69283-ea0b-495f-813f-115a9b8203b7" (UID: "abc69283-ea0b-495f-813f-115a9b8203b7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:43:19.479140 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.479099    2562 generic.go:356] "Generic (PLEG): container finished" podID="abc69283-ea0b-495f-813f-115a9b8203b7" containerID="b4d2e3925da68ebaf9f84edcccd8b3314c781e0735bc95182c54c369d922876d" exitCode=137
Sep 16 17:43:19.479248 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.479151    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" event={"ID":"abc69283-ea0b-495f-813f-115a9b8203b7","Type":"ContainerDied","Data":"b4d2e3925da68ebaf9f84edcccd8b3314c781e0735bc95182c54c369d922876d"}
Sep 16 17:43:19.479319 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.479237    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz" event={"ID":"abc69283-ea0b-495f-813f-115a9b8203b7","Type":"ContainerDied","Data":"d443b6e7cba9ed976990111f256a75085467807548e1d5b29a9af75c5d8bbb3c"}
Sep 16 17:43:19.479319 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.479292    2562 scope.go:117] "RemoveContainer" containerID="bcad3c5bbf3337eb5a5c9c1dd180e4cf61b86c1337b352fd9cce1ab8d96aa13a"
Sep 16 17:43:19.479415 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.479398    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"
Sep 16 17:43:19.493602 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.493584    2562 scope.go:117] "RemoveContainer" containerID="b4d2e3925da68ebaf9f84edcccd8b3314c781e0735bc95182c54c369d922876d"
Sep 16 17:43:19.501208 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.501192    2562 scope.go:117] "RemoveContainer" containerID="0a74dc011f5c0517abb74a737d095637e2d0a5544e69c5b54bebc1026a722453"
Sep 16 17:43:19.504494 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.504473    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"]
Sep 16 17:43:19.509891 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.509871    2562 scope.go:117] "RemoveContainer" containerID="bcad3c5bbf3337eb5a5c9c1dd180e4cf61b86c1337b352fd9cce1ab8d96aa13a"
Sep 16 17:43:19.510208 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:43:19.510189    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcad3c5bbf3337eb5a5c9c1dd180e4cf61b86c1337b352fd9cce1ab8d96aa13a\": container with ID starting with bcad3c5bbf3337eb5a5c9c1dd180e4cf61b86c1337b352fd9cce1ab8d96aa13a not found: ID does not exist" containerID="bcad3c5bbf3337eb5a5c9c1dd180e4cf61b86c1337b352fd9cce1ab8d96aa13a"
Sep 16 17:43:19.510257 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.510218    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcad3c5bbf3337eb5a5c9c1dd180e4cf61b86c1337b352fd9cce1ab8d96aa13a"} err="failed to get container status \"bcad3c5bbf3337eb5a5c9c1dd180e4cf61b86c1337b352fd9cce1ab8d96aa13a\": rpc error: code = NotFound desc = could not find container \"bcad3c5bbf3337eb5a5c9c1dd180e4cf61b86c1337b352fd9cce1ab8d96aa13a\": container with ID starting with bcad3c5bbf3337eb5a5c9c1dd180e4cf61b86c1337b352fd9cce1ab8d96aa13a not found: ID does not exist"
Sep 16 17:43:19.510257 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.510237    2562 scope.go:117] "RemoveContainer" containerID="b4d2e3925da68ebaf9f84edcccd8b3314c781e0735bc95182c54c369d922876d"
Sep 16 17:43:19.510503 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:43:19.510479    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4d2e3925da68ebaf9f84edcccd8b3314c781e0735bc95182c54c369d922876d\": container with ID starting with b4d2e3925da68ebaf9f84edcccd8b3314c781e0735bc95182c54c369d922876d not found: ID does not exist" containerID="b4d2e3925da68ebaf9f84edcccd8b3314c781e0735bc95182c54c369d922876d"
Sep 16 17:43:19.510554 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.510505    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4d2e3925da68ebaf9f84edcccd8b3314c781e0735bc95182c54c369d922876d"} err="failed to get container status \"b4d2e3925da68ebaf9f84edcccd8b3314c781e0735bc95182c54c369d922876d\": rpc error: code = NotFound desc = could not find container \"b4d2e3925da68ebaf9f84edcccd8b3314c781e0735bc95182c54c369d922876d\": container with ID starting with b4d2e3925da68ebaf9f84edcccd8b3314c781e0735bc95182c54c369d922876d not found: ID does not exist"
Sep 16 17:43:19.510554 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.510518    2562 scope.go:117] "RemoveContainer" containerID="0a74dc011f5c0517abb74a737d095637e2d0a5544e69c5b54bebc1026a722453"
Sep 16 17:43:19.510639 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.510603    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-fgvhz"]
Sep 16 17:43:19.510770 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:43:19.510750    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a74dc011f5c0517abb74a737d095637e2d0a5544e69c5b54bebc1026a722453\": container with ID starting with 0a74dc011f5c0517abb74a737d095637e2d0a5544e69c5b54bebc1026a722453 not found: ID does not exist" containerID="0a74dc011f5c0517abb74a737d095637e2d0a5544e69c5b54bebc1026a722453"
Sep 16 17:43:19.510850 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.510784    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a74dc011f5c0517abb74a737d095637e2d0a5544e69c5b54bebc1026a722453"} err="failed to get container status \"0a74dc011f5c0517abb74a737d095637e2d0a5544e69c5b54bebc1026a722453\": rpc error: code = NotFound desc = could not find container \"0a74dc011f5c0517abb74a737d095637e2d0a5544e69c5b54bebc1026a722453\": container with ID starting with 0a74dc011f5c0517abb74a737d095637e2d0a5544e69c5b54bebc1026a722453 not found: ID does not exist"
Sep 16 17:43:19.574237 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.574197    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abc69283-ea0b-495f-813f-115a9b8203b7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:43:19.574237 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.574237    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abc69283-ea0b-495f-813f-115a9b8203b7-proxy-tls\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:43:19.574237 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.574247    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7cjg\" (UniqueName: \"kubernetes.io/projected/abc69283-ea0b-495f-813f-115a9b8203b7-kube-api-access-q7cjg\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:43:19.574442 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:19.574256    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abc69283-ea0b-495f-813f-115a9b8203b7-cabundle-cert\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:43:21.004048 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:21.001832    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" path="/var/lib/kubelet/pods/abc69283-ea0b-495f-813f-115a9b8203b7/volumes"
Sep 16 17:43:21.004048 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:21.002404    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"
Sep 16 17:43:46.659484 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:46.659436    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"]
Sep 16 17:43:46.659941 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:46.659683    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv" podUID="f5f347d3-d6a5-4942-ac50-3205c8459da3" containerName="kserve-container" containerID="cri-o://24b4f91ad28fd1b550717ea2e781acb25020c286b2a5ba6664d737225a990f03" gracePeriod=30
Sep 16 17:43:46.659941 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:46.659727    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv" podUID="f5f347d3-d6a5-4942-ac50-3205c8459da3" containerName="kube-rbac-proxy" containerID="cri-o://c5d08cb4c8a26b3b766b8b7855f201f6b1d7231c52a2b2767bb85e94e6973aef" gracePeriod=30
Sep 16 17:43:46.675844 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:46.675816    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"]
Sep 16 17:43:46.676086 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:46.676046    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8" podUID="9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" containerName="kserve-container" containerID="cri-o://f7d38ccf096ccfe0b1a40a6fb7ccb43718999f83c03da46a7cd79b53da06d718" gracePeriod=30
Sep 16 17:43:46.676148 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:46.676118    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8" podUID="9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" containerName="kube-rbac-proxy" containerID="cri-o://beefc1e20d3788224a44580a6ea87f72cf38a67545c3dd1110aafce536af7234" gracePeriod=30
Sep 16 17:43:47.195045 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:47.195003    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8" podUID="9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 16 17:43:47.200374 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:47.200345    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8" podUID="9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 16 17:43:47.594903 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:47.594798    2562 generic.go:356] "Generic (PLEG): container finished" podID="f5f347d3-d6a5-4942-ac50-3205c8459da3" containerID="c5d08cb4c8a26b3b766b8b7855f201f6b1d7231c52a2b2767bb85e94e6973aef" exitCode=2
Sep 16 17:43:47.594903 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:47.594872    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv" event={"ID":"f5f347d3-d6a5-4942-ac50-3205c8459da3","Type":"ContainerDied","Data":"c5d08cb4c8a26b3b766b8b7855f201f6b1d7231c52a2b2767bb85e94e6973aef"}
Sep 16 17:43:47.596773 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:47.596751    2562 generic.go:356] "Generic (PLEG): container finished" podID="9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" containerID="beefc1e20d3788224a44580a6ea87f72cf38a67545c3dd1110aafce536af7234" exitCode=2
Sep 16 17:43:47.596921 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:47.596796    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8" event={"ID":"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1","Type":"ContainerDied","Data":"beefc1e20d3788224a44580a6ea87f72cf38a67545c3dd1110aafce536af7234"}
Sep 16 17:43:50.323727 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.323702    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"
Sep 16 17:43:50.429763 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.429729    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-kube-rbac-proxy-sar-config\") pod \"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1\" (UID: \"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1\") "
Sep 16 17:43:50.429973 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.429846    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-proxy-tls\") pod \"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1\" (UID: \"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1\") "
Sep 16 17:43:50.429973 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.429889    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnm6d\" (UniqueName: \"kubernetes.io/projected/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-kube-api-access-qnm6d\") pod \"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1\" (UID: \"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1\") "
Sep 16 17:43:50.429973 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.429916    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-cabundle-cert\") pod \"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1\" (UID: \"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1\") "
Sep 16 17:43:50.429973 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.429955    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-kserve-provision-location\") pod \"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1\" (UID: \"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1\") "
Sep 16 17:43:50.430309 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.430286    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" (UID: "9fd22c7d-30d0-4dd7-b7e9-da14faace1e1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:43:50.430375 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.430326    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" (UID: "9fd22c7d-30d0-4dd7-b7e9-da14faace1e1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:43:50.430411 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.430389    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" (UID: "9fd22c7d-30d0-4dd7-b7e9-da14faace1e1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:43:50.432093 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.432073    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-kube-api-access-qnm6d" (OuterVolumeSpecName: "kube-api-access-qnm6d") pod "9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" (UID: "9fd22c7d-30d0-4dd7-b7e9-da14faace1e1"). InnerVolumeSpecName "kube-api-access-qnm6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:43:50.432163 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.432075    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" (UID: "9fd22c7d-30d0-4dd7-b7e9-da14faace1e1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:43:50.531479 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.531439    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-cabundle-cert\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:43:50.531479 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.531473    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-kserve-provision-location\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:43:50.531479 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.531485    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:43:50.531708 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.531493    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-proxy-tls\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:43:50.531708 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.531502    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnm6d\" (UniqueName: \"kubernetes.io/projected/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1-kube-api-access-qnm6d\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:43:50.610852 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.610818    2562 generic.go:356] "Generic (PLEG): container finished" podID="9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" containerID="f7d38ccf096ccfe0b1a40a6fb7ccb43718999f83c03da46a7cd79b53da06d718" exitCode=0
Sep 16 17:43:50.611012 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.610875    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"
Sep 16 17:43:50.611012 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.610879    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8" event={"ID":"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1","Type":"ContainerDied","Data":"f7d38ccf096ccfe0b1a40a6fb7ccb43718999f83c03da46a7cd79b53da06d718"}
Sep 16 17:43:50.611084 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.611013    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8" event={"ID":"9fd22c7d-30d0-4dd7-b7e9-da14faace1e1","Type":"ContainerDied","Data":"f4b8e989629a57654b6b18f2bb9a4eadbb9af9b48718adeceee24a283037f4ef"}
Sep 16 17:43:50.611084 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.611039    2562 scope.go:117] "RemoveContainer" containerID="beefc1e20d3788224a44580a6ea87f72cf38a67545c3dd1110aafce536af7234"
Sep 16 17:43:50.619345 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.619323    2562 scope.go:117] "RemoveContainer" containerID="f7d38ccf096ccfe0b1a40a6fb7ccb43718999f83c03da46a7cd79b53da06d718"
Sep 16 17:43:50.626776 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.626757    2562 scope.go:117] "RemoveContainer" containerID="ba99e4bd4354691f464553c740556275727ef859b78e2874f63a1c9de6d17539"
Sep 16 17:43:50.633769 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.633745    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"]
Sep 16 17:43:50.634171 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.634153    2562 scope.go:117] "RemoveContainer" containerID="beefc1e20d3788224a44580a6ea87f72cf38a67545c3dd1110aafce536af7234"
Sep 16 17:43:50.634434 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:43:50.634414    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beefc1e20d3788224a44580a6ea87f72cf38a67545c3dd1110aafce536af7234\": container with ID starting with beefc1e20d3788224a44580a6ea87f72cf38a67545c3dd1110aafce536af7234 not found: ID does not exist" containerID="beefc1e20d3788224a44580a6ea87f72cf38a67545c3dd1110aafce536af7234"
Sep 16 17:43:50.634483 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.634442    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beefc1e20d3788224a44580a6ea87f72cf38a67545c3dd1110aafce536af7234"} err="failed to get container status \"beefc1e20d3788224a44580a6ea87f72cf38a67545c3dd1110aafce536af7234\": rpc error: code = NotFound desc = could not find container \"beefc1e20d3788224a44580a6ea87f72cf38a67545c3dd1110aafce536af7234\": container with ID starting with beefc1e20d3788224a44580a6ea87f72cf38a67545c3dd1110aafce536af7234 not found: ID does not exist"
Sep 16 17:43:50.634483 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.634458    2562 scope.go:117] "RemoveContainer" containerID="f7d38ccf096ccfe0b1a40a6fb7ccb43718999f83c03da46a7cd79b53da06d718"
Sep 16 17:43:50.634676 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:43:50.634662    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7d38ccf096ccfe0b1a40a6fb7ccb43718999f83c03da46a7cd79b53da06d718\": container with ID starting with f7d38ccf096ccfe0b1a40a6fb7ccb43718999f83c03da46a7cd79b53da06d718 not found: ID does not exist" containerID="f7d38ccf096ccfe0b1a40a6fb7ccb43718999f83c03da46a7cd79b53da06d718"
Sep 16 17:43:50.634716 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.634679    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7d38ccf096ccfe0b1a40a6fb7ccb43718999f83c03da46a7cd79b53da06d718"} err="failed to get container status \"f7d38ccf096ccfe0b1a40a6fb7ccb43718999f83c03da46a7cd79b53da06d718\": rpc error: code = NotFound desc = could not find container \"f7d38ccf096ccfe0b1a40a6fb7ccb43718999f83c03da46a7cd79b53da06d718\": container with ID starting with f7d38ccf096ccfe0b1a40a6fb7ccb43718999f83c03da46a7cd79b53da06d718 not found: ID does not exist"
Sep 16 17:43:50.634716 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.634692    2562 scope.go:117] "RemoveContainer" containerID="ba99e4bd4354691f464553c740556275727ef859b78e2874f63a1c9de6d17539"
Sep 16 17:43:50.634953 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:43:50.634936    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba99e4bd4354691f464553c740556275727ef859b78e2874f63a1c9de6d17539\": container with ID starting with ba99e4bd4354691f464553c740556275727ef859b78e2874f63a1c9de6d17539 not found: ID does not exist" containerID="ba99e4bd4354691f464553c740556275727ef859b78e2874f63a1c9de6d17539"
Sep 16 17:43:50.635000 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.634967    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba99e4bd4354691f464553c740556275727ef859b78e2874f63a1c9de6d17539"} err="failed to get container status \"ba99e4bd4354691f464553c740556275727ef859b78e2874f63a1c9de6d17539\": rpc error: code = NotFound desc = could not find container \"ba99e4bd4354691f464553c740556275727ef859b78e2874f63a1c9de6d17539\": container with ID starting with ba99e4bd4354691f464553c740556275727ef859b78e2874f63a1c9de6d17539 not found: ID does not exist"
Sep 16 17:43:50.638959 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.638934    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c0e2d-predictor-987857857-2v2m8"]
Sep 16 17:43:50.996413 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:50.996370    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv" podUID="f5f347d3-d6a5-4942-ac50-3205c8459da3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 17:43:51.001288 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.001257    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" path="/var/lib/kubelet/pods/9fd22c7d-30d0-4dd7-b7e9-da14faace1e1/volumes"
Sep 16 17:43:51.119961 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.119924    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv" podUID="f5f347d3-d6a5-4942-ac50-3205c8459da3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 16 17:43:51.303137 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.303112    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"
Sep 16 17:43:51.439791 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.439756    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5f347d3-d6a5-4942-ac50-3205c8459da3-kube-rbac-proxy-sar-config\") pod \"f5f347d3-d6a5-4942-ac50-3205c8459da3\" (UID: \"f5f347d3-d6a5-4942-ac50-3205c8459da3\") "
Sep 16 17:43:51.440234 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.439843    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5f347d3-d6a5-4942-ac50-3205c8459da3-kserve-provision-location\") pod \"f5f347d3-d6a5-4942-ac50-3205c8459da3\" (UID: \"f5f347d3-d6a5-4942-ac50-3205c8459da3\") "
Sep 16 17:43:51.440234 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.439887    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm8ql\" (UniqueName: \"kubernetes.io/projected/f5f347d3-d6a5-4942-ac50-3205c8459da3-kube-api-access-zm8ql\") pod \"f5f347d3-d6a5-4942-ac50-3205c8459da3\" (UID: \"f5f347d3-d6a5-4942-ac50-3205c8459da3\") "
Sep 16 17:43:51.440234 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.439918    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5f347d3-d6a5-4942-ac50-3205c8459da3-proxy-tls\") pod \"f5f347d3-d6a5-4942-ac50-3205c8459da3\" (UID: \"f5f347d3-d6a5-4942-ac50-3205c8459da3\") "
Sep 16 17:43:51.440234 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.439938    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5f347d3-d6a5-4942-ac50-3205c8459da3-cabundle-cert\") pod \"f5f347d3-d6a5-4942-ac50-3205c8459da3\" (UID: \"f5f347d3-d6a5-4942-ac50-3205c8459da3\") "
Sep 16 17:43:51.440234 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.440147    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5f347d3-d6a5-4942-ac50-3205c8459da3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f5f347d3-d6a5-4942-ac50-3205c8459da3" (UID: "f5f347d3-d6a5-4942-ac50-3205c8459da3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:43:51.440452 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.440196    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f347d3-d6a5-4942-ac50-3205c8459da3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f5f347d3-d6a5-4942-ac50-3205c8459da3" (UID: "f5f347d3-d6a5-4942-ac50-3205c8459da3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:43:51.440452 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.440340    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5f347d3-d6a5-4942-ac50-3205c8459da3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f5f347d3-d6a5-4942-ac50-3205c8459da3" (UID: "f5f347d3-d6a5-4942-ac50-3205c8459da3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:43:51.441998 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.441977    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5f347d3-d6a5-4942-ac50-3205c8459da3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f5f347d3-d6a5-4942-ac50-3205c8459da3" (UID: "f5f347d3-d6a5-4942-ac50-3205c8459da3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:43:51.442087 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.442068    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f347d3-d6a5-4942-ac50-3205c8459da3-kube-api-access-zm8ql" (OuterVolumeSpecName: "kube-api-access-zm8ql") pod "f5f347d3-d6a5-4942-ac50-3205c8459da3" (UID: "f5f347d3-d6a5-4942-ac50-3205c8459da3"). InnerVolumeSpecName "kube-api-access-zm8ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:43:51.541369 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.541283    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5f347d3-d6a5-4942-ac50-3205c8459da3-kserve-provision-location\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:43:51.541369 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.541313    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm8ql\" (UniqueName: \"kubernetes.io/projected/f5f347d3-d6a5-4942-ac50-3205c8459da3-kube-api-access-zm8ql\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:43:51.541369 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.541322    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5f347d3-d6a5-4942-ac50-3205c8459da3-proxy-tls\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:43:51.541369 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.541332    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5f347d3-d6a5-4942-ac50-3205c8459da3-cabundle-cert\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:43:51.541369 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.541340    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5f347d3-d6a5-4942-ac50-3205c8459da3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:43:51.619938 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.619904    2562 generic.go:356] "Generic (PLEG): container finished" podID="f5f347d3-d6a5-4942-ac50-3205c8459da3" containerID="24b4f91ad28fd1b550717ea2e781acb25020c286b2a5ba6664d737225a990f03" exitCode=0
Sep 16 17:43:51.620126 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.619961    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"
Sep 16 17:43:51.620126 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.619990    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv" event={"ID":"f5f347d3-d6a5-4942-ac50-3205c8459da3","Type":"ContainerDied","Data":"24b4f91ad28fd1b550717ea2e781acb25020c286b2a5ba6664d737225a990f03"}
Sep 16 17:43:51.620126 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.620068    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv" event={"ID":"f5f347d3-d6a5-4942-ac50-3205c8459da3","Type":"ContainerDied","Data":"40079be846a526bd5183288bfc31e32b075f31c11fa4a480421e7b8b26c2b04a"}
Sep 16 17:43:51.620126 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.620107    2562 scope.go:117] "RemoveContainer" containerID="c5d08cb4c8a26b3b766b8b7855f201f6b1d7231c52a2b2767bb85e94e6973aef"
Sep 16 17:43:51.628871 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.628848    2562 scope.go:117] "RemoveContainer" containerID="24b4f91ad28fd1b550717ea2e781acb25020c286b2a5ba6664d737225a990f03"
Sep 16 17:43:51.636211 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.636195    2562 scope.go:117] "RemoveContainer" containerID="137d128fe2a5e8ac693cee3202e4c6f4afb37b6fa3e17991c2764b0158ed3252"
Sep 16 17:43:51.643083 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.643065    2562 scope.go:117] "RemoveContainer" containerID="c5d08cb4c8a26b3b766b8b7855f201f6b1d7231c52a2b2767bb85e94e6973aef"
Sep 16 17:43:51.643354 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:43:51.643328    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5d08cb4c8a26b3b766b8b7855f201f6b1d7231c52a2b2767bb85e94e6973aef\": container with ID starting with c5d08cb4c8a26b3b766b8b7855f201f6b1d7231c52a2b2767bb85e94e6973aef not found: ID does not exist" containerID="c5d08cb4c8a26b3b766b8b7855f201f6b1d7231c52a2b2767bb85e94e6973aef"
Sep 16 17:43:51.643392 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.643358    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5d08cb4c8a26b3b766b8b7855f201f6b1d7231c52a2b2767bb85e94e6973aef"} err="failed to get container status \"c5d08cb4c8a26b3b766b8b7855f201f6b1d7231c52a2b2767bb85e94e6973aef\": rpc error: code = NotFound desc = could not find container \"c5d08cb4c8a26b3b766b8b7855f201f6b1d7231c52a2b2767bb85e94e6973aef\": container with ID starting with c5d08cb4c8a26b3b766b8b7855f201f6b1d7231c52a2b2767bb85e94e6973aef not found: ID does not exist"
Sep 16 17:43:51.643392 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.643374    2562 scope.go:117] "RemoveContainer" containerID="24b4f91ad28fd1b550717ea2e781acb25020c286b2a5ba6664d737225a990f03"
Sep 16 17:43:51.643645 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:43:51.643629    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24b4f91ad28fd1b550717ea2e781acb25020c286b2a5ba6664d737225a990f03\": container with ID starting with 24b4f91ad28fd1b550717ea2e781acb25020c286b2a5ba6664d737225a990f03 not found: ID does not exist" containerID="24b4f91ad28fd1b550717ea2e781acb25020c286b2a5ba6664d737225a990f03"
Sep 16 17:43:51.643690 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.643659    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24b4f91ad28fd1b550717ea2e781acb25020c286b2a5ba6664d737225a990f03"} err="failed to get container status \"24b4f91ad28fd1b550717ea2e781acb25020c286b2a5ba6664d737225a990f03\": rpc error: code = NotFound desc = could not find container \"24b4f91ad28fd1b550717ea2e781acb25020c286b2a5ba6664d737225a990f03\": container with ID starting with 24b4f91ad28fd1b550717ea2e781acb25020c286b2a5ba6664d737225a990f03 not found: ID does not exist"
Sep 16 17:43:51.643690 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.643675    2562 scope.go:117] "RemoveContainer" containerID="137d128fe2a5e8ac693cee3202e4c6f4afb37b6fa3e17991c2764b0158ed3252"
Sep 16 17:43:51.643918 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:43:51.643901    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"137d128fe2a5e8ac693cee3202e4c6f4afb37b6fa3e17991c2764b0158ed3252\": container with ID starting with 137d128fe2a5e8ac693cee3202e4c6f4afb37b6fa3e17991c2764b0158ed3252 not found: ID does not exist" containerID="137d128fe2a5e8ac693cee3202e4c6f4afb37b6fa3e17991c2764b0158ed3252"
Sep 16 17:43:51.643969 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.643922    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"137d128fe2a5e8ac693cee3202e4c6f4afb37b6fa3e17991c2764b0158ed3252"} err="failed to get container status \"137d128fe2a5e8ac693cee3202e4c6f4afb37b6fa3e17991c2764b0158ed3252\": rpc error: code = NotFound desc = could not find container \"137d128fe2a5e8ac693cee3202e4c6f4afb37b6fa3e17991c2764b0158ed3252\": container with ID starting with 137d128fe2a5e8ac693cee3202e4c6f4afb37b6fa3e17991c2764b0158ed3252 not found: ID does not exist"
Sep 16 17:43:51.647873 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.647852    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"]
Sep 16 17:43:51.653419 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:51.653395    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c0e2d-predictor-686b5ccdb6-25snv"]
Sep 16 17:43:53.000992 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:43:53.000956    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5f347d3-d6a5-4942-ac50-3205c8459da3" path="/var/lib/kubelet/pods/f5f347d3-d6a5-4942-ac50-3205c8459da3/volumes"
Sep 16 17:44:36.871822 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:36.871775    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-czzhc_a1ff3897-38d1-48bd-be6e-a8a698fcdfee/console-operator/1.log"
Sep 16 17:44:36.874039 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:36.874017    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-czzhc_a1ff3897-38d1-48bd-be6e-a8a698fcdfee/console-operator/1.log"
Sep 16 17:44:36.875769 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:36.875752    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6p26_1d970e5c-29fc-44af-b138-0e60309f2ac0/ovn-acl-logging/0.log"
Sep 16 17:44:36.878046 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:36.878031    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6p26_1d970e5c-29fc-44af-b138-0e60309f2ac0/ovn-acl-logging/0.log"
Sep 16 17:44:51.642720 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.642682    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5"]
Sep 16 17:44:51.643180 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.643164    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f347d3-d6a5-4942-ac50-3205c8459da3" containerName="storage-initializer"
Sep 16 17:44:51.643227 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.643184    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f347d3-d6a5-4942-ac50-3205c8459da3" containerName="storage-initializer"
Sep 16 17:44:51.643227 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.643203    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f347d3-d6a5-4942-ac50-3205c8459da3" containerName="kserve-container"
Sep 16 17:44:51.643227 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.643212    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f347d3-d6a5-4942-ac50-3205c8459da3" containerName="kserve-container"
Sep 16 17:44:51.643227 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.643222    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" containerName="kube-rbac-proxy"
Sep 16 17:44:51.643347 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.643231    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" containerName="kube-rbac-proxy"
Sep 16 17:44:51.643347 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.643247    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" containerName="kserve-container"
Sep 16 17:44:51.643347 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.643256    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" containerName="kserve-container"
Sep 16 17:44:51.643347 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.643271    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" containerName="storage-initializer"
Sep 16 17:44:51.643347 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.643279    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" containerName="storage-initializer"
Sep 16 17:44:51.643347 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.643290    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" containerName="storage-initializer"
Sep 16 17:44:51.643347 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.643299    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" containerName="storage-initializer"
Sep 16 17:44:51.643347 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.643309    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f347d3-d6a5-4942-ac50-3205c8459da3" containerName="kube-rbac-proxy"
Sep 16 17:44:51.643347 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.643318    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f347d3-d6a5-4942-ac50-3205c8459da3" containerName="kube-rbac-proxy"
Sep 16 17:44:51.643347 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.643327    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" containerName="kserve-container"
Sep 16 17:44:51.643347 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.643336    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" containerName="kserve-container"
Sep 16 17:44:51.643654 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.643352    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" containerName="kube-rbac-proxy"
Sep 16 17:44:51.643654 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.643361    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" containerName="kube-rbac-proxy"
Sep 16 17:44:51.643654 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.643425    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" containerName="kserve-container"
Sep 16 17:44:51.643654 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.643436    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" containerName="kserve-container"
Sep 16 17:44:51.643654 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.643447    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5f347d3-d6a5-4942-ac50-3205c8459da3" containerName="kube-rbac-proxy"
Sep 16 17:44:51.643654 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.643457    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="abc69283-ea0b-495f-813f-115a9b8203b7" containerName="kube-rbac-proxy"
Sep 16 17:44:51.643654 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.643466    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fd22c7d-30d0-4dd7-b7e9-da14faace1e1" containerName="kube-rbac-proxy"
Sep 16 17:44:51.643654 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.643475    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5f347d3-d6a5-4942-ac50-3205c8459da3" containerName="kserve-container"
Sep 16 17:44:51.646564 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.646548    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5"
Sep 16 17:44:51.649168 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.649146    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 17:44:51.649168 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.649159    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 17:44:51.650272 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.650249    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-9501e1-dockercfg-tm4wg\""
Sep 16 17:44:51.650374 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.650284    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-9501e1-predictor-serving-cert\""
Sep 16 17:44:51.650548 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.650524    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-9501e1\""
Sep 16 17:44:51.650661 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.650564    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-9501e1-kube-rbac-proxy-sar-config\""
Sep 16 17:44:51.650661 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.650629    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:44:51.659704 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.659664    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5"]
Sep 16 17:44:51.731773 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.731731    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc6f8464-b7dc-4ba7-8470-d31acd135a81-cabundle-cert\") pod \"isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5\" (UID: \"fc6f8464-b7dc-4ba7-8470-d31acd135a81\") " pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5"
Sep 16 17:44:51.731773 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.731773    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc6f8464-b7dc-4ba7-8470-d31acd135a81-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5\" (UID: \"fc6f8464-b7dc-4ba7-8470-d31acd135a81\") " pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5"
Sep 16 17:44:51.732047 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.731836    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdqdz\" (UniqueName: \"kubernetes.io/projected/fc6f8464-b7dc-4ba7-8470-d31acd135a81-kube-api-access-mdqdz\") pod \"isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5\" (UID: \"fc6f8464-b7dc-4ba7-8470-d31acd135a81\") " pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5"
Sep 16 17:44:51.732047 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.731858    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc6f8464-b7dc-4ba7-8470-d31acd135a81-kserve-provision-location\") pod \"isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5\" (UID: \"fc6f8464-b7dc-4ba7-8470-d31acd135a81\") " pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5"
Sep 16 17:44:51.732047 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.731876    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc6f8464-b7dc-4ba7-8470-d31acd135a81-proxy-tls\") pod \"isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5\" (UID: \"fc6f8464-b7dc-4ba7-8470-d31acd135a81\") " pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5"
Sep 16 17:44:51.832797 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.832745    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdqdz\" (UniqueName: \"kubernetes.io/projected/fc6f8464-b7dc-4ba7-8470-d31acd135a81-kube-api-access-mdqdz\") pod \"isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5\" (UID: \"fc6f8464-b7dc-4ba7-8470-d31acd135a81\") " pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5"
Sep 16 17:44:51.832797 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.832819    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc6f8464-b7dc-4ba7-8470-d31acd135a81-kserve-provision-location\") pod \"isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5\" (UID: \"fc6f8464-b7dc-4ba7-8470-d31acd135a81\") " pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5"
Sep 16 17:44:51.833084 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.832843    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc6f8464-b7dc-4ba7-8470-d31acd135a81-proxy-tls\") pod \"isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5\" (UID: \"fc6f8464-b7dc-4ba7-8470-d31acd135a81\") " pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5"
Sep 16 17:44:51.833084 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.832897    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc6f8464-b7dc-4ba7-8470-d31acd135a81-cabundle-cert\") pod \"isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5\" (UID: \"fc6f8464-b7dc-4ba7-8470-d31acd135a81\") " pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5"
Sep 16 17:44:51.833084 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:44:51.832973    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-serving-cert: secret "isvc-secondary-9501e1-predictor-serving-cert" not found
Sep 16 17:44:51.833084 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.833012    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc6f8464-b7dc-4ba7-8470-d31acd135a81-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5\" (UID: \"fc6f8464-b7dc-4ba7-8470-d31acd135a81\") " pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5"
Sep 16 17:44:51.833084 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:44:51.833036    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc6f8464-b7dc-4ba7-8470-d31acd135a81-proxy-tls podName:fc6f8464-b7dc-4ba7-8470-d31acd135a81 nodeName:}" failed. No retries permitted until 2026-09-16 17:44:52.333017045 +0000 UTC m=+1215.921407006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fc6f8464-b7dc-4ba7-8470-d31acd135a81-proxy-tls") pod "isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5" (UID: "fc6f8464-b7dc-4ba7-8470-d31acd135a81") : secret "isvc-secondary-9501e1-predictor-serving-cert" not found
Sep 16 17:44:51.833350 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.833324    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc6f8464-b7dc-4ba7-8470-d31acd135a81-kserve-provision-location\") pod \"isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5\" (UID: \"fc6f8464-b7dc-4ba7-8470-d31acd135a81\") " pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5"
Sep 16 17:44:51.833601 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.833583    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc6f8464-b7dc-4ba7-8470-d31acd135a81-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5\" (UID: \"fc6f8464-b7dc-4ba7-8470-d31acd135a81\") " pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5"
Sep 16 17:44:51.833663 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.833603    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc6f8464-b7dc-4ba7-8470-d31acd135a81-cabundle-cert\") pod \"isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5\" (UID: \"fc6f8464-b7dc-4ba7-8470-d31acd135a81\") " pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5"
Sep 16 17:44:51.841908 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:51.841889    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdqdz\" (UniqueName: \"kubernetes.io/projected/fc6f8464-b7dc-4ba7-8470-d31acd135a81-kube-api-access-mdqdz\") pod \"isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5\" (UID: \"fc6f8464-b7dc-4ba7-8470-d31acd135a81\") " pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5"
Sep 16 17:44:52.337609 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:52.337568    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc6f8464-b7dc-4ba7-8470-d31acd135a81-proxy-tls\") pod \"isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5\" (UID: \"fc6f8464-b7dc-4ba7-8470-d31acd135a81\") " pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5"
Sep 16 17:44:52.338082 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:44:52.337742    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-serving-cert: secret "isvc-secondary-9501e1-predictor-serving-cert" not found
Sep 16 17:44:52.338082 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:44:52.337859    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc6f8464-b7dc-4ba7-8470-d31acd135a81-proxy-tls podName:fc6f8464-b7dc-4ba7-8470-d31acd135a81 nodeName:}" failed. No retries permitted until 2026-09-16 17:44:53.337837548 +0000 UTC m=+1216.926227528 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fc6f8464-b7dc-4ba7-8470-d31acd135a81-proxy-tls") pod "isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5" (UID: "fc6f8464-b7dc-4ba7-8470-d31acd135a81") : secret "isvc-secondary-9501e1-predictor-serving-cert" not found
Sep 16 17:44:53.344492 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:53.344461    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc6f8464-b7dc-4ba7-8470-d31acd135a81-proxy-tls\") pod \"isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5\" (UID: \"fc6f8464-b7dc-4ba7-8470-d31acd135a81\") " pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5"
Sep 16 17:44:53.347021 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:53.346988    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc6f8464-b7dc-4ba7-8470-d31acd135a81-proxy-tls\") pod \"isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5\" (UID: \"fc6f8464-b7dc-4ba7-8470-d31acd135a81\") " pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5"
Sep 16 17:44:53.459083 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:53.459051    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5"
Sep 16 17:44:53.728617 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:53.728568    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5"]
Sep 16 17:44:53.731455 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:44:53.731428    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc6f8464_b7dc_4ba7_8470_d31acd135a81.slice/crio-d3566e5faffa76a1dbf79aefc419f14d19bdbea0ebc0f461949c1fa7f7dda83d WatchSource:0}: Error finding container d3566e5faffa76a1dbf79aefc419f14d19bdbea0ebc0f461949c1fa7f7dda83d: Status 404 returned error can't find the container with id d3566e5faffa76a1dbf79aefc419f14d19bdbea0ebc0f461949c1fa7f7dda83d
Sep 16 17:44:53.869591 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:53.869553    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5" event={"ID":"fc6f8464-b7dc-4ba7-8470-d31acd135a81","Type":"ContainerStarted","Data":"e57fe5805f7c4c3d085c68d24c390b847060df010fee2810da94154603535dbe"}
Sep 16 17:44:53.869757 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:53.869613    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5" event={"ID":"fc6f8464-b7dc-4ba7-8470-d31acd135a81","Type":"ContainerStarted","Data":"d3566e5faffa76a1dbf79aefc419f14d19bdbea0ebc0f461949c1fa7f7dda83d"}
Sep 16 17:44:59.894902 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:59.894872    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5_fc6f8464-b7dc-4ba7-8470-d31acd135a81/storage-initializer/0.log"
Sep 16 17:44:59.895303 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:59.894911    2562 generic.go:356] "Generic (PLEG): container finished" podID="fc6f8464-b7dc-4ba7-8470-d31acd135a81" containerID="e57fe5805f7c4c3d085c68d24c390b847060df010fee2810da94154603535dbe" exitCode=1
Sep 16 17:44:59.895303 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:44:59.894978    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5" event={"ID":"fc6f8464-b7dc-4ba7-8470-d31acd135a81","Type":"ContainerDied","Data":"e57fe5805f7c4c3d085c68d24c390b847060df010fee2810da94154603535dbe"}
Sep 16 17:45:00.900611 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:00.900579    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5_fc6f8464-b7dc-4ba7-8470-d31acd135a81/storage-initializer/0.log"
Sep 16 17:45:00.901016 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:00.900648    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5" event={"ID":"fc6f8464-b7dc-4ba7-8470-d31acd135a81","Type":"ContainerStarted","Data":"1294f2eff15730691dacfc485bb48e9d7fbba5182dafb18a7121f84cfc6e0f18"}
Sep 16 17:45:04.916635 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:04.916609    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5_fc6f8464-b7dc-4ba7-8470-d31acd135a81/storage-initializer/1.log"
Sep 16 17:45:04.917092 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:04.917054    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5_fc6f8464-b7dc-4ba7-8470-d31acd135a81/storage-initializer/0.log"
Sep 16 17:45:04.917178 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:04.917093    2562 generic.go:356] "Generic (PLEG): container finished" podID="fc6f8464-b7dc-4ba7-8470-d31acd135a81" containerID="1294f2eff15730691dacfc485bb48e9d7fbba5182dafb18a7121f84cfc6e0f18" exitCode=1
Sep 16 17:45:04.917219 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:04.917168    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5" event={"ID":"fc6f8464-b7dc-4ba7-8470-d31acd135a81","Type":"ContainerDied","Data":"1294f2eff15730691dacfc485bb48e9d7fbba5182dafb18a7121f84cfc6e0f18"}
Sep 16 17:45:04.917279 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:04.917266    2562 scope.go:117] "RemoveContainer" containerID="e57fe5805f7c4c3d085c68d24c390b847060df010fee2810da94154603535dbe"
Sep 16 17:45:04.917605 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:04.917588    2562 scope.go:117] "RemoveContainer" containerID="e57fe5805f7c4c3d085c68d24c390b847060df010fee2810da94154603535dbe"
Sep 16 17:45:04.927326 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:45:04.927299    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5_kserve-ci-e2e-test(fc6f8464-b7dc-4ba7-8470-d31acd135a81)\"" pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5" podUID="fc6f8464-b7dc-4ba7-8470-d31acd135a81"
Sep 16 17:45:04.927496 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:45:04.927460    2562 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5_kserve-ci-e2e-test_fc6f8464-b7dc-4ba7-8470-d31acd135a81_0 in pod sandbox d3566e5faffa76a1dbf79aefc419f14d19bdbea0ebc0f461949c1fa7f7dda83d from index: no such id: 'e57fe5805f7c4c3d085c68d24c390b847060df010fee2810da94154603535dbe'" containerID="e57fe5805f7c4c3d085c68d24c390b847060df010fee2810da94154603535dbe"
Sep 16 17:45:04.927575 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:04.927496    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e57fe5805f7c4c3d085c68d24c390b847060df010fee2810da94154603535dbe"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5_kserve-ci-e2e-test_fc6f8464-b7dc-4ba7-8470-d31acd135a81_0 in pod sandbox d3566e5faffa76a1dbf79aefc419f14d19bdbea0ebc0f461949c1fa7f7dda83d from index: no such id: 'e57fe5805f7c4c3d085c68d24c390b847060df010fee2810da94154603535dbe'"
Sep 16 17:45:05.922354 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:05.922323    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5_fc6f8464-b7dc-4ba7-8470-d31acd135a81/storage-initializer/1.log"
Sep 16 17:45:11.697559 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:11.697526    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5"]
Sep 16 17:45:11.816448 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:11.816427    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5_fc6f8464-b7dc-4ba7-8470-d31acd135a81/storage-initializer/1.log"
Sep 16 17:45:11.819005 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:11.818988    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5"
Sep 16 17:45:11.901056 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:11.901024    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc6f8464-b7dc-4ba7-8470-d31acd135a81-kube-rbac-proxy-sar-config\") pod \"fc6f8464-b7dc-4ba7-8470-d31acd135a81\" (UID: \"fc6f8464-b7dc-4ba7-8470-d31acd135a81\") "
Sep 16 17:45:11.901223 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:11.901073    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc6f8464-b7dc-4ba7-8470-d31acd135a81-cabundle-cert\") pod \"fc6f8464-b7dc-4ba7-8470-d31acd135a81\" (UID: \"fc6f8464-b7dc-4ba7-8470-d31acd135a81\") "
Sep 16 17:45:11.901223 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:11.901099    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdqdz\" (UniqueName: \"kubernetes.io/projected/fc6f8464-b7dc-4ba7-8470-d31acd135a81-kube-api-access-mdqdz\") pod \"fc6f8464-b7dc-4ba7-8470-d31acd135a81\" (UID: \"fc6f8464-b7dc-4ba7-8470-d31acd135a81\") "
Sep 16 17:45:11.901223 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:11.901173    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc6f8464-b7dc-4ba7-8470-d31acd135a81-proxy-tls\") pod \"fc6f8464-b7dc-4ba7-8470-d31acd135a81\" (UID: \"fc6f8464-b7dc-4ba7-8470-d31acd135a81\") "
Sep 16 17:45:11.901348 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:11.901223    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc6f8464-b7dc-4ba7-8470-d31acd135a81-kserve-provision-location\") pod \"fc6f8464-b7dc-4ba7-8470-d31acd135a81\" (UID: \"fc6f8464-b7dc-4ba7-8470-d31acd135a81\") "
Sep 16 17:45:11.901497 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:11.901475    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc6f8464-b7dc-4ba7-8470-d31acd135a81-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fc6f8464-b7dc-4ba7-8470-d31acd135a81" (UID: "fc6f8464-b7dc-4ba7-8470-d31acd135a81"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:45:11.901571 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:11.901485    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc6f8464-b7dc-4ba7-8470-d31acd135a81-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fc6f8464-b7dc-4ba7-8470-d31acd135a81" (UID: "fc6f8464-b7dc-4ba7-8470-d31acd135a81"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:45:11.901627 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:11.901583    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc6f8464-b7dc-4ba7-8470-d31acd135a81-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc6f8464-b7dc-4ba7-8470-d31acd135a81" (UID: "fc6f8464-b7dc-4ba7-8470-d31acd135a81"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:45:11.903262 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:11.903242    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc6f8464-b7dc-4ba7-8470-d31acd135a81-kube-api-access-mdqdz" (OuterVolumeSpecName: "kube-api-access-mdqdz") pod "fc6f8464-b7dc-4ba7-8470-d31acd135a81" (UID: "fc6f8464-b7dc-4ba7-8470-d31acd135a81"). InnerVolumeSpecName "kube-api-access-mdqdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:45:11.903325 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:11.903267    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc6f8464-b7dc-4ba7-8470-d31acd135a81-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc6f8464-b7dc-4ba7-8470-d31acd135a81" (UID: "fc6f8464-b7dc-4ba7-8470-d31acd135a81"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:45:11.951867 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:11.951779    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5_fc6f8464-b7dc-4ba7-8470-d31acd135a81/storage-initializer/1.log"
Sep 16 17:45:11.951996 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:11.951900    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5" event={"ID":"fc6f8464-b7dc-4ba7-8470-d31acd135a81","Type":"ContainerDied","Data":"d3566e5faffa76a1dbf79aefc419f14d19bdbea0ebc0f461949c1fa7f7dda83d"}
Sep 16 17:45:11.951996 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:11.951955    2562 scope.go:117] "RemoveContainer" containerID="1294f2eff15730691dacfc485bb48e9d7fbba5182dafb18a7121f84cfc6e0f18"
Sep 16 17:45:11.951996 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:11.951907    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5"
Sep 16 17:45:11.992989 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:11.992962    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5"]
Sep 16 17:45:11.997710 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:11.997685    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9501e1-predictor-6779bdbd7b-d4fh5"]
Sep 16 17:45:12.001832 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:12.001791    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc6f8464-b7dc-4ba7-8470-d31acd135a81-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:45:12.001926 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:12.001838    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc6f8464-b7dc-4ba7-8470-d31acd135a81-cabundle-cert\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:45:12.001926 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:12.001851    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdqdz\" (UniqueName: \"kubernetes.io/projected/fc6f8464-b7dc-4ba7-8470-d31acd135a81-kube-api-access-mdqdz\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:45:12.001926 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:12.001860    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc6f8464-b7dc-4ba7-8470-d31acd135a81-proxy-tls\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:45:12.001926 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:12.001868    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc6f8464-b7dc-4ba7-8470-d31acd135a81-kserve-provision-location\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:45:13.000425 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:13.000390    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc6f8464-b7dc-4ba7-8470-d31acd135a81" path="/var/lib/kubelet/pods/fc6f8464-b7dc-4ba7-8470-d31acd135a81/volumes"
Sep 16 17:45:17.038870 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.038832    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj"]
Sep 16 17:45:17.041186 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.039251    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc6f8464-b7dc-4ba7-8470-d31acd135a81" containerName="storage-initializer"
Sep 16 17:45:17.041186 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.039268    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc6f8464-b7dc-4ba7-8470-d31acd135a81" containerName="storage-initializer"
Sep 16 17:45:17.041186 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.039298    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc6f8464-b7dc-4ba7-8470-d31acd135a81" containerName="storage-initializer"
Sep 16 17:45:17.041186 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.039307    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc6f8464-b7dc-4ba7-8470-d31acd135a81" containerName="storage-initializer"
Sep 16 17:45:17.041186 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.039377    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc6f8464-b7dc-4ba7-8470-d31acd135a81" containerName="storage-initializer"
Sep 16 17:45:17.041186 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.039390    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc6f8464-b7dc-4ba7-8470-d31acd135a81" containerName="storage-initializer"
Sep 16 17:45:17.042310 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.042295    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj"
Sep 16 17:45:17.047183 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.047160    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 17:45:17.047324 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.047303    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-6b13d5-predictor-serving-cert\""
Sep 16 17:45:17.052050 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.052030    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 17:45:17.052167 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.052030    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-6b13d5-kube-rbac-proxy-sar-config\""
Sep 16 17:45:17.052167 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.052066    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-6b13d5\""
Sep 16 17:45:17.052293 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.052201    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:45:17.052293 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.052224    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-6b13d5-dockercfg-65j5x\""
Sep 16 17:45:17.070427 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.070396    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj"]
Sep 16 17:45:17.136227 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.136195    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-proxy-tls\") pod \"isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj\" (UID: \"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf\") " pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj"
Sep 16 17:45:17.136434 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.136246    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxlkf\" (UniqueName: \"kubernetes.io/projected/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-kube-api-access-vxlkf\") pod \"isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj\" (UID: \"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf\") " pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj"
Sep 16 17:45:17.136434 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.136348    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj\" (UID: \"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf\") " pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj"
Sep 16 17:45:17.136434 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.136404    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-kserve-provision-location\") pod \"isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj\" (UID: \"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf\") " pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj"
Sep 16 17:45:17.136604 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.136438    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-cabundle-cert\") pod \"isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj\" (UID: \"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf\") " pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj"
Sep 16 17:45:17.236974 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.236927    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj\" (UID: \"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf\") " pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj"
Sep 16 17:45:17.237176 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.236992    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-kserve-provision-location\") pod \"isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj\" (UID: \"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf\") " pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj"
Sep 16 17:45:17.237176 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.237019    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-cabundle-cert\") pod \"isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj\" (UID: \"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf\") " pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj"
Sep 16 17:45:17.237176 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.237057    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-proxy-tls\") pod \"isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj\" (UID: \"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf\") " pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj"
Sep 16 17:45:17.237176 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.237093    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxlkf\" (UniqueName: \"kubernetes.io/projected/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-kube-api-access-vxlkf\") pod \"isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj\" (UID: \"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf\") " pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj"
Sep 16 17:45:17.237385 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:45:17.237201    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-serving-cert: secret "isvc-init-fail-6b13d5-predictor-serving-cert" not found
Sep 16 17:45:17.237385 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:45:17.237278    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-proxy-tls podName:7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf nodeName:}" failed. No retries permitted until 2026-09-16 17:45:17.737257385 +0000 UTC m=+1241.325647365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-proxy-tls") pod "isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj" (UID: "7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf") : secret "isvc-init-fail-6b13d5-predictor-serving-cert" not found
Sep 16 17:45:17.237513 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.237485    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-kserve-provision-location\") pod \"isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj\" (UID: \"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf\") " pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj"
Sep 16 17:45:17.237702 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.237686    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj\" (UID: \"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf\") " pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj"
Sep 16 17:45:17.237761 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.237702    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-cabundle-cert\") pod \"isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj\" (UID: \"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf\") " pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj"
Sep 16 17:45:17.251045 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.251022    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxlkf\" (UniqueName: \"kubernetes.io/projected/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-kube-api-access-vxlkf\") pod \"isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj\" (UID: \"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf\") " pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj"
Sep 16 17:45:17.741593 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:17.741553    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-proxy-tls\") pod \"isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj\" (UID: \"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf\") " pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj"
Sep 16 17:45:17.741799 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:45:17.741707    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-serving-cert: secret "isvc-init-fail-6b13d5-predictor-serving-cert" not found
Sep 16 17:45:17.741799 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:45:17.741780    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-proxy-tls podName:7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf nodeName:}" failed. No retries permitted until 2026-09-16 17:45:18.741765225 +0000 UTC m=+1242.330155186 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-proxy-tls") pod "isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj" (UID: "7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf") : secret "isvc-init-fail-6b13d5-predictor-serving-cert" not found
Sep 16 17:45:18.750133 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:18.750098    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-proxy-tls\") pod \"isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj\" (UID: \"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf\") " pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj"
Sep 16 17:45:18.752530 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:18.752510    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-proxy-tls\") pod \"isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj\" (UID: \"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf\") " pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj"
Sep 16 17:45:18.852849 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:18.852789    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj"
Sep 16 17:45:19.124764 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:19.124718    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj"]
Sep 16 17:45:19.127332 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:45:19.127307    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa35c80_a2e7_4b90_a0ab_11e7c4b476bf.slice/crio-35b7656ec0f4dc09be5e898027f973801182659fc388a9349a95b56bea4b2395 WatchSource:0}: Error finding container 35b7656ec0f4dc09be5e898027f973801182659fc388a9349a95b56bea4b2395: Status 404 returned error can't find the container with id 35b7656ec0f4dc09be5e898027f973801182659fc388a9349a95b56bea4b2395
Sep 16 17:45:19.985491 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:19.985458    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj" event={"ID":"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf","Type":"ContainerStarted","Data":"50504e7c2595bddd406d03be445ecd4fc99202e5f7142554804a066317198f48"}
Sep 16 17:45:19.985907 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:19.985519    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj" event={"ID":"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf","Type":"ContainerStarted","Data":"35b7656ec0f4dc09be5e898027f973801182659fc388a9349a95b56bea4b2395"}
Sep 16 17:45:22.998869 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:22.998843    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj_7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf/storage-initializer/0.log"
Sep 16 17:45:22.999249 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:22.998877    2562 generic.go:356] "Generic (PLEG): container finished" podID="7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf" containerID="50504e7c2595bddd406d03be445ecd4fc99202e5f7142554804a066317198f48" exitCode=1
Sep 16 17:45:23.000768 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:23.000744    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj" event={"ID":"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf","Type":"ContainerDied","Data":"50504e7c2595bddd406d03be445ecd4fc99202e5f7142554804a066317198f48"}
Sep 16 17:45:24.004599 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:24.004571    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj_7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf/storage-initializer/0.log"
Sep 16 17:45:24.005030 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:24.004658    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj" event={"ID":"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf","Type":"ContainerStarted","Data":"147208c5bb064f34c0c2c3811a7ce272c0a488a7b9acf6ce72ab97ca1310821e"}
Sep 16 17:45:26.973686 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:26.973641    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj"]
Sep 16 17:45:26.974131 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:26.973909    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj" podUID="7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf" containerName="storage-initializer" containerID="cri-o://147208c5bb064f34c0c2c3811a7ce272c0a488a7b9acf6ce72ab97ca1310821e" gracePeriod=30
Sep 16 17:45:28.021488 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:28.021463    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj_7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf/storage-initializer/1.log"
Sep 16 17:45:28.021915 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:28.021901    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj_7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf/storage-initializer/0.log"
Sep 16 17:45:28.021968 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:28.021937    2562 generic.go:356] "Generic (PLEG): container finished" podID="7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf" containerID="147208c5bb064f34c0c2c3811a7ce272c0a488a7b9acf6ce72ab97ca1310821e" exitCode=1
Sep 16 17:45:28.022027 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:28.022010    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj" event={"ID":"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf","Type":"ContainerDied","Data":"147208c5bb064f34c0c2c3811a7ce272c0a488a7b9acf6ce72ab97ca1310821e"}
Sep 16 17:45:28.022081 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:28.022071    2562 scope.go:117] "RemoveContainer" containerID="50504e7c2595bddd406d03be445ecd4fc99202e5f7142554804a066317198f48"
Sep 16 17:45:28.113046 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:28.113023    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj_7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf/storage-initializer/1.log"
Sep 16 17:45:28.116591 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:28.116575    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj"
Sep 16 17:45:28.230538 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:28.230502    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-kube-rbac-proxy-sar-config\") pod \"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf\" (UID: \"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf\") "
Sep 16 17:45:28.230706 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:28.230555    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-kserve-provision-location\") pod \"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf\" (UID: \"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf\") "
Sep 16 17:45:28.230706 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:28.230597    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-cabundle-cert\") pod \"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf\" (UID: \"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf\") "
Sep 16 17:45:28.230706 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:28.230637    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxlkf\" (UniqueName: \"kubernetes.io/projected/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-kube-api-access-vxlkf\") pod \"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf\" (UID: \"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf\") "
Sep 16 17:45:28.230706 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:28.230673    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-proxy-tls\") pod \"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf\" (UID: \"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf\") "
Sep 16 17:45:28.230956 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:28.230894    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf" (UID: "7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:45:28.230995 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:28.230953    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf" (UID: "7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:45:28.230995 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:28.230979    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf" (UID: "7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:45:28.232716 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:28.232694    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-kube-api-access-vxlkf" (OuterVolumeSpecName: "kube-api-access-vxlkf") pod "7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf" (UID: "7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf"). InnerVolumeSpecName "kube-api-access-vxlkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:45:28.232716 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:28.232703    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf" (UID: "7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:45:28.331482 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:28.331389    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:45:28.331482 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:28.331429    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-kserve-provision-location\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:45:28.331482 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:28.331439    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-cabundle-cert\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:45:28.331482 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:28.331448    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxlkf\" (UniqueName: \"kubernetes.io/projected/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-kube-api-access-vxlkf\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:45:28.331482 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:28.331459    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf-proxy-tls\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:45:29.027231 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:29.027204    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj_7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf/storage-initializer/1.log"
Sep 16 17:45:29.027571 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:29.027298    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj" event={"ID":"7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf","Type":"ContainerDied","Data":"35b7656ec0f4dc09be5e898027f973801182659fc388a9349a95b56bea4b2395"}
Sep 16 17:45:29.027571 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:29.027306    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj"
Sep 16 17:45:29.027571 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:29.027350    2562 scope.go:117] "RemoveContainer" containerID="147208c5bb064f34c0c2c3811a7ce272c0a488a7b9acf6ce72ab97ca1310821e"
Sep 16 17:45:29.064367 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:29.064333    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj"]
Sep 16 17:45:29.068595 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:29.068569    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6b13d5-predictor-69c94c595b-ltlhj"]
Sep 16 17:45:31.000573 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:45:31.000531    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf" path="/var/lib/kubelet/pods/7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf/volumes"
Sep 16 17:48:43.219708 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.219677    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"]
Sep 16 17:48:43.220155 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.220019    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf" containerName="storage-initializer"
Sep 16 17:48:43.220155 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.220032    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf" containerName="storage-initializer"
Sep 16 17:48:43.220155 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.220040    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf" containerName="storage-initializer"
Sep 16 17:48:43.220155 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.220045    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf" containerName="storage-initializer"
Sep 16 17:48:43.220155 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.220123    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf" containerName="storage-initializer"
Sep 16 17:48:43.220317 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.220237    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aa35c80-a2e7-4b90-a0ab-11e7c4b476bf" containerName="storage-initializer"
Sep 16 17:48:43.223086 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.223071    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"
Sep 16 17:48:43.236948 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.236921    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:48:43.237242 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.237227    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 17:48:43.238118 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.238099    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 16 17:48:43.238208 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.238141    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 17:48:43.238249 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.238141    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 17:48:43.238249 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.238139    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 16 17:48:43.238327 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.238148    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9bc9t\""
Sep 16 17:48:43.258700 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.258673    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"]
Sep 16 17:48:43.334165 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.334127    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa2ae494-3e26-4949-bba6-36b8669aa656-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-854596548c-2nwrv\" (UID: \"fa2ae494-3e26-4949-bba6-36b8669aa656\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"
Sep 16 17:48:43.334332 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.334228    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa2ae494-3e26-4949-bba6-36b8669aa656-proxy-tls\") pod \"isvc-canary-create-predictor-854596548c-2nwrv\" (UID: \"fa2ae494-3e26-4949-bba6-36b8669aa656\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"
Sep 16 17:48:43.334332 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.334282    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa2ae494-3e26-4949-bba6-36b8669aa656-kserve-provision-location\") pod \"isvc-canary-create-predictor-854596548c-2nwrv\" (UID: \"fa2ae494-3e26-4949-bba6-36b8669aa656\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"
Sep 16 17:48:43.334422 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.334352    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa2ae494-3e26-4949-bba6-36b8669aa656-cabundle-cert\") pod \"isvc-canary-create-predictor-854596548c-2nwrv\" (UID: \"fa2ae494-3e26-4949-bba6-36b8669aa656\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"
Sep 16 17:48:43.334422 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.334392    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk9rr\" (UniqueName: \"kubernetes.io/projected/fa2ae494-3e26-4949-bba6-36b8669aa656-kube-api-access-gk9rr\") pod \"isvc-canary-create-predictor-854596548c-2nwrv\" (UID: \"fa2ae494-3e26-4949-bba6-36b8669aa656\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"
Sep 16 17:48:43.435166 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.435136    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa2ae494-3e26-4949-bba6-36b8669aa656-proxy-tls\") pod \"isvc-canary-create-predictor-854596548c-2nwrv\" (UID: \"fa2ae494-3e26-4949-bba6-36b8669aa656\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"
Sep 16 17:48:43.435359 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.435183    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa2ae494-3e26-4949-bba6-36b8669aa656-kserve-provision-location\") pod \"isvc-canary-create-predictor-854596548c-2nwrv\" (UID: \"fa2ae494-3e26-4949-bba6-36b8669aa656\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"
Sep 16 17:48:43.435359 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.435251    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa2ae494-3e26-4949-bba6-36b8669aa656-cabundle-cert\") pod \"isvc-canary-create-predictor-854596548c-2nwrv\" (UID: \"fa2ae494-3e26-4949-bba6-36b8669aa656\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"
Sep 16 17:48:43.435359 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.435276    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk9rr\" (UniqueName: \"kubernetes.io/projected/fa2ae494-3e26-4949-bba6-36b8669aa656-kube-api-access-gk9rr\") pod \"isvc-canary-create-predictor-854596548c-2nwrv\" (UID: \"fa2ae494-3e26-4949-bba6-36b8669aa656\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"
Sep 16 17:48:43.435359 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.435309    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa2ae494-3e26-4949-bba6-36b8669aa656-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-854596548c-2nwrv\" (UID: \"fa2ae494-3e26-4949-bba6-36b8669aa656\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"
Sep 16 17:48:43.435706 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.435680    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa2ae494-3e26-4949-bba6-36b8669aa656-kserve-provision-location\") pod \"isvc-canary-create-predictor-854596548c-2nwrv\" (UID: \"fa2ae494-3e26-4949-bba6-36b8669aa656\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"
Sep 16 17:48:43.435987 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.435966    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa2ae494-3e26-4949-bba6-36b8669aa656-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-854596548c-2nwrv\" (UID: \"fa2ae494-3e26-4949-bba6-36b8669aa656\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"
Sep 16 17:48:43.436075 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.435997    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa2ae494-3e26-4949-bba6-36b8669aa656-cabundle-cert\") pod \"isvc-canary-create-predictor-854596548c-2nwrv\" (UID: \"fa2ae494-3e26-4949-bba6-36b8669aa656\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"
Sep 16 17:48:43.437560 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.437538    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa2ae494-3e26-4949-bba6-36b8669aa656-proxy-tls\") pod \"isvc-canary-create-predictor-854596548c-2nwrv\" (UID: \"fa2ae494-3e26-4949-bba6-36b8669aa656\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"
Sep 16 17:48:43.445317 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.445295    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk9rr\" (UniqueName: \"kubernetes.io/projected/fa2ae494-3e26-4949-bba6-36b8669aa656-kube-api-access-gk9rr\") pod \"isvc-canary-create-predictor-854596548c-2nwrv\" (UID: \"fa2ae494-3e26-4949-bba6-36b8669aa656\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"
Sep 16 17:48:43.532368 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.532273    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"
Sep 16 17:48:43.813946 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:43.813873    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"]
Sep 16 17:48:43.817008 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:48:43.816977    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa2ae494_3e26_4949_bba6_36b8669aa656.slice/crio-86b2579103b275ed85ddd99f19e7d62aceff95e76bfd8c1ccb2f97ed70d4dd5c WatchSource:0}: Error finding container 86b2579103b275ed85ddd99f19e7d62aceff95e76bfd8c1ccb2f97ed70d4dd5c: Status 404 returned error can't find the container with id 86b2579103b275ed85ddd99f19e7d62aceff95e76bfd8c1ccb2f97ed70d4dd5c
Sep 16 17:48:44.788448 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:44.788408    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv" event={"ID":"fa2ae494-3e26-4949-bba6-36b8669aa656","Type":"ContainerStarted","Data":"f8b1fa2fa59954a9f098d0941fe864492ebe1989e1b30f7eba3b228345149760"}
Sep 16 17:48:44.788856 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:44.788493    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv" event={"ID":"fa2ae494-3e26-4949-bba6-36b8669aa656","Type":"ContainerStarted","Data":"86b2579103b275ed85ddd99f19e7d62aceff95e76bfd8c1ccb2f97ed70d4dd5c"}
Sep 16 17:48:47.801743 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:47.801709    2562 generic.go:356] "Generic (PLEG): container finished" podID="fa2ae494-3e26-4949-bba6-36b8669aa656" containerID="f8b1fa2fa59954a9f098d0941fe864492ebe1989e1b30f7eba3b228345149760" exitCode=0
Sep 16 17:48:47.802123 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:47.801763    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv" event={"ID":"fa2ae494-3e26-4949-bba6-36b8669aa656","Type":"ContainerDied","Data":"f8b1fa2fa59954a9f098d0941fe864492ebe1989e1b30f7eba3b228345149760"}
Sep 16 17:48:48.807732 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:48.807694    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv" event={"ID":"fa2ae494-3e26-4949-bba6-36b8669aa656","Type":"ContainerStarted","Data":"a76bbe69a37acf99d5a730a379420cfd56257818015ca488ce9985d1256a1e1d"}
Sep 16 17:48:48.808168 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:48.807758    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv" event={"ID":"fa2ae494-3e26-4949-bba6-36b8669aa656","Type":"ContainerStarted","Data":"e58f052517f1e52ba6d5371968a5b8de8225e1867755bbdc725077ad76fe8c67"}
Sep 16 17:48:48.808168 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:48.807966    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"
Sep 16 17:48:48.873244 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:48.873190    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv" podStartSLOduration=2.10169046 podStartE2EDuration="5.873171539s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.771481079s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:48:48.870399452 +0000 UTC m=+1452.458789435" watchObservedRunningTime="2026-09-16 17:48:48.873171539 +0000 UTC m=+1452.461561524"
Sep 16 17:48:49.812614 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:49.812577    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"
Sep 16 17:48:49.813872 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:49.813845    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv" podUID="fa2ae494-3e26-4949-bba6-36b8669aa656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 17:48:50.816815 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:50.816772    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv" podUID="fa2ae494-3e26-4949-bba6-36b8669aa656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 17:48:55.821366 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:55.821339    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"
Sep 16 17:48:55.821925 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:48:55.821894    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv" podUID="fa2ae494-3e26-4949-bba6-36b8669aa656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 17:49:05.821927 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:49:05.821886    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv" podUID="fa2ae494-3e26-4949-bba6-36b8669aa656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 17:49:15.822049 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:49:15.822008    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv" podUID="fa2ae494-3e26-4949-bba6-36b8669aa656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 17:49:25.822061 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:49:25.822020    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv" podUID="fa2ae494-3e26-4949-bba6-36b8669aa656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 17:49:35.822668 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:49:35.822629    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv" podUID="fa2ae494-3e26-4949-bba6-36b8669aa656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 17:49:36.898113 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:49:36.898077    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-czzhc_a1ff3897-38d1-48bd-be6e-a8a698fcdfee/console-operator/1.log"
Sep 16 17:49:36.901018 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:49:36.900991    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-czzhc_a1ff3897-38d1-48bd-be6e-a8a698fcdfee/console-operator/1.log"
Sep 16 17:49:36.901923 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:49:36.901906    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6p26_1d970e5c-29fc-44af-b138-0e60309f2ac0/ovn-acl-logging/0.log"
Sep 16 17:49:36.904956 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:49:36.904936    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6p26_1d970e5c-29fc-44af-b138-0e60309f2ac0/ovn-acl-logging/0.log"
Sep 16 17:49:45.822051 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:49:45.822009    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv" podUID="fa2ae494-3e26-4949-bba6-36b8669aa656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 17:49:55.821785 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:49:55.821745    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv" podUID="fa2ae494-3e26-4949-bba6-36b8669aa656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 17:50:05.823011 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:05.822929    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"
Sep 16 17:50:13.162158 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:13.162126    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"]
Sep 16 17:50:13.162707 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:13.162364    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv" podUID="fa2ae494-3e26-4949-bba6-36b8669aa656" containerName="kserve-container" containerID="cri-o://e58f052517f1e52ba6d5371968a5b8de8225e1867755bbdc725077ad76fe8c67" gracePeriod=30
Sep 16 17:50:13.162707 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:13.162385    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv" podUID="fa2ae494-3e26-4949-bba6-36b8669aa656" containerName="kube-rbac-proxy" containerID="cri-o://a76bbe69a37acf99d5a730a379420cfd56257818015ca488ce9985d1256a1e1d" gracePeriod=30
Sep 16 17:50:14.141696 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:14.141662    2562 generic.go:356] "Generic (PLEG): container finished" podID="fa2ae494-3e26-4949-bba6-36b8669aa656" containerID="a76bbe69a37acf99d5a730a379420cfd56257818015ca488ce9985d1256a1e1d" exitCode=2
Sep 16 17:50:14.141883 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:14.141731    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv" event={"ID":"fa2ae494-3e26-4949-bba6-36b8669aa656","Type":"ContainerDied","Data":"a76bbe69a37acf99d5a730a379420cfd56257818015ca488ce9985d1256a1e1d"}
Sep 16 17:50:15.817956 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:15.817914    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv" podUID="fa2ae494-3e26-4949-bba6-36b8669aa656" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 16 17:50:15.822386 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:15.822355    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv" podUID="fa2ae494-3e26-4949-bba6-36b8669aa656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 17:50:17.698661 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:17.698638    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"
Sep 16 17:50:17.838238 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:17.838138    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa2ae494-3e26-4949-bba6-36b8669aa656-kserve-provision-location\") pod \"fa2ae494-3e26-4949-bba6-36b8669aa656\" (UID: \"fa2ae494-3e26-4949-bba6-36b8669aa656\") "
Sep 16 17:50:17.838238 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:17.838187    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa2ae494-3e26-4949-bba6-36b8669aa656-kube-rbac-proxy-sar-config\") pod \"fa2ae494-3e26-4949-bba6-36b8669aa656\" (UID: \"fa2ae494-3e26-4949-bba6-36b8669aa656\") "
Sep 16 17:50:17.838238 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:17.838216    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa2ae494-3e26-4949-bba6-36b8669aa656-proxy-tls\") pod \"fa2ae494-3e26-4949-bba6-36b8669aa656\" (UID: \"fa2ae494-3e26-4949-bba6-36b8669aa656\") "
Sep 16 17:50:17.838529 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:17.838250    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gk9rr\" (UniqueName: \"kubernetes.io/projected/fa2ae494-3e26-4949-bba6-36b8669aa656-kube-api-access-gk9rr\") pod \"fa2ae494-3e26-4949-bba6-36b8669aa656\" (UID: \"fa2ae494-3e26-4949-bba6-36b8669aa656\") "
Sep 16 17:50:17.838529 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:17.838289    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa2ae494-3e26-4949-bba6-36b8669aa656-cabundle-cert\") pod \"fa2ae494-3e26-4949-bba6-36b8669aa656\" (UID: \"fa2ae494-3e26-4949-bba6-36b8669aa656\") "
Sep 16 17:50:17.838655 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:17.838561    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2ae494-3e26-4949-bba6-36b8669aa656-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fa2ae494-3e26-4949-bba6-36b8669aa656" (UID: "fa2ae494-3e26-4949-bba6-36b8669aa656"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:50:17.838701 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:17.838648    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2ae494-3e26-4949-bba6-36b8669aa656-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fa2ae494-3e26-4949-bba6-36b8669aa656" (UID: "fa2ae494-3e26-4949-bba6-36b8669aa656"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:50:17.838793 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:17.838770    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2ae494-3e26-4949-bba6-36b8669aa656-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fa2ae494-3e26-4949-bba6-36b8669aa656" (UID: "fa2ae494-3e26-4949-bba6-36b8669aa656"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:50:17.840348 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:17.840318    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa2ae494-3e26-4949-bba6-36b8669aa656-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fa2ae494-3e26-4949-bba6-36b8669aa656" (UID: "fa2ae494-3e26-4949-bba6-36b8669aa656"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:50:17.840467 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:17.840398    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2ae494-3e26-4949-bba6-36b8669aa656-kube-api-access-gk9rr" (OuterVolumeSpecName: "kube-api-access-gk9rr") pod "fa2ae494-3e26-4949-bba6-36b8669aa656" (UID: "fa2ae494-3e26-4949-bba6-36b8669aa656"). InnerVolumeSpecName "kube-api-access-gk9rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:50:17.939633 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:17.939600    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa2ae494-3e26-4949-bba6-36b8669aa656-cabundle-cert\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:50:17.939633 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:17.939627    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa2ae494-3e26-4949-bba6-36b8669aa656-kserve-provision-location\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:50:17.939633 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:17.939637    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa2ae494-3e26-4949-bba6-36b8669aa656-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:50:17.939874 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:17.939647    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa2ae494-3e26-4949-bba6-36b8669aa656-proxy-tls\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:50:17.939874 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:17.939655    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk9rr\" (UniqueName: \"kubernetes.io/projected/fa2ae494-3e26-4949-bba6-36b8669aa656-kube-api-access-gk9rr\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:50:18.159730 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.159639    2562 generic.go:356] "Generic (PLEG): container finished" podID="fa2ae494-3e26-4949-bba6-36b8669aa656" containerID="e58f052517f1e52ba6d5371968a5b8de8225e1867755bbdc725077ad76fe8c67" exitCode=0
Sep 16 17:50:18.159730 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.159697    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"
Sep 16 17:50:18.159935 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.159730    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv" event={"ID":"fa2ae494-3e26-4949-bba6-36b8669aa656","Type":"ContainerDied","Data":"e58f052517f1e52ba6d5371968a5b8de8225e1867755bbdc725077ad76fe8c67"}
Sep 16 17:50:18.159935 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.159787    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv" event={"ID":"fa2ae494-3e26-4949-bba6-36b8669aa656","Type":"ContainerDied","Data":"86b2579103b275ed85ddd99f19e7d62aceff95e76bfd8c1ccb2f97ed70d4dd5c"}
Sep 16 17:50:18.159935 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.159829    2562 scope.go:117] "RemoveContainer" containerID="a76bbe69a37acf99d5a730a379420cfd56257818015ca488ce9985d1256a1e1d"
Sep 16 17:50:18.168367 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.168351    2562 scope.go:117] "RemoveContainer" containerID="e58f052517f1e52ba6d5371968a5b8de8225e1867755bbdc725077ad76fe8c67"
Sep 16 17:50:18.175591 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.175575    2562 scope.go:117] "RemoveContainer" containerID="f8b1fa2fa59954a9f098d0941fe864492ebe1989e1b30f7eba3b228345149760"
Sep 16 17:50:18.186175 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.186152    2562 scope.go:117] "RemoveContainer" containerID="a76bbe69a37acf99d5a730a379420cfd56257818015ca488ce9985d1256a1e1d"
Sep 16 17:50:18.186464 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:50:18.186445    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a76bbe69a37acf99d5a730a379420cfd56257818015ca488ce9985d1256a1e1d\": container with ID starting with a76bbe69a37acf99d5a730a379420cfd56257818015ca488ce9985d1256a1e1d not found: ID does not exist" containerID="a76bbe69a37acf99d5a730a379420cfd56257818015ca488ce9985d1256a1e1d"
Sep 16 17:50:18.186526 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.186473    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a76bbe69a37acf99d5a730a379420cfd56257818015ca488ce9985d1256a1e1d"} err="failed to get container status \"a76bbe69a37acf99d5a730a379420cfd56257818015ca488ce9985d1256a1e1d\": rpc error: code = NotFound desc = could not find container \"a76bbe69a37acf99d5a730a379420cfd56257818015ca488ce9985d1256a1e1d\": container with ID starting with a76bbe69a37acf99d5a730a379420cfd56257818015ca488ce9985d1256a1e1d not found: ID does not exist"
Sep 16 17:50:18.186526 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.186493    2562 scope.go:117] "RemoveContainer" containerID="e58f052517f1e52ba6d5371968a5b8de8225e1867755bbdc725077ad76fe8c67"
Sep 16 17:50:18.186759 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:50:18.186741    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e58f052517f1e52ba6d5371968a5b8de8225e1867755bbdc725077ad76fe8c67\": container with ID starting with e58f052517f1e52ba6d5371968a5b8de8225e1867755bbdc725077ad76fe8c67 not found: ID does not exist" containerID="e58f052517f1e52ba6d5371968a5b8de8225e1867755bbdc725077ad76fe8c67"
Sep 16 17:50:18.186818 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.186764    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e58f052517f1e52ba6d5371968a5b8de8225e1867755bbdc725077ad76fe8c67"} err="failed to get container status \"e58f052517f1e52ba6d5371968a5b8de8225e1867755bbdc725077ad76fe8c67\": rpc error: code = NotFound desc = could not find container \"e58f052517f1e52ba6d5371968a5b8de8225e1867755bbdc725077ad76fe8c67\": container with ID starting with e58f052517f1e52ba6d5371968a5b8de8225e1867755bbdc725077ad76fe8c67 not found: ID does not exist"
Sep 16 17:50:18.186818 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.186778    2562 scope.go:117] "RemoveContainer" containerID="f8b1fa2fa59954a9f098d0941fe864492ebe1989e1b30f7eba3b228345149760"
Sep 16 17:50:18.187022 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:50:18.187003    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b1fa2fa59954a9f098d0941fe864492ebe1989e1b30f7eba3b228345149760\": container with ID starting with f8b1fa2fa59954a9f098d0941fe864492ebe1989e1b30f7eba3b228345149760 not found: ID does not exist" containerID="f8b1fa2fa59954a9f098d0941fe864492ebe1989e1b30f7eba3b228345149760"
Sep 16 17:50:18.187067 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.187026    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b1fa2fa59954a9f098d0941fe864492ebe1989e1b30f7eba3b228345149760"} err="failed to get container status \"f8b1fa2fa59954a9f098d0941fe864492ebe1989e1b30f7eba3b228345149760\": rpc error: code = NotFound desc = could not find container \"f8b1fa2fa59954a9f098d0941fe864492ebe1989e1b30f7eba3b228345149760\": container with ID starting with f8b1fa2fa59954a9f098d0941fe864492ebe1989e1b30f7eba3b228345149760 not found: ID does not exist"
Sep 16 17:50:18.197025 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.197001    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"]
Sep 16 17:50:18.204137 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.204116    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-2nwrv"]
Sep 16 17:50:18.507414 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.507376    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"]
Sep 16 17:50:18.507707 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.507693    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa2ae494-3e26-4949-bba6-36b8669aa656" containerName="kserve-container"
Sep 16 17:50:18.507778 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.507708    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa2ae494-3e26-4949-bba6-36b8669aa656" containerName="kserve-container"
Sep 16 17:50:18.507778 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.507716    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa2ae494-3e26-4949-bba6-36b8669aa656" containerName="kube-rbac-proxy"
Sep 16 17:50:18.507778 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.507722    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa2ae494-3e26-4949-bba6-36b8669aa656" containerName="kube-rbac-proxy"
Sep 16 17:50:18.507778 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.507730    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa2ae494-3e26-4949-bba6-36b8669aa656" containerName="storage-initializer"
Sep 16 17:50:18.507778 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.507735    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa2ae494-3e26-4949-bba6-36b8669aa656" containerName="storage-initializer"
Sep 16 17:50:18.507778 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.507777    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa2ae494-3e26-4949-bba6-36b8669aa656" containerName="kserve-container"
Sep 16 17:50:18.508107 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.507786    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa2ae494-3e26-4949-bba6-36b8669aa656" containerName="kube-rbac-proxy"
Sep 16 17:50:18.512587 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.512564    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"
Sep 16 17:50:18.515550 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.515530    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9bc9t\""
Sep 16 17:50:18.515871 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.515854    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 17:50:18.515971 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.515945    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 17:50:18.516402 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.516383    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:50:18.516511 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.516420    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 16 17:50:18.516511 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.516438    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 16 17:50:18.516511 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.516493    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 17:50:18.539084 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.539057    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"]
Sep 16 17:50:18.646043 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.646007    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-cabundle-cert\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-hv2f9\" (UID: \"12d82fb3-ef0b-4cd7-a584-c62c712dbd19\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"
Sep 16 17:50:18.646043 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.646045    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-hv2f9\" (UID: \"12d82fb3-ef0b-4cd7-a584-c62c712dbd19\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"
Sep 16 17:50:18.646292 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.646149    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-kserve-provision-location\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-hv2f9\" (UID: \"12d82fb3-ef0b-4cd7-a584-c62c712dbd19\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"
Sep 16 17:50:18.646292 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.646184    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-hv2f9\" (UID: \"12d82fb3-ef0b-4cd7-a584-c62c712dbd19\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"
Sep 16 17:50:18.646292 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.646264    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmp8t\" (UniqueName: \"kubernetes.io/projected/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-kube-api-access-gmp8t\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-hv2f9\" (UID: \"12d82fb3-ef0b-4cd7-a584-c62c712dbd19\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"
Sep 16 17:50:18.746938 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.746905    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-kserve-provision-location\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-hv2f9\" (UID: \"12d82fb3-ef0b-4cd7-a584-c62c712dbd19\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"
Sep 16 17:50:18.746938 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.746939    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-hv2f9\" (UID: \"12d82fb3-ef0b-4cd7-a584-c62c712dbd19\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"
Sep 16 17:50:18.747390 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.746982    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmp8t\" (UniqueName: \"kubernetes.io/projected/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-kube-api-access-gmp8t\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-hv2f9\" (UID: \"12d82fb3-ef0b-4cd7-a584-c62c712dbd19\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"
Sep 16 17:50:18.747390 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.747039    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-cabundle-cert\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-hv2f9\" (UID: \"12d82fb3-ef0b-4cd7-a584-c62c712dbd19\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"
Sep 16 17:50:18.747390 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.747071    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-hv2f9\" (UID: \"12d82fb3-ef0b-4cd7-a584-c62c712dbd19\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"
Sep 16 17:50:18.747390 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:50:18.747211    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 16 17:50:18.747390 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:50:18.747284    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-proxy-tls podName:12d82fb3-ef0b-4cd7-a584-c62c712dbd19 nodeName:}" failed. No retries permitted until 2026-09-16 17:50:19.247249697 +0000 UTC m=+1542.835639661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-proxy-tls") pod "isvc-canary-promote-predictor-65ff4fc5f5-hv2f9" (UID: "12d82fb3-ef0b-4cd7-a584-c62c712dbd19") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 16 17:50:18.747390 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.747340    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-kserve-provision-location\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-hv2f9\" (UID: \"12d82fb3-ef0b-4cd7-a584-c62c712dbd19\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"
Sep 16 17:50:18.747760 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.747737    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-cabundle-cert\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-hv2f9\" (UID: \"12d82fb3-ef0b-4cd7-a584-c62c712dbd19\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"
Sep 16 17:50:18.747797 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.747737    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-hv2f9\" (UID: \"12d82fb3-ef0b-4cd7-a584-c62c712dbd19\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"
Sep 16 17:50:18.757611 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:18.757556    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmp8t\" (UniqueName: \"kubernetes.io/projected/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-kube-api-access-gmp8t\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-hv2f9\" (UID: \"12d82fb3-ef0b-4cd7-a584-c62c712dbd19\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"
Sep 16 17:50:19.000384 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:19.000350    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa2ae494-3e26-4949-bba6-36b8669aa656" path="/var/lib/kubelet/pods/fa2ae494-3e26-4949-bba6-36b8669aa656/volumes"
Sep 16 17:50:19.251956 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:19.251927    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-hv2f9\" (UID: \"12d82fb3-ef0b-4cd7-a584-c62c712dbd19\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"
Sep 16 17:50:19.254296 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:19.254270    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-hv2f9\" (UID: \"12d82fb3-ef0b-4cd7-a584-c62c712dbd19\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"
Sep 16 17:50:19.424152 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:19.424096    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"
Sep 16 17:50:19.709380 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:19.709358    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"]
Sep 16 17:50:19.711230 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:50:19.711198    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d82fb3_ef0b_4cd7_a584_c62c712dbd19.slice/crio-57350a0e228ba27bc377aeafbb29c2fb821e7d311d1f1bcb6b5ebdd6ab040999 WatchSource:0}: Error finding container 57350a0e228ba27bc377aeafbb29c2fb821e7d311d1f1bcb6b5ebdd6ab040999: Status 404 returned error can't find the container with id 57350a0e228ba27bc377aeafbb29c2fb821e7d311d1f1bcb6b5ebdd6ab040999
Sep 16 17:50:20.172029 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:20.171989    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9" event={"ID":"12d82fb3-ef0b-4cd7-a584-c62c712dbd19","Type":"ContainerStarted","Data":"8414987bfc51e10444e5581cb9bb26b2f3c61facb9ed25f0b582b2212abd074a"}
Sep 16 17:50:20.172400 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:20.172054    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9" event={"ID":"12d82fb3-ef0b-4cd7-a584-c62c712dbd19","Type":"ContainerStarted","Data":"57350a0e228ba27bc377aeafbb29c2fb821e7d311d1f1bcb6b5ebdd6ab040999"}
Sep 16 17:50:24.187869 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:24.187838    2562 generic.go:356] "Generic (PLEG): container finished" podID="12d82fb3-ef0b-4cd7-a584-c62c712dbd19" containerID="8414987bfc51e10444e5581cb9bb26b2f3c61facb9ed25f0b582b2212abd074a" exitCode=0
Sep 16 17:50:24.188228 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:24.187898    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9" event={"ID":"12d82fb3-ef0b-4cd7-a584-c62c712dbd19","Type":"ContainerDied","Data":"8414987bfc51e10444e5581cb9bb26b2f3c61facb9ed25f0b582b2212abd074a"}
Sep 16 17:50:25.194198 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:25.194164    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9" event={"ID":"12d82fb3-ef0b-4cd7-a584-c62c712dbd19","Type":"ContainerStarted","Data":"daef9406c69bd72ed2232cdc77008aaf8dec126bd72442125904088c53961d78"}
Sep 16 17:50:25.194561 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:25.194224    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9" event={"ID":"12d82fb3-ef0b-4cd7-a584-c62c712dbd19","Type":"ContainerStarted","Data":"1932d0c357c404e693ea160b631444951e849555b1d36dddfad18cc1c72024f4"}
Sep 16 17:50:25.194561 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:25.194424    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"
Sep 16 17:50:25.216328 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:25.216282    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9" podStartSLOduration=3.575636539 podStartE2EDuration="7.216265489s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.64062895s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:50:25.215769125 +0000 UTC m=+1548.804159109" watchObservedRunningTime="2026-09-16 17:50:25.216265489 +0000 UTC m=+1548.804655474"
Sep 16 17:50:26.198020 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:26.197988    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"
Sep 16 17:50:26.199121 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:26.199095    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9" podUID="12d82fb3-ef0b-4cd7-a584-c62c712dbd19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 17:50:27.201572 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:27.201532    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9" podUID="12d82fb3-ef0b-4cd7-a584-c62c712dbd19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 17:50:32.205889 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:32.205854    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"
Sep 16 17:50:32.206339 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:32.206313    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9" podUID="12d82fb3-ef0b-4cd7-a584-c62c712dbd19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 17:50:42.207012 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:42.206969    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9" podUID="12d82fb3-ef0b-4cd7-a584-c62c712dbd19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 17:50:52.206566 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:50:52.206525    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9" podUID="12d82fb3-ef0b-4cd7-a584-c62c712dbd19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 17:51:02.206778 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:02.206736    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9" podUID="12d82fb3-ef0b-4cd7-a584-c62c712dbd19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 17:51:12.206685 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:12.206640    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9" podUID="12d82fb3-ef0b-4cd7-a584-c62c712dbd19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 17:51:22.207052 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:22.207010    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9" podUID="12d82fb3-ef0b-4cd7-a584-c62c712dbd19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 17:51:32.206510 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:32.206474    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9" podUID="12d82fb3-ef0b-4cd7-a584-c62c712dbd19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 17:51:38.752245 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:38.752210    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"]
Sep 16 17:51:38.752783 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:38.752468    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9" podUID="12d82fb3-ef0b-4cd7-a584-c62c712dbd19" containerName="kserve-container" containerID="cri-o://1932d0c357c404e693ea160b631444951e849555b1d36dddfad18cc1c72024f4" gracePeriod=30
Sep 16 17:51:38.752783 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:38.752569    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9" podUID="12d82fb3-ef0b-4cd7-a584-c62c712dbd19" containerName="kube-rbac-proxy" containerID="cri-o://daef9406c69bd72ed2232cdc77008aaf8dec126bd72442125904088c53961d78" gracePeriod=30
Sep 16 17:51:38.753433 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:38.753416    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"
Sep 16 17:51:39.482260 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:39.482223    2562 generic.go:356] "Generic (PLEG): container finished" podID="12d82fb3-ef0b-4cd7-a584-c62c712dbd19" containerID="daef9406c69bd72ed2232cdc77008aaf8dec126bd72442125904088c53961d78" exitCode=2
Sep 16 17:51:39.482434 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:39.482301    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9" event={"ID":"12d82fb3-ef0b-4cd7-a584-c62c712dbd19","Type":"ContainerDied","Data":"daef9406c69bd72ed2232cdc77008aaf8dec126bd72442125904088c53961d78"}
Sep 16 17:51:42.202303 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:42.202254    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9" podUID="12d82fb3-ef0b-4cd7-a584-c62c712dbd19" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 16 17:51:43.305634 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.305608    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"
Sep 16 17:51:43.354506 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.354475    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-cabundle-cert\") pod \"12d82fb3-ef0b-4cd7-a584-c62c712dbd19\" (UID: \"12d82fb3-ef0b-4cd7-a584-c62c712dbd19\") "
Sep 16 17:51:43.354506 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.354508    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-kube-rbac-proxy-sar-config\") pod \"12d82fb3-ef0b-4cd7-a584-c62c712dbd19\" (UID: \"12d82fb3-ef0b-4cd7-a584-c62c712dbd19\") "
Sep 16 17:51:43.354718 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.354573    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-kserve-provision-location\") pod \"12d82fb3-ef0b-4cd7-a584-c62c712dbd19\" (UID: \"12d82fb3-ef0b-4cd7-a584-c62c712dbd19\") "
Sep 16 17:51:43.354718 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.354682    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmp8t\" (UniqueName: \"kubernetes.io/projected/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-kube-api-access-gmp8t\") pod \"12d82fb3-ef0b-4cd7-a584-c62c712dbd19\" (UID: \"12d82fb3-ef0b-4cd7-a584-c62c712dbd19\") "
Sep 16 17:51:43.354788 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.354745    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-proxy-tls\") pod \"12d82fb3-ef0b-4cd7-a584-c62c712dbd19\" (UID: \"12d82fb3-ef0b-4cd7-a584-c62c712dbd19\") "
Sep 16 17:51:43.354906 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.354884    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "12d82fb3-ef0b-4cd7-a584-c62c712dbd19" (UID: "12d82fb3-ef0b-4cd7-a584-c62c712dbd19"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:51:43.354972 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.354928    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "12d82fb3-ef0b-4cd7-a584-c62c712dbd19" (UID: "12d82fb3-ef0b-4cd7-a584-c62c712dbd19"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:51:43.355036 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.354934    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "12d82fb3-ef0b-4cd7-a584-c62c712dbd19" (UID: "12d82fb3-ef0b-4cd7-a584-c62c712dbd19"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:51:43.355099 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.355079    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-kserve-provision-location\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:51:43.355153 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.355106    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-cabundle-cert\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:51:43.355153 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.355124    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:51:43.356720 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.356697    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-kube-api-access-gmp8t" (OuterVolumeSpecName: "kube-api-access-gmp8t") pod "12d82fb3-ef0b-4cd7-a584-c62c712dbd19" (UID: "12d82fb3-ef0b-4cd7-a584-c62c712dbd19"). InnerVolumeSpecName "kube-api-access-gmp8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:51:43.356843 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.356726    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "12d82fb3-ef0b-4cd7-a584-c62c712dbd19" (UID: "12d82fb3-ef0b-4cd7-a584-c62c712dbd19"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:51:43.456489 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.456449    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmp8t\" (UniqueName: \"kubernetes.io/projected/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-kube-api-access-gmp8t\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:51:43.456489 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.456486    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12d82fb3-ef0b-4cd7-a584-c62c712dbd19-proxy-tls\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:51:43.499007 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.498980    2562 generic.go:356] "Generic (PLEG): container finished" podID="12d82fb3-ef0b-4cd7-a584-c62c712dbd19" containerID="1932d0c357c404e693ea160b631444951e849555b1d36dddfad18cc1c72024f4" exitCode=0
Sep 16 17:51:43.499147 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.499039    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9" event={"ID":"12d82fb3-ef0b-4cd7-a584-c62c712dbd19","Type":"ContainerDied","Data":"1932d0c357c404e693ea160b631444951e849555b1d36dddfad18cc1c72024f4"}
Sep 16 17:51:43.499147 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.499086    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9" event={"ID":"12d82fb3-ef0b-4cd7-a584-c62c712dbd19","Type":"ContainerDied","Data":"57350a0e228ba27bc377aeafbb29c2fb821e7d311d1f1bcb6b5ebdd6ab040999"}
Sep 16 17:51:43.499147 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.499042    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"
Sep 16 17:51:43.499147 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.499119    2562 scope.go:117] "RemoveContainer" containerID="daef9406c69bd72ed2232cdc77008aaf8dec126bd72442125904088c53961d78"
Sep 16 17:51:43.507864 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.507847    2562 scope.go:117] "RemoveContainer" containerID="1932d0c357c404e693ea160b631444951e849555b1d36dddfad18cc1c72024f4"
Sep 16 17:51:43.515181 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.515165    2562 scope.go:117] "RemoveContainer" containerID="8414987bfc51e10444e5581cb9bb26b2f3c61facb9ed25f0b582b2212abd074a"
Sep 16 17:51:43.522190 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.522174    2562 scope.go:117] "RemoveContainer" containerID="daef9406c69bd72ed2232cdc77008aaf8dec126bd72442125904088c53961d78"
Sep 16 17:51:43.522460 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:51:43.522441    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daef9406c69bd72ed2232cdc77008aaf8dec126bd72442125904088c53961d78\": container with ID starting with daef9406c69bd72ed2232cdc77008aaf8dec126bd72442125904088c53961d78 not found: ID does not exist" containerID="daef9406c69bd72ed2232cdc77008aaf8dec126bd72442125904088c53961d78"
Sep 16 17:51:43.522519 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.522469    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daef9406c69bd72ed2232cdc77008aaf8dec126bd72442125904088c53961d78"} err="failed to get container status \"daef9406c69bd72ed2232cdc77008aaf8dec126bd72442125904088c53961d78\": rpc error: code = NotFound desc = could not find container \"daef9406c69bd72ed2232cdc77008aaf8dec126bd72442125904088c53961d78\": container with ID starting with daef9406c69bd72ed2232cdc77008aaf8dec126bd72442125904088c53961d78 not found: ID does not exist"
Sep 16 17:51:43.522519 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.522485    2562 scope.go:117] "RemoveContainer" containerID="1932d0c357c404e693ea160b631444951e849555b1d36dddfad18cc1c72024f4"
Sep 16 17:51:43.522712 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:51:43.522699    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1932d0c357c404e693ea160b631444951e849555b1d36dddfad18cc1c72024f4\": container with ID starting with 1932d0c357c404e693ea160b631444951e849555b1d36dddfad18cc1c72024f4 not found: ID does not exist" containerID="1932d0c357c404e693ea160b631444951e849555b1d36dddfad18cc1c72024f4"
Sep 16 17:51:43.522763 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.522715    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1932d0c357c404e693ea160b631444951e849555b1d36dddfad18cc1c72024f4"} err="failed to get container status \"1932d0c357c404e693ea160b631444951e849555b1d36dddfad18cc1c72024f4\": rpc error: code = NotFound desc = could not find container \"1932d0c357c404e693ea160b631444951e849555b1d36dddfad18cc1c72024f4\": container with ID starting with 1932d0c357c404e693ea160b631444951e849555b1d36dddfad18cc1c72024f4 not found: ID does not exist"
Sep 16 17:51:43.522763 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.522727    2562 scope.go:117] "RemoveContainer" containerID="8414987bfc51e10444e5581cb9bb26b2f3c61facb9ed25f0b582b2212abd074a"
Sep 16 17:51:43.522962 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:51:43.522946    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8414987bfc51e10444e5581cb9bb26b2f3c61facb9ed25f0b582b2212abd074a\": container with ID starting with 8414987bfc51e10444e5581cb9bb26b2f3c61facb9ed25f0b582b2212abd074a not found: ID does not exist" containerID="8414987bfc51e10444e5581cb9bb26b2f3c61facb9ed25f0b582b2212abd074a"
Sep 16 17:51:43.523003 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.522965    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8414987bfc51e10444e5581cb9bb26b2f3c61facb9ed25f0b582b2212abd074a"} err="failed to get container status \"8414987bfc51e10444e5581cb9bb26b2f3c61facb9ed25f0b582b2212abd074a\": rpc error: code = NotFound desc = could not find container \"8414987bfc51e10444e5581cb9bb26b2f3c61facb9ed25f0b582b2212abd074a\": container with ID starting with 8414987bfc51e10444e5581cb9bb26b2f3c61facb9ed25f0b582b2212abd074a not found: ID does not exist"
Sep 16 17:51:43.523301 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.523283    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"]
Sep 16 17:51:43.527136 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:43.527116    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-hv2f9"]
Sep 16 17:51:45.000660 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:51:45.000624    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12d82fb3-ef0b-4cd7-a584-c62c712dbd19" path="/var/lib/kubelet/pods/12d82fb3-ef0b-4cd7-a584-c62c712dbd19/volumes"
Sep 16 17:54:36.928204 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:54:36.928172    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-czzhc_a1ff3897-38d1-48bd-be6e-a8a698fcdfee/console-operator/1.log"
Sep 16 17:54:36.932062 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:54:36.932041    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6p26_1d970e5c-29fc-44af-b138-0e60309f2ac0/ovn-acl-logging/0.log"
Sep 16 17:54:36.932695 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:54:36.932673    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-czzhc_a1ff3897-38d1-48bd-be6e-a8a698fcdfee/console-operator/1.log"
Sep 16 17:54:36.937153 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:54:36.937135    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t6p26_1d970e5c-29fc-44af-b138-0e60309f2ac0/ovn-acl-logging/0.log"
Sep 16 17:55:19.059852 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:19.059818    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-npmt6/must-gather-8mt29"]
Sep 16 17:55:19.062045 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:19.060113    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12d82fb3-ef0b-4cd7-a584-c62c712dbd19" containerName="kube-rbac-proxy"
Sep 16 17:55:19.062045 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:19.060124    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="12d82fb3-ef0b-4cd7-a584-c62c712dbd19" containerName="kube-rbac-proxy"
Sep 16 17:55:19.062045 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:19.060136    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12d82fb3-ef0b-4cd7-a584-c62c712dbd19" containerName="storage-initializer"
Sep 16 17:55:19.062045 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:19.060142    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="12d82fb3-ef0b-4cd7-a584-c62c712dbd19" containerName="storage-initializer"
Sep 16 17:55:19.062045 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:19.060151    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12d82fb3-ef0b-4cd7-a584-c62c712dbd19" containerName="kserve-container"
Sep 16 17:55:19.062045 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:19.060157    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="12d82fb3-ef0b-4cd7-a584-c62c712dbd19" containerName="kserve-container"
Sep 16 17:55:19.062045 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:19.060203    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="12d82fb3-ef0b-4cd7-a584-c62c712dbd19" containerName="kube-rbac-proxy"
Sep 16 17:55:19.062045 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:19.060215    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="12d82fb3-ef0b-4cd7-a584-c62c712dbd19" containerName="kserve-container"
Sep 16 17:55:19.063440 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:19.063422    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-npmt6/must-gather-8mt29"
Sep 16 17:55:19.068950 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:19.068927    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-npmt6\"/\"openshift-service-ca.crt\""
Sep 16 17:55:19.069072 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:19.069049    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-npmt6\"/\"default-dockercfg-22rrn\""
Sep 16 17:55:19.069120 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:19.069114    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-npmt6\"/\"kube-root-ca.crt\""
Sep 16 17:55:19.072648 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:19.072628    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-npmt6/must-gather-8mt29"]
Sep 16 17:55:19.153701 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:19.153664    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d7a62d74-3630-412e-bdda-601d57cfbbd5-must-gather-output\") pod \"must-gather-8mt29\" (UID: \"d7a62d74-3630-412e-bdda-601d57cfbbd5\") " pod="openshift-must-gather-npmt6/must-gather-8mt29"
Sep 16 17:55:19.153901 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:19.153717    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdds4\" (UniqueName: \"kubernetes.io/projected/d7a62d74-3630-412e-bdda-601d57cfbbd5-kube-api-access-zdds4\") pod \"must-gather-8mt29\" (UID: \"d7a62d74-3630-412e-bdda-601d57cfbbd5\") " pod="openshift-must-gather-npmt6/must-gather-8mt29"
Sep 16 17:55:19.254790 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:19.254750    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d7a62d74-3630-412e-bdda-601d57cfbbd5-must-gather-output\") pod \"must-gather-8mt29\" (UID: \"d7a62d74-3630-412e-bdda-601d57cfbbd5\") " pod="openshift-must-gather-npmt6/must-gather-8mt29"
Sep 16 17:55:19.254991 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:19.254862    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdds4\" (UniqueName: \"kubernetes.io/projected/d7a62d74-3630-412e-bdda-601d57cfbbd5-kube-api-access-zdds4\") pod \"must-gather-8mt29\" (UID: \"d7a62d74-3630-412e-bdda-601d57cfbbd5\") " pod="openshift-must-gather-npmt6/must-gather-8mt29"
Sep 16 17:55:19.255160 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:19.255137    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d7a62d74-3630-412e-bdda-601d57cfbbd5-must-gather-output\") pod \"must-gather-8mt29\" (UID: \"d7a62d74-3630-412e-bdda-601d57cfbbd5\") " pod="openshift-must-gather-npmt6/must-gather-8mt29"
Sep 16 17:55:19.275442 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:19.275412    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdds4\" (UniqueName: \"kubernetes.io/projected/d7a62d74-3630-412e-bdda-601d57cfbbd5-kube-api-access-zdds4\") pod \"must-gather-8mt29\" (UID: \"d7a62d74-3630-412e-bdda-601d57cfbbd5\") " pod="openshift-must-gather-npmt6/must-gather-8mt29"
Sep 16 17:55:19.385519 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:19.385433    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-npmt6/must-gather-8mt29"
Sep 16 17:55:19.657911 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:19.657881    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-npmt6/must-gather-8mt29"]
Sep 16 17:55:19.660137 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:55:19.660108    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7a62d74_3630_412e_bdda_601d57cfbbd5.slice/crio-85d8cc22e97a94fe399a1567cc019f761153f48f2568aa234344e95e51b8a3ac WatchSource:0}: Error finding container 85d8cc22e97a94fe399a1567cc019f761153f48f2568aa234344e95e51b8a3ac: Status 404 returned error can't find the container with id 85d8cc22e97a94fe399a1567cc019f761153f48f2568aa234344e95e51b8a3ac
Sep 16 17:55:19.661799 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:19.661782    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:55:20.328817 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:20.328718    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-npmt6/must-gather-8mt29" event={"ID":"d7a62d74-3630-412e-bdda-601d57cfbbd5","Type":"ContainerStarted","Data":"85d8cc22e97a94fe399a1567cc019f761153f48f2568aa234344e95e51b8a3ac"}
Sep 16 17:55:26.357770 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:26.357727    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-npmt6/must-gather-8mt29" event={"ID":"d7a62d74-3630-412e-bdda-601d57cfbbd5","Type":"ContainerStarted","Data":"5086083691e7525ca25318b8cdb5f1a759f8ea6099e30ae10f530b4689e7675a"}
Sep 16 17:55:26.358246 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:26.357780    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-npmt6/must-gather-8mt29" event={"ID":"d7a62d74-3630-412e-bdda-601d57cfbbd5","Type":"ContainerStarted","Data":"c327ab436bcc28d44ce4d4ca11e8336fa00580af6d5c787a7aa5b8f2891152de"}
Sep 16 17:55:26.378032 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:26.377980    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-npmt6/must-gather-8mt29" podStartSLOduration=1.357672593 podStartE2EDuration="7.377963201s" totalImagesPullingTime="6.020290608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:55:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:55:26.377585761 +0000 UTC m=+1849.965975747" watchObservedRunningTime="2026-09-16 17:55:26.377963201 +0000 UTC m=+1849.966353184"
Sep 16 17:55:49.459377 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:49.459344    2562 generic.go:356] "Generic (PLEG): container finished" podID="d7a62d74-3630-412e-bdda-601d57cfbbd5" containerID="c327ab436bcc28d44ce4d4ca11e8336fa00580af6d5c787a7aa5b8f2891152de" exitCode=0
Sep 16 17:55:49.459851 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:49.459387    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-npmt6/must-gather-8mt29" event={"ID":"d7a62d74-3630-412e-bdda-601d57cfbbd5","Type":"ContainerDied","Data":"c327ab436bcc28d44ce4d4ca11e8336fa00580af6d5c787a7aa5b8f2891152de"}
Sep 16 17:55:49.459851 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:49.459698    2562 scope.go:117] "RemoveContainer" containerID="c327ab436bcc28d44ce4d4ca11e8336fa00580af6d5c787a7aa5b8f2891152de"
Sep 16 17:55:49.872675 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:49.872599    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-npmt6_must-gather-8mt29_d7a62d74-3630-412e-bdda-601d57cfbbd5/gather/0.log"
Sep 16 17:55:53.827131 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:53.827099    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-scrxg_4b47007d-992e-4660-900c-b4807ce86b3f/global-pull-secret-syncer/0.log"
Sep 16 17:55:54.036022 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:54.035993    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-r67pk_15ea9542-9e0b-4000-8e55-0a4b734105ae/konnectivity-agent/0.log"
Sep 16 17:55:54.089165 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:54.089090    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-160.ec2.internal_0df2195e00f1241c11a998a6a4518995/haproxy/0.log"
Sep 16 17:55:55.435316 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:55.435283    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-npmt6/must-gather-8mt29"]
Sep 16 17:55:55.435730 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:55.435455    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-npmt6/must-gather-8mt29" podUID="d7a62d74-3630-412e-bdda-601d57cfbbd5" containerName="copy" containerID="cri-o://5086083691e7525ca25318b8cdb5f1a759f8ea6099e30ae10f530b4689e7675a" gracePeriod=2
Sep 16 17:55:55.439609 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:55.439066    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-npmt6/must-gather-8mt29"]
Sep 16 17:55:55.440781 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:55:55.440754    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-8mt29\" is forbidden: User \"system:node:ip-10-0-129-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-npmt6\": no relationship found between node 'ip-10-0-129-160.ec2.internal' and this object" podUID="d7a62d74-3630-412e-bdda-601d57cfbbd5" pod="openshift-must-gather-npmt6/must-gather-8mt29"
Sep 16 17:55:55.666253 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:55.666231    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-npmt6_must-gather-8mt29_d7a62d74-3630-412e-bdda-601d57cfbbd5/copy/0.log"
Sep 16 17:55:55.670118 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:55.670100    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-npmt6/must-gather-8mt29"
Sep 16 17:55:55.672732 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:55:55.672710    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-8mt29\" is forbidden: User \"system:node:ip-10-0-129-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-npmt6\": no relationship found between node 'ip-10-0-129-160.ec2.internal' and this object" podUID="d7a62d74-3630-412e-bdda-601d57cfbbd5" pod="openshift-must-gather-npmt6/must-gather-8mt29"
Sep 16 17:55:55.768542 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:55.768448    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdds4\" (UniqueName: \"kubernetes.io/projected/d7a62d74-3630-412e-bdda-601d57cfbbd5-kube-api-access-zdds4\") pod \"d7a62d74-3630-412e-bdda-601d57cfbbd5\" (UID: \"d7a62d74-3630-412e-bdda-601d57cfbbd5\") "
Sep 16 17:55:55.768542 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:55.768501    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d7a62d74-3630-412e-bdda-601d57cfbbd5-must-gather-output\") pod \"d7a62d74-3630-412e-bdda-601d57cfbbd5\" (UID: \"d7a62d74-3630-412e-bdda-601d57cfbbd5\") "
Sep 16 17:55:55.770612 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:55.770576    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7a62d74-3630-412e-bdda-601d57cfbbd5-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "d7a62d74-3630-412e-bdda-601d57cfbbd5" (UID: "d7a62d74-3630-412e-bdda-601d57cfbbd5"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:55:55.770801 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:55.770778    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a62d74-3630-412e-bdda-601d57cfbbd5-kube-api-access-zdds4" (OuterVolumeSpecName: "kube-api-access-zdds4") pod "d7a62d74-3630-412e-bdda-601d57cfbbd5" (UID: "d7a62d74-3630-412e-bdda-601d57cfbbd5"). InnerVolumeSpecName "kube-api-access-zdds4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:55:55.870091 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:55.870041    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdds4\" (UniqueName: \"kubernetes.io/projected/d7a62d74-3630-412e-bdda-601d57cfbbd5-kube-api-access-zdds4\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:55:55.870091 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:55.870083    2562 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d7a62d74-3630-412e-bdda-601d57cfbbd5-must-gather-output\") on node \"ip-10-0-129-160.ec2.internal\" DevicePath \"\""
Sep 16 17:55:56.487999 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:56.487972    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-npmt6_must-gather-8mt29_d7a62d74-3630-412e-bdda-601d57cfbbd5/copy/0.log"
Sep 16 17:55:56.488446 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:56.488341    2562 generic.go:356] "Generic (PLEG): container finished" podID="d7a62d74-3630-412e-bdda-601d57cfbbd5" containerID="5086083691e7525ca25318b8cdb5f1a759f8ea6099e30ae10f530b4689e7675a" exitCode=143
Sep 16 17:55:56.488446 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:56.488397    2562 scope.go:117] "RemoveContainer" containerID="5086083691e7525ca25318b8cdb5f1a759f8ea6099e30ae10f530b4689e7675a"
Sep 16 17:55:56.488572 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:56.488403    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-npmt6/must-gather-8mt29"
Sep 16 17:55:56.491159 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:55:56.491126    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-8mt29\" is forbidden: User \"system:node:ip-10-0-129-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-npmt6\": no relationship found between node 'ip-10-0-129-160.ec2.internal' and this object" podUID="d7a62d74-3630-412e-bdda-601d57cfbbd5" pod="openshift-must-gather-npmt6/must-gather-8mt29"
Sep 16 17:55:56.498394 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:56.498381    2562 scope.go:117] "RemoveContainer" containerID="c327ab436bcc28d44ce4d4ca11e8336fa00580af6d5c787a7aa5b8f2891152de"
Sep 16 17:55:56.501629 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:55:56.501603    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-8mt29\" is forbidden: User \"system:node:ip-10-0-129-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-npmt6\": no relationship found between node 'ip-10-0-129-160.ec2.internal' and this object" podUID="d7a62d74-3630-412e-bdda-601d57cfbbd5" pod="openshift-must-gather-npmt6/must-gather-8mt29"
Sep 16 17:55:56.511263 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:56.511241    2562 scope.go:117] "RemoveContainer" containerID="5086083691e7525ca25318b8cdb5f1a759f8ea6099e30ae10f530b4689e7675a"
Sep 16 17:55:56.511624 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:55:56.511606    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5086083691e7525ca25318b8cdb5f1a759f8ea6099e30ae10f530b4689e7675a\": container with ID starting with 5086083691e7525ca25318b8cdb5f1a759f8ea6099e30ae10f530b4689e7675a not found: ID does not exist" containerID="5086083691e7525ca25318b8cdb5f1a759f8ea6099e30ae10f530b4689e7675a"
Sep 16 17:55:56.511692 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:56.511637    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5086083691e7525ca25318b8cdb5f1a759f8ea6099e30ae10f530b4689e7675a"} err="failed to get container status \"5086083691e7525ca25318b8cdb5f1a759f8ea6099e30ae10f530b4689e7675a\": rpc error: code = NotFound desc = could not find container \"5086083691e7525ca25318b8cdb5f1a759f8ea6099e30ae10f530b4689e7675a\": container with ID starting with 5086083691e7525ca25318b8cdb5f1a759f8ea6099e30ae10f530b4689e7675a not found: ID does not exist"
Sep 16 17:55:56.511692 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:56.511662    2562 scope.go:117] "RemoveContainer" containerID="c327ab436bcc28d44ce4d4ca11e8336fa00580af6d5c787a7aa5b8f2891152de"
Sep 16 17:55:56.512341 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:55:56.512323    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c327ab436bcc28d44ce4d4ca11e8336fa00580af6d5c787a7aa5b8f2891152de\": container with ID starting with c327ab436bcc28d44ce4d4ca11e8336fa00580af6d5c787a7aa5b8f2891152de not found: ID does not exist" containerID="c327ab436bcc28d44ce4d4ca11e8336fa00580af6d5c787a7aa5b8f2891152de"
Sep 16 17:55:56.512401 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:56.512347    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c327ab436bcc28d44ce4d4ca11e8336fa00580af6d5c787a7aa5b8f2891152de"} err="failed to get container status \"c327ab436bcc28d44ce4d4ca11e8336fa00580af6d5c787a7aa5b8f2891152de\": rpc error: code = NotFound desc = could not find container \"c327ab436bcc28d44ce4d4ca11e8336fa00580af6d5c787a7aa5b8f2891152de\": container with ID starting with c327ab436bcc28d44ce4d4ca11e8336fa00580af6d5c787a7aa5b8f2891152de not found: ID does not exist"
Sep 16 17:55:57.003131 ip-10-0-129-160 kubenswrapper[2562]: E0916 17:55:57.003088    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-8mt29\" is forbidden: User \"system:node:ip-10-0-129-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-npmt6\": no relationship found between node 'ip-10-0-129-160.ec2.internal' and this object" podUID="d7a62d74-3630-412e-bdda-601d57cfbbd5" pod="openshift-must-gather-npmt6/must-gather-8mt29"
Sep 16 17:55:57.004887 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:57.004861    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7a62d74-3630-412e-bdda-601d57cfbbd5" path="/var/lib/kubelet/pods/d7a62d74-3630-412e-bdda-601d57cfbbd5/volumes"
Sep 16 17:55:58.170597 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:58.170569    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-dcsg4_bf8577d8-003e-4084-afe8-f76e6aa93a8d/kube-state-metrics/0.log"
Sep 16 17:55:58.197824 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:58.197775    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-dcsg4_bf8577d8-003e-4084-afe8-f76e6aa93a8d/kube-rbac-proxy-main/0.log"
Sep 16 17:55:58.231826 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:58.231785    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-dcsg4_bf8577d8-003e-4084-afe8-f76e6aa93a8d/kube-rbac-proxy-self/0.log"
Sep 16 17:55:58.704471 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:58.704442    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pkc6m_1853113c-2c05-4c44-b4a1-cb84d1f7f954/node-exporter/0.log"
Sep 16 17:55:58.758124 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:58.758099    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pkc6m_1853113c-2c05-4c44-b4a1-cb84d1f7f954/kube-rbac-proxy/0.log"
Sep 16 17:55:58.798316 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:58.798287    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pkc6m_1853113c-2c05-4c44-b4a1-cb84d1f7f954/init-textfile/0.log"
Sep 16 17:55:58.858393 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:58.858361    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-26vr9_8ac71616-9ee5-41ef-8e28-0d4beda37f01/kube-rbac-proxy-main/0.log"
Sep 16 17:55:58.897081 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:58.897052    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-26vr9_8ac71616-9ee5-41ef-8e28-0d4beda37f01/kube-rbac-proxy-self/0.log"
Sep 16 17:55:58.937205 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:58.937179    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-26vr9_8ac71616-9ee5-41ef-8e28-0d4beda37f01/openshift-state-metrics/0.log"
Sep 16 17:55:58.990057 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:58.989983    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17bbe9b1-d647-4c0a-a30e-23fabb8d0075/prometheus/0.log"
Sep 16 17:55:59.062979 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:59.062953    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17bbe9b1-d647-4c0a-a30e-23fabb8d0075/config-reloader/0.log"
Sep 16 17:55:59.104838 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:59.104737    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17bbe9b1-d647-4c0a-a30e-23fabb8d0075/thanos-sidecar/0.log"
Sep 16 17:55:59.149909 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:59.149885    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17bbe9b1-d647-4c0a-a30e-23fabb8d0075/kube-rbac-proxy-web/0.log"
Sep 16 17:55:59.186464 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:59.186436    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17bbe9b1-d647-4c0a-a30e-23fabb8d0075/kube-rbac-proxy/0.log"
Sep 16 17:55:59.221540 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:59.221518    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17bbe9b1-d647-4c0a-a30e-23fabb8d0075/kube-rbac-proxy-thanos/0.log"
Sep 16 17:55:59.270637 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:59.270555    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17bbe9b1-d647-4c0a-a30e-23fabb8d0075/init-config-reloader/0.log"
Sep 16 17:55:59.319705 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:59.319682    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-clv5f_3c0e1c10-67d7-4804-a977-829534bf6416/prometheus-operator/0.log"
Sep 16 17:55:59.387272 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:55:59.387246    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-clv5f_3c0e1c10-67d7-4804-a977-829534bf6416/kube-rbac-proxy/0.log"
Sep 16 17:56:01.282480 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.282450    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-ckczd_d2b67354-93c8-419c-87f0-6d817d914033/networking-console-plugin/0.log"
Sep 16 17:56:01.611740 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.611659    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-znfk5/perf-node-gather-daemonset-p28nn"]
Sep 16 17:56:01.612003 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.611989    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7a62d74-3630-412e-bdda-601d57cfbbd5" containerName="copy"
Sep 16 17:56:01.612051 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.612006    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7a62d74-3630-412e-bdda-601d57cfbbd5" containerName="copy"
Sep 16 17:56:01.612051 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.612027    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7a62d74-3630-412e-bdda-601d57cfbbd5" containerName="gather"
Sep 16 17:56:01.612051 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.612033    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7a62d74-3630-412e-bdda-601d57cfbbd5" containerName="gather"
Sep 16 17:56:01.612155 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.612088    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7a62d74-3630-412e-bdda-601d57cfbbd5" containerName="gather"
Sep 16 17:56:01.612155 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.612098    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7a62d74-3630-412e-bdda-601d57cfbbd5" containerName="copy"
Sep 16 17:56:01.616627 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.616607    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-p28nn"
Sep 16 17:56:01.619518 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.619497    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-znfk5\"/\"kube-root-ca.crt\""
Sep 16 17:56:01.620678 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.620661    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-znfk5\"/\"openshift-service-ca.crt\""
Sep 16 17:56:01.620734 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.620666    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-znfk5\"/\"default-dockercfg-bh4xb\""
Sep 16 17:56:01.630540 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.630517    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-znfk5/perf-node-gather-daemonset-p28nn"]
Sep 16 17:56:01.717315 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.717278    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln84r\" (UniqueName: \"kubernetes.io/projected/74243d89-efea-495a-9e92-29743f4b5a8e-kube-api-access-ln84r\") pod \"perf-node-gather-daemonset-p28nn\" (UID: \"74243d89-efea-495a-9e92-29743f4b5a8e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-p28nn"
Sep 16 17:56:01.717499 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.717326    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74243d89-efea-495a-9e92-29743f4b5a8e-lib-modules\") pod \"perf-node-gather-daemonset-p28nn\" (UID: \"74243d89-efea-495a-9e92-29743f4b5a8e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-p28nn"
Sep 16 17:56:01.717499 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.717344    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/74243d89-efea-495a-9e92-29743f4b5a8e-proc\") pod \"perf-node-gather-daemonset-p28nn\" (UID: \"74243d89-efea-495a-9e92-29743f4b5a8e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-p28nn"
Sep 16 17:56:01.717499 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.717414    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/74243d89-efea-495a-9e92-29743f4b5a8e-podres\") pod \"perf-node-gather-daemonset-p28nn\" (UID: \"74243d89-efea-495a-9e92-29743f4b5a8e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-p28nn"
Sep 16 17:56:01.717499 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.717437    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74243d89-efea-495a-9e92-29743f4b5a8e-sys\") pod \"perf-node-gather-daemonset-p28nn\" (UID: \"74243d89-efea-495a-9e92-29743f4b5a8e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-p28nn"
Sep 16 17:56:01.818455 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.818421    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/74243d89-efea-495a-9e92-29743f4b5a8e-podres\") pod \"perf-node-gather-daemonset-p28nn\" (UID: \"74243d89-efea-495a-9e92-29743f4b5a8e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-p28nn"
Sep 16 17:56:01.818619 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.818468    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74243d89-efea-495a-9e92-29743f4b5a8e-sys\") pod \"perf-node-gather-daemonset-p28nn\" (UID: \"74243d89-efea-495a-9e92-29743f4b5a8e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-p28nn"
Sep 16 17:56:01.818619 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.818504    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln84r\" (UniqueName: \"kubernetes.io/projected/74243d89-efea-495a-9e92-29743f4b5a8e-kube-api-access-ln84r\") pod \"perf-node-gather-daemonset-p28nn\" (UID: \"74243d89-efea-495a-9e92-29743f4b5a8e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-p28nn"
Sep 16 17:56:01.818619 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.818529    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74243d89-efea-495a-9e92-29743f4b5a8e-lib-modules\") pod \"perf-node-gather-daemonset-p28nn\" (UID: \"74243d89-efea-495a-9e92-29743f4b5a8e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-p28nn"
Sep 16 17:56:01.818619 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.818545    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/74243d89-efea-495a-9e92-29743f4b5a8e-proc\") pod \"perf-node-gather-daemonset-p28nn\" (UID: \"74243d89-efea-495a-9e92-29743f4b5a8e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-p28nn"
Sep 16 17:56:01.818619 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.818609    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/74243d89-efea-495a-9e92-29743f4b5a8e-podres\") pod \"perf-node-gather-daemonset-p28nn\" (UID: \"74243d89-efea-495a-9e92-29743f4b5a8e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-p28nn"
Sep 16 17:56:01.818789 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.818618    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/74243d89-efea-495a-9e92-29743f4b5a8e-proc\") pod \"perf-node-gather-daemonset-p28nn\" (UID: \"74243d89-efea-495a-9e92-29743f4b5a8e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-p28nn"
Sep 16 17:56:01.818789 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.818618    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74243d89-efea-495a-9e92-29743f4b5a8e-sys\") pod \"perf-node-gather-daemonset-p28nn\" (UID: \"74243d89-efea-495a-9e92-29743f4b5a8e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-p28nn"
Sep 16 17:56:01.818789 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.818685    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74243d89-efea-495a-9e92-29743f4b5a8e-lib-modules\") pod \"perf-node-gather-daemonset-p28nn\" (UID: \"74243d89-efea-495a-9e92-29743f4b5a8e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-p28nn"
Sep 16 17:56:01.827579 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.827549    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln84r\" (UniqueName: \"kubernetes.io/projected/74243d89-efea-495a-9e92-29743f4b5a8e-kube-api-access-ln84r\") pod \"perf-node-gather-daemonset-p28nn\" (UID: \"74243d89-efea-495a-9e92-29743f4b5a8e\") " pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-p28nn"
Sep 16 17:56:01.838135 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.838109    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-czzhc_a1ff3897-38d1-48bd-be6e-a8a698fcdfee/console-operator/1.log"
Sep 16 17:56:01.842507 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.842485    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-czzhc_a1ff3897-38d1-48bd-be6e-a8a698fcdfee/console-operator/2.log"
Sep 16 17:56:01.926204 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:01.926173    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-p28nn"
Sep 16 17:56:02.201776 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:02.201705    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-znfk5/perf-node-gather-daemonset-p28nn"]
Sep 16 17:56:02.204911 ip-10-0-129-160 kubenswrapper[2562]: W0916 17:56:02.204883    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod74243d89_efea_495a_9e92_29743f4b5a8e.slice/crio-8cba0a4d113ac065a125f4648a2a21989fa20cb1e93753fa6f62c866769e2c8f WatchSource:0}: Error finding container 8cba0a4d113ac065a125f4648a2a21989fa20cb1e93753fa6f62c866769e2c8f: Status 404 returned error can't find the container with id 8cba0a4d113ac065a125f4648a2a21989fa20cb1e93753fa6f62c866769e2c8f
Sep 16 17:56:02.512409 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:02.512326    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-p28nn" event={"ID":"74243d89-efea-495a-9e92-29743f4b5a8e","Type":"ContainerStarted","Data":"91bb4671e1b078e8db69389b98e49730ba197b3949bcf6712021e5ef25dcf130"}
Sep 16 17:56:02.512409 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:02.512378    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-p28nn" event={"ID":"74243d89-efea-495a-9e92-29743f4b5a8e","Type":"ContainerStarted","Data":"8cba0a4d113ac065a125f4648a2a21989fa20cb1e93753fa6f62c866769e2c8f"}
Sep 16 17:56:02.512789 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:02.512458    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-p28nn"
Sep 16 17:56:02.530643 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:02.530600    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-p28nn" podStartSLOduration=1.5305839209999998 podStartE2EDuration="1.530583921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:56:02.529837464 +0000 UTC m=+1886.118227448" watchObservedRunningTime="2026-09-16 17:56:02.530583921 +0000 UTC m=+1886.118973916"
Sep 16 17:56:02.866192 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:02.866114    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-fpb8s_5a6c8d6d-d244-42c9-bc8a-25f5c01ae41f/volume-data-source-validator/0.log"
Sep 16 17:56:03.852080 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:03.852049    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4lcwz_afb12787-82e5-40ce-b4f5-b34655ba032f/dns/0.log"
Sep 16 17:56:03.886544 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:03.886512    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4lcwz_afb12787-82e5-40ce-b4f5-b34655ba032f/kube-rbac-proxy/0.log"
Sep 16 17:56:04.138003 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:04.137933    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2r4vj_79c16240-59ee-4b5e-98c9-86ef04179efa/dns-node-resolver/0.log"
Sep 16 17:56:04.820260 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:04.820235    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-868c457c6b-pgc4m_dc3dea1b-13cf-444b-bb84-42d022358828/registry/0.log"
Sep 16 17:56:04.933480 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:04.933453    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ddqwd_f689aa90-24ff-48a5-8e66-f2e5d749dbd4/node-ca/0.log"
Sep 16 17:56:06.502135 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:06.502091    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2n9xt_64f7e8df-d87e-426f-acca-39781d05567c/serve-healthcheck-canary/0.log"
Sep 16 17:56:06.990039 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:06.990009    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-gghkt_6ad7439f-6922-4550-84fa-5ea0e6ff8081/insights-operator/0.log"
Sep 16 17:56:06.990454 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:06.990426    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-gghkt_6ad7439f-6922-4550-84fa-5ea0e6ff8081/insights-operator/1.log"
Sep 16 17:56:07.105303 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:07.105271    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-stbrb_2ff86f78-bedb-4545-8cdf-9eea531efcf9/kube-rbac-proxy/0.log"
Sep 16 17:56:07.129009 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:07.128980    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-stbrb_2ff86f78-bedb-4545-8cdf-9eea531efcf9/exporter/0.log"
Sep 16 17:56:07.153140 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:07.153110    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-stbrb_2ff86f78-bedb-4545-8cdf-9eea531efcf9/extractor/0.log"
Sep 16 17:56:08.526075 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:08.526048    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-znfk5/perf-node-gather-daemonset-p28nn"
Sep 16 17:56:09.886887 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:09.886851    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-9rm2h_2aa7f4d5-ad0a-40cd-8395-7752814f1525/s3-init/0.log"
Sep 16 17:56:09.910695 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:09.910672    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-9rm2h_2aa7f4d5-ad0a-40cd-8395-7752814f1525/download-hf-model/0.log"
Sep 16 17:56:17.134717 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:17.134687    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tkzcs_af87e079-c946-4c98-86fc-4be5a3098d8a/kube-multus-additional-cni-plugins/0.log"
Sep 16 17:56:17.177175 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:17.177145    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tkzcs_af87e079-c946-4c98-86fc-4be5a3098d8a/egress-router-binary-copy/0.log"
Sep 16 17:56:17.220407 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:17.220382    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tkzcs_af87e079-c946-4c98-86fc-4be5a3098d8a/cni-plugins/0.log"
Sep 16 17:56:17.287493 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:17.287466    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tkzcs_af87e079-c946-4c98-86fc-4be5a3098d8a/bond-cni-plugin/0.log"
Sep 16 17:56:17.335398 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:17.335368    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tkzcs_af87e079-c946-4c98-86fc-4be5a3098d8a/routeoverride-cni/0.log"
Sep 16 17:56:17.407675 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:17.407591    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tkzcs_af87e079-c946-4c98-86fc-4be5a3098d8a/whereabouts-cni-bincopy/0.log"
Sep 16 17:56:17.446408 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:17.446378    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tkzcs_af87e079-c946-4c98-86fc-4be5a3098d8a/whereabouts-cni/0.log"
Sep 16 17:56:17.498876 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:17.498844    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-lpd4g_a83c017c-9dbf-4051-a74c-e5fa85eda19f/kube-multus/0.log"
Sep 16 17:56:17.534569 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:17.534543    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fw78m_9d27ec00-06e7-431b-925f-addd45ff0115/network-metrics-daemon/0.log"
Sep 16 17:56:17.571618 ip-10-0-129-160 kubenswrapper[2562]: I0916 17:56:17.571592    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fw78m_9d27ec00-06e7-431b-925f-addd45ff0115/kube-rbac-proxy/0.log"