Sep 18 16:13:52.700034 ip-10-0-138-142 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 18 16:13:52.700047 ip-10-0-138-142 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 18 16:13:52.700054 ip-10-0-138-142 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 18 16:13:52.700346 ip-10-0-138-142 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 18 16:14:02.179354 ip-10-0-138-142 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Sep 18 16:14:02.179368 ip-10-0-138-142 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ba2898603f0441d79bf0ee0268761719 --
Sep 18 16:15:59.914400 ip-10-0-138-142 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 16:16:03.489828 ip-10-0-138-142 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:16:03.489828 ip-10-0-138-142 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 16:16:03.489828 ip-10-0-138-142 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:16:03.489828 ip-10-0-138-142 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 16:16:03.489828 ip-10-0-138-142 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:16:03.490710 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.489914    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 16:16:03.493882 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493862    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:16:03.493882 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493878    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:16:03.493882 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493883    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:16:03.493882 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493887    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:16:03.494145 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493891    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:16:03.494145 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493896    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:16:03.494145 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493899    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:16:03.494145 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493903    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:16:03.494145 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493907    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:16:03.494145 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493910    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:16:03.494145 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493924    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:16:03.494145 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493930    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:16:03.494145 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493935    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:16:03.494145 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493939    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:16:03.494145 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493944    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:16:03.494145 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493948    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:16:03.494145 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493952    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:16:03.494145 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493956    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:16:03.494145 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493961    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:16:03.494145 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493965    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:16:03.494145 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493969    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:16:03.494145 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493973    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:16:03.494145 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493977    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:16:03.494930 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493981    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:16:03.494930 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493985    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:16:03.494930 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493989    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:16:03.494930 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493993    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:16:03.494930 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.493997    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:16:03.494930 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494001    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:16:03.494930 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494005    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:16:03.494930 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494009    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:16:03.494930 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494013    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:16:03.494930 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494017    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:16:03.494930 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494022    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:16:03.494930 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494026    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:16:03.494930 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494033    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:16:03.494930 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494039    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:16:03.494930 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494044    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:16:03.494930 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494048    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:16:03.494930 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494053    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:16:03.494930 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494057    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:16:03.494930 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494062    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:16:03.494930 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494066    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:16:03.495839 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494070    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:16:03.495839 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494090    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:16:03.495839 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494095    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:16:03.495839 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494099    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:16:03.495839 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494104    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:16:03.495839 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494108    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:16:03.495839 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494114    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:16:03.495839 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494118    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:16:03.495839 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494122    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:16:03.495839 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494126    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:16:03.495839 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494131    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:16:03.495839 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494135    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:16:03.495839 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494139    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:16:03.495839 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494145    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:16:03.495839 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494150    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:16:03.495839 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494155    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:16:03.495839 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494160    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:16:03.495839 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494164    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:16:03.495839 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494169    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:16:03.496717 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494174    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:16:03.496717 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494180    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:16:03.496717 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494184    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:16:03.496717 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494190    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:16:03.496717 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494196    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:16:03.496717 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494239    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:16:03.496717 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494248    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:16:03.496717 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494254    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:16:03.496717 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494259    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:16:03.496717 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494264    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:16:03.496717 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494269    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:16:03.496717 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494273    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:16:03.496717 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494278    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:16:03.496717 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494282    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:16:03.496717 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494286    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:16:03.496717 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494290    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:16:03.496717 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494294    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:16:03.496717 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494298    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:16:03.496717 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494303    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:16:03.496717 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494307    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:16:03.497582 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494312    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:16:03.497582 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494316    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:16:03.497582 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494322    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:16:03.497582 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494326    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:16:03.497582 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494330    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:16:03.497582 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494335    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:16:03.497582 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494339    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:16:03.497582 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494343    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:16:03.497582 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494348    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:16:03.497582 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494353    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:16:03.497582 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494357    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:16:03.497582 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494361    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:16:03.497582 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494366    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:16:03.497582 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494370    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:16:03.497582 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494374    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:16:03.497582 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494379    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:16:03.497582 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.494383    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:16:03.497582 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.495956    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:16:03.497582 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.495966    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:16:03.498403 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.495971    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:16:03.498403 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.495976    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:16:03.498403 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.495981    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:16:03.498403 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.495985    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:16:03.498403 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.495990    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:16:03.498403 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.495994    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:16:03.498403 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.495998    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:16:03.498403 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496003    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:16:03.498403 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496007    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:16:03.498403 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496011    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:16:03.498403 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496015    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:16:03.498403 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496020    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:16:03.498403 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496024    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:16:03.498403 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496029    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:16:03.498403 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496033    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:16:03.498403 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496037    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:16:03.498403 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496041    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:16:03.498403 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496045    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:16:03.498403 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496049    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:16:03.498403 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496053    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:16:03.499225 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496058    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:16:03.499225 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496061    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:16:03.499225 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496066    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:16:03.499225 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496070    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:16:03.499225 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496092    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:16:03.499225 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496097    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:16:03.499225 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496103    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:16:03.499225 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496108    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:16:03.499225 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496113    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:16:03.499225 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496117    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:16:03.499225 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496122    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:16:03.499225 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496126    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:16:03.499225 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496131    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:16:03.499225 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496136    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:16:03.499225 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496140    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:16:03.499225 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496144    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:16:03.499225 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496148    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:16:03.499225 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496152    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:16:03.499225 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496156    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:16:03.499225 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496160    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:16:03.499860 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496165    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:16:03.499860 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496169    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:16:03.499860 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496178    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:16:03.499860 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496184    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:16:03.499860 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496189    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:16:03.499860 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496193    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:16:03.499860 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496198    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:16:03.499860 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496203    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:16:03.499860 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496208    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:16:03.499860 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496212    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:16:03.499860 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496217    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:16:03.499860 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496221    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:16:03.499860 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496226    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:16:03.499860 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496230    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:16:03.499860 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496235    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:16:03.499860 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496240    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:16:03.499860 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496244    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:16:03.499860 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496249    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:16:03.499860 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496253    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:16:03.500381 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496257    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:16:03.500381 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496262    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:16:03.500381 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496266    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:16:03.500381 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496270    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:16:03.500381 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496275    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:16:03.500381 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496280    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:16:03.500381 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496284    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:16:03.500381 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496292    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:16:03.500381 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496299    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:16:03.500381 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496305    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:16:03.500381 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496309    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:16:03.500381 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496313    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:16:03.500381 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496317    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:16:03.500381 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496322    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:16:03.500381 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496326    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:16:03.500381 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496330    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:16:03.500381 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496334    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:16:03.500381 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496338    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:16:03.500381 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496342    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:16:03.500840 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496346    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:16:03.500840 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496350    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:16:03.500840 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496354    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:16:03.500840 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496358    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:16:03.500840 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496363    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:16:03.500840 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496367    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:16:03.500840 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496372    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:16:03.500840 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496377    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:16:03.500840 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496381    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:16:03.500840 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496385    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:16:03.500840 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496389    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:16:03.500840 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496393    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:16:03.500840 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496397    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:16:03.500840 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496401    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:16:03.500840 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496406    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:16:03.500840 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496410    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:16:03.500840 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496414    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:16:03.500840 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496418    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:16:03.500840 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.496422    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:16:03.500840 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497010    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:16:03.501357 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497018    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:16:03.501357 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497023    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:16:03.501357 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497027    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:16:03.501357 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497031    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:16:03.501357 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497036    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:16:03.501357 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497040    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:16:03.501357 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497044    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:16:03.501357 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497048    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:16:03.501357 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497053    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:16:03.501357 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497057    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:16:03.501357 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497061    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:16:03.501357 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497065    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:16:03.501357 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497069    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:16:03.501357 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497091    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:16:03.501357 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497096    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:16:03.501357 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497100    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:16:03.501357 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497104    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:16:03.501357 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497108    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:16:03.501357 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497113    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:16:03.501803 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497117    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:16:03.501803 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497121    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:16:03.501803 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497125    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:16:03.501803 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497129    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:16:03.501803 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497133    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:16:03.501803 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497137    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:16:03.501803 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497142    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:16:03.501803 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497146    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:16:03.501803 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497150    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:16:03.501803 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497154    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:16:03.501803 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497158    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:16:03.501803 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497165    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:16:03.501803 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497171    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:16:03.501803 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497175    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:16:03.501803 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497180    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:16:03.501803 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497185    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:16:03.501803 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497189    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:16:03.501803 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497193    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:16:03.501803 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497197    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:16:03.501803 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497203    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:16:03.502320 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497207    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:16:03.502320 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497212    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:16:03.502320 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497216    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:16:03.502320 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497219    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:16:03.502320 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497224    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:16:03.502320 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497228    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:16:03.502320 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497232    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:16:03.502320 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497236    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:16:03.502320 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497240    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:16:03.502320 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497245    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:16:03.502320 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497249    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:16:03.502320 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497253    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:16:03.502320 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497257    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:16:03.502320 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497261    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:16:03.502320 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497265    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:16:03.502320 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497269    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:16:03.502320 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497274    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:16:03.502320 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497278    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:16:03.502320 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497282    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:16:03.502818 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497286    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:16:03.502818 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497290    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:16:03.502818 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497295    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:16:03.502818 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497299    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:16:03.502818 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497303    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:16:03.502818 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497307    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:16:03.502818 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497311    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:16:03.502818 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497315    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:16:03.502818 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497320    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:16:03.502818 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497328    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:16:03.502818 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497333    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:16:03.502818 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497338    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:16:03.502818 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497342    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:16:03.502818 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497346    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:16:03.502818 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497351    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:16:03.502818 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497355    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:16:03.502818 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497360    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:16:03.502818 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497364    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:16:03.502818 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497369    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:16:03.502818 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497372    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:16:03.503313 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497377    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:16:03.503313 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497381    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:16:03.503313 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497385    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:16:03.503313 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497390    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:16:03.503313 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497394    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:16:03.503313 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497398    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:16:03.503313 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497402    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:16:03.503313 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497408    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:16:03.503313 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497414    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:16:03.503313 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497418    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:16:03.503313 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497422    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:16:03.503313 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497426    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:16:03.503313 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497431    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:16:03.503313 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497435    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:16:03.503313 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497439    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:16:03.503313 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497443    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:16:03.503313 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497447    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:16:03.503313 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497451    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:16:03.503313 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497456    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:16:03.503313 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.497460    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:16:03.503809 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497557    2566 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 16:16:03.503809 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497603    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 16:16:03.503809 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497611    2566 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 16:16:03.503809 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497619    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 16:16:03.503809 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497626    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 16:16:03.503809 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497631    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 16:16:03.503809 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497638    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 16:16:03.503809 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497644    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 16:16:03.503809 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497649    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 16:16:03.503809 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497654    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 16:16:03.503809 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497659    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 16:16:03.503809 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497665    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 16:16:03.503809 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497670    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 16:16:03.503809 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497675    2566 flags.go:64] FLAG: --cgroup-root=""
Sep 18 16:16:03.503809 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497680    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 16:16:03.503809 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497684    2566 flags.go:64] FLAG: --client-ca-file=""
Sep 18 16:16:03.503809 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497689    2566 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 16:16:03.503809 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497693    2566 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 16:16:03.503809 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497700    2566 flags.go:64] FLAG: --cluster-domain=""
Sep 18 16:16:03.503809 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497704    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 16:16:03.503809 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497709    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 16:16:03.503809 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497714    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 16:16:03.503809 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497720    2566 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 16:16:03.503809 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497726    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 16:16:03.504414 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497730    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 16:16:03.504414 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497735    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 16:16:03.504414 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497741    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 16:16:03.504414 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497745    2566 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 16:16:03.504414 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497750    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 16:16:03.504414 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497755    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 16:16:03.504414 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497760    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 16:16:03.504414 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497765    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 16:16:03.504414 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497771    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 16:16:03.504414 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497775    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 16:16:03.504414 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497781    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 16:16:03.504414 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497786    2566 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 16:16:03.504414 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497791    2566 flags.go:64] FLAG: --enable-server="true"
Sep 18 16:16:03.504414 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497796    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 16:16:03.504414 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497801    2566 flags.go:64] FLAG: --event-burst="100"
Sep 18 16:16:03.504414 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497806    2566 flags.go:64] FLAG: --event-qps="50"
Sep 18 16:16:03.504414 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497811    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 16:16:03.504414 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497815    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 16:16:03.504414 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497820    2566 flags.go:64] FLAG: --eviction-hard=""
Sep 18 16:16:03.504414 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497826    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 16:16:03.504414 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497831    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 16:16:03.504414 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497836    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 16:16:03.504414 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497840    2566 flags.go:64] FLAG: --eviction-soft=""
Sep 18 16:16:03.504414 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497845    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 16:16:03.504414 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497849    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 16:16:03.505118 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497854    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 16:16:03.505118 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497858    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 16:16:03.505118 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497864    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 16:16:03.505118 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497868    2566 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 16:16:03.505118 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497874    2566 flags.go:64] FLAG: --feature-gates=""
Sep 18 16:16:03.505118 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497883    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 16:16:03.505118 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497887    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 16:16:03.505118 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497892    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 16:16:03.505118 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497897    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 16:16:03.505118 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497902    2566 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 16:16:03.505118 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497907    2566 flags.go:64] FLAG: --help="false"
Sep 18 16:16:03.505118 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497912    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-138-142.ec2.internal"
Sep 18 16:16:03.505118 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497917    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 16:16:03.505118 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497922    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 16:16:03.505118 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497926    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 16:16:03.505118 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497932    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 16:16:03.505118 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497937    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 16:16:03.505118 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497942    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 16:16:03.505118 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497948    2566 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 16:16:03.505118 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497952    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 16:16:03.505118 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497957    2566 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 16:16:03.505118 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497962    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 16:16:03.505118 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497967    2566 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 16:16:03.505118 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497973    2566 flags.go:64] FLAG: --kube-reserved=""
Sep 18 16:16:03.505712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497978    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 16:16:03.505712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497983    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 16:16:03.505712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497988    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 16:16:03.505712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497992    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 16:16:03.505712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.497997    2566 flags.go:64] FLAG: --lock-file=""
Sep 18 16:16:03.505712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498001    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 16:16:03.505712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498006    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 16:16:03.505712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498011    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 16:16:03.505712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498020    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 16:16:03.505712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498025    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 16:16:03.505712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498029    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 16:16:03.505712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498034    2566 flags.go:64] FLAG: --logging-format="text"
Sep 18 16:16:03.505712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498039    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 16:16:03.505712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498046    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 16:16:03.505712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498051    2566 flags.go:64] FLAG: --manifest-url=""
Sep 18 16:16:03.505712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498055    2566 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 16:16:03.505712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498062    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 16:16:03.505712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498067    2566 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 16:16:03.505712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498089    2566 flags.go:64] FLAG: --max-pods="110"
Sep 18 16:16:03.505712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498094    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 16:16:03.505712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498099    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 16:16:03.505712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498103    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 16:16:03.505712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498108    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 16:16:03.505712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498113    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 16:16:03.505712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498118    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 16:16:03.506387 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498123    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 16:16:03.506387 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498134    2566 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 16:16:03.506387 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498139    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 16:16:03.506387 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498145    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 16:16:03.506387 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498150    2566 flags.go:64] FLAG: --pod-cidr=""
Sep 18 16:16:03.506387 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498155    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 16:16:03.506387 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498162    2566 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 16:16:03.506387 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498169    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 16:16:03.506387 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498174    2566 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 16:16:03.506387 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498179    2566 flags.go:64] FLAG: --port="10250"
Sep 18 16:16:03.506387 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498184    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 16:16:03.506387 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498188    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-058862cfeee08ed08"
Sep 18 16:16:03.506387 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498193    2566 flags.go:64] FLAG: --qos-reserved=""
Sep 18 16:16:03.506387 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498198    2566 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 16:16:03.506387 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498203    2566 flags.go:64] FLAG: --register-node="true"
Sep 18 16:16:03.506387 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498207    2566 flags.go:64] FLAG: --register-with-taints=""
Sep 18 16:16:03.506387 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498213    2566 flags.go:64] FLAG: --registry-burst="10"
Sep 18 16:16:03.506387 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498217    2566 flags.go:64] FLAG: --registry-qps="5"
Sep 18 16:16:03.506387 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498222    2566 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 16:16:03.506387 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498227    2566 flags.go:64] FLAG: --reserved-memory=""
Sep 18 16:16:03.506387 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498234    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 16:16:03.506387 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498239    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 16:16:03.506387 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498244    2566 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 16:16:03.506387 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498248    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 16:16:03.506387 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498252    2566 flags.go:64] FLAG: --runonce="false"
Sep 18 16:16:03.507004 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498257    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 16:16:03.507004 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498262    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 16:16:03.507004 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498267    2566 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 16:16:03.507004 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498272    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 16:16:03.507004 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498276    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 16:16:03.507004 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498282    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 16:16:03.507004 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498286    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 16:16:03.507004 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498291    2566 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 16:16:03.507004 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498296    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 16:16:03.507004 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498300    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 16:16:03.507004 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498306    2566 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 16:16:03.507004 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498311    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 16:16:03.507004 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498316    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 16:16:03.507004 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498321    2566 flags.go:64] FLAG: --system-cgroups=""
Sep 18 16:16:03.507004 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498327    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 16:16:03.507004 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498335    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 16:16:03.507004 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498340    2566 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 16:16:03.507004 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498345    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 16:16:03.507004 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498351    2566 flags.go:64] FLAG: --tls-min-version=""
Sep 18 16:16:03.507004 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498355    2566 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 16:16:03.507004 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498360    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 16:16:03.507004 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498364    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 16:16:03.507004 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498368    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 16:16:03.507004 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498373    2566 flags.go:64] FLAG: --v="2"
Sep 18 16:16:03.507004 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498379    2566 flags.go:64] FLAG: --version="false"
Sep 18 16:16:03.507645 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498385    2566 flags.go:64] FLAG: --vmodule=""
Sep 18 16:16:03.507645 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498391    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 16:16:03.507645 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.498398    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 16:16:03.507645 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498572    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:16:03.507645 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498579    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:16:03.507645 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498584    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:16:03.507645 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498588    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:16:03.507645 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498593    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:16:03.507645 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498597    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:16:03.507645 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498601    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:16:03.507645 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498606    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:16:03.507645 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498610    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:16:03.507645 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498614    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:16:03.507645 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498618    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:16:03.507645 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498623    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:16:03.507645 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498627    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:16:03.507645 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498631    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:16:03.507645 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498636    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:16:03.507645 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498697    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:16:03.507645 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498749    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:16:03.508153 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498754    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:16:03.508153 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498762    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:16:03.508153 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498766    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:16:03.508153 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498782    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:16:03.508153 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498786    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:16:03.508153 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498791    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:16:03.508153 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498795    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:16:03.508153 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498800    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:16:03.508153 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498805    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:16:03.508153 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498810    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:16:03.508153 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498869    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:16:03.508153 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498888    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:16:03.508153 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498895    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:16:03.508153 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498900    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:16:03.508153 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498906    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:16:03.508153 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498910    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:16:03.508153 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498915    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:16:03.508153 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498919    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:16:03.508153 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498924    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:16:03.508153 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498929    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:16:03.508637 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498934    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:16:03.508637 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498938    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:16:03.508637 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498949    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:16:03.508637 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498954    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:16:03.508637 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498958    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:16:03.508637 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498963    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:16:03.508637 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498967    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:16:03.508637 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498972    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:16:03.508637 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498977    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:16:03.508637 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498981    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:16:03.508637 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498985    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:16:03.508637 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498990    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:16:03.508637 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498994    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:16:03.508637 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.498998    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:16:03.508637 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499003    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:16:03.508637 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499011    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:16:03.508637 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499015    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:16:03.508637 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499019    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:16:03.508637 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499023    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:16:03.508637 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499028    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:16:03.509152 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499032    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:16:03.509152 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499036    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:16:03.509152 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499040    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:16:03.509152 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499046    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:16:03.509152 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499052    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:16:03.509152 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499060    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:16:03.509152 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499067    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:16:03.509152 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499091    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:16:03.509152 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499098    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:16:03.509152 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499104    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:16:03.509152 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499109    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:16:03.509152 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499113    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:16:03.509152 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499118    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:16:03.509152 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499122    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:16:03.509152 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499127    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:16:03.509152 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499131    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:16:03.509152 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499135    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:16:03.509152 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499139    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:16:03.509152 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499143    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:16:03.509664 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499152    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:16:03.509664 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499157    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:16:03.509664 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499161    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:16:03.509664 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499165    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:16:03.509664 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499169    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:16:03.509664 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499173    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:16:03.509664 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499178    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:16:03.509664 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499184    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:16:03.509664 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499189    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:16:03.509664 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499194    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:16:03.509664 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499198    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:16:03.509664 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499203    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:16:03.509664 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499212    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:16:03.509664 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499217    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:16:03.509664 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499222    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:16:03.509664 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499227    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:16:03.509664 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499233    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:16:03.509664 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499239    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:16:03.509664 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499245    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:16:03.509664 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499249    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:16:03.510287 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499254    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:16:03.510287 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499259    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:16:03.510287 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.499262    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:16:03.510287 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.500004    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:16:03.510287 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.507731    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 16:16:03.510287 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.507746    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 16:16:03.510287 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.507767    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 16:16:03.510287 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.507773    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 16:16:03.510287 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507843    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:16:03.510287 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507848    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:16:03.510287 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507852    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:16:03.510287 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507855    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:16:03.510287 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507858    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:16:03.510287 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507861    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:16:03.510653 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507865    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:16:03.510653 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507870    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:16:03.510653 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507874    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:16:03.510653 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507876    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:16:03.510653 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507879    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:16:03.510653 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507883    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:16:03.510653 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507886    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:16:03.510653 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507889    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:16:03.510653 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507891    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:16:03.510653 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507894    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:16:03.510653 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507897    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:16:03.510653 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507900    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:16:03.510653 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507903    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:16:03.510653 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507906    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:16:03.510653 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507909    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:16:03.510653 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507912    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:16:03.510653 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507915    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:16:03.510653 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507918    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:16:03.510653 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507920    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:16:03.510653 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507923    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:16:03.511158 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507925    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:16:03.511158 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507928    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:16:03.511158 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507931    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:16:03.511158 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507934    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:16:03.511158 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507937    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:16:03.511158 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507939    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:16:03.511158 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507942    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:16:03.511158 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507945    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:16:03.511158 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507948    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:16:03.511158 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507950    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:16:03.511158 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507953    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:16:03.511158 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507955    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:16:03.511158 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507959    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:16:03.511158 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507961    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:16:03.511158 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507964    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:16:03.511158 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507967    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:16:03.511158 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507969    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:16:03.511158 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507972    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:16:03.511158 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507975    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:16:03.511158 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507978    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:16:03.511663 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507980    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:16:03.511663 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507983    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:16:03.511663 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507986    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:16:03.511663 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507989    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:16:03.511663 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507991    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:16:03.511663 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507994    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:16:03.511663 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507996    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:16:03.511663 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.507999    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:16:03.511663 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508003    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:16:03.511663 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508005    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:16:03.511663 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508008    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:16:03.511663 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508010    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:16:03.511663 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508013    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:16:03.511663 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508016    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:16:03.511663 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508019    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:16:03.511663 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508021    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:16:03.511663 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508025    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:16:03.511663 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508030    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:16:03.511663 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508032    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:16:03.511663 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508036    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:16:03.512478 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508039    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:16:03.512478 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508041    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:16:03.512478 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508044    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:16:03.512478 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508047    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:16:03.512478 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508050    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:16:03.512478 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508052    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:16:03.512478 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508055    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:16:03.512478 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508058    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:16:03.512478 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508061    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:16:03.512478 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508065    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:16:03.512478 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508068    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:16:03.512478 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508071    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:16:03.512478 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508086    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:16:03.512478 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508089    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:16:03.512478 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508092    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:16:03.512478 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508095    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:16:03.512478 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508098    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:16:03.512478 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508101    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:16:03.512478 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508103    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:16:03.513056 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508106    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:16:03.513056 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508109    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:16:03.513056 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508111    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:16:03.513056 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508114    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:16:03.513056 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508116    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:16:03.513056 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508119    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:16:03.513056 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508121    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:16:03.513056 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508124    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:16:03.513056 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508127    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:16:03.513056 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508130    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:16:03.513056 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508132    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:16:03.513056 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508135    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:16:03.513056 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508137    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:16:03.513056 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508140    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:16:03.513056 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.508145    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:16:03.513561 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508257    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:16:03.513561 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508262    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:16:03.513561 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508265    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:16:03.513561 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508268    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:16:03.513561 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508271    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:16:03.513561 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508273    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:16:03.513561 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508276    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:16:03.513561 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508280    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:16:03.513561 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508282    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:16:03.513561 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508286    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:16:03.513561 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508289    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:16:03.513561 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508291    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:16:03.513561 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508294    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:16:03.513561 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508296    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:16:03.513561 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508299    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:16:03.513561 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508302    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:16:03.513561 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508304    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:16:03.513561 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508307    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:16:03.513561 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508309    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:16:03.513561 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508312    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:16:03.514183 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508314    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:16:03.514183 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508317    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:16:03.514183 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508320    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:16:03.514183 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508322    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:16:03.514183 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508325    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:16:03.514183 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508328    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:16:03.514183 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508330    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:16:03.514183 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508333    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:16:03.514183 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508335    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:16:03.514183 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508338    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:16:03.514183 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508340    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:16:03.514183 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508343    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:16:03.514183 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508347    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:16:03.514183 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508349    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:16:03.514183 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508352    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:16:03.514183 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508355    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:16:03.514183 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508357    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:16:03.514183 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508360    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:16:03.514183 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508363    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:16:03.514688 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508365    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:16:03.514688 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508367    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:16:03.514688 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508370    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:16:03.514688 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508373    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:16:03.514688 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508376    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:16:03.514688 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508379    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:16:03.514688 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508381    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:16:03.514688 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508384    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:16:03.514688 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508387    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:16:03.514688 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508389    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:16:03.514688 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508392    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:16:03.514688 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508394    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:16:03.514688 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508397    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:16:03.514688 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508399    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:16:03.514688 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508402    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:16:03.514688 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508404    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:16:03.514688 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508407    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:16:03.514688 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508409    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:16:03.514688 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508412    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:16:03.514688 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508414    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:16:03.515191 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508417    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:16:03.515191 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508421    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:16:03.515191 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508424    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:16:03.515191 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508426    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:16:03.515191 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508430    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:16:03.515191 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508433    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:16:03.515191 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508437    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:16:03.515191 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508440    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:16:03.515191 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508443    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:16:03.515191 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508446    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:16:03.515191 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508449    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:16:03.515191 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508451    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:16:03.515191 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508454    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:16:03.515191 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508457    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:16:03.515191 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508459    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:16:03.515191 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508462    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:16:03.515191 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508465    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:16:03.515191 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508467    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:16:03.515191 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508470    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:16:03.515704 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508472    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:16:03.515704 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508475    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:16:03.515704 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508477    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:16:03.515704 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508480    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:16:03.515704 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508482    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:16:03.515704 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508485    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:16:03.515704 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508488    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:16:03.515704 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508490    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:16:03.515704 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508493    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:16:03.515704 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508496    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:16:03.515704 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508498    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:16:03.515704 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508501    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:16:03.515704 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508503    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:16:03.515704 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508506    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:16:03.515704 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508508    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:16:03.515704 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508511    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:16:03.515704 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508513    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:16:03.515704 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508516    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:16:03.515704 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508518    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:16:03.516385 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508522    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:16:03.516385 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:03.508526    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:16:03.516385 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.508531    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:16:03.516385 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.508648    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 16:16:03.516385 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.510603    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 16:16:03.516385 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.511581    2566 server.go:1019] "Starting client certificate rotation"
Sep 18 16:16:03.516385 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.511700    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:16:03.516385 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.511746    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:16:03.516385 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.514156    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 16:16:03.516385 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.515274    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 16:16:03.520767 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.520754    2566 log.go:25] "Validated CRI v1 runtime API"
Sep 18 16:16:03.522219 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.522203    2566 log.go:25] "Validated CRI v1 image API"
Sep 18 16:16:03.523087 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.523062    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 16:16:03.525548 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.525525    2566 fs.go:135] Filesystem UUIDs: map[154f3231-9876-4a63-845a-d75cf6dd2aaa:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fbdb67cc-2955-493d-88d6-2cb99cc865bc:/dev/nvme0n1p4]
Sep 18 16:16:03.525601 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.525549    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 16:16:03.531782 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.531667    2566 manager.go:216] Machine: {Timestamp:2026-09-18 16:16:03.531190472 +0000 UTC m=+1.986643543 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103623 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20a6295567cefe392741d4b4274d3b SystemUUID:ec20a629-5567-cefe-3927-41d4b4274d3b BootID:ba289860-3f04-41d7-9bf0-ee0268761719 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f7:fd:58:b3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f7:fd:58:b3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:f1:6d:43:a7:32 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 16:16:03.531782 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.531771    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 16:16:03.531915 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.531849    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 16:16:03.533588 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.533553    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 16:16:03.533723 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.533591    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-142.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 16:16:03.534758 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.534746    2566 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 16:16:03.534797 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.534760    2566 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 16:16:03.534797 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.534769    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 16:16:03.534797 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.534779    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 16:16:03.534887 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.534852    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 16:16:03.538257 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.538245    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 16:16:03.538370 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.538361    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 16:16:03.538400 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.538392    2566 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 16:16:03.538438 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.538402    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 16:16:03.538438 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.538416    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 16:16:03.538438 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.538432    2566 kubelet.go:398] "Adding apiserver pod source"
Sep 18 16:16:03.538438 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.538442    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 16:16:03.538779 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.538764    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 16:16:03.538841 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.538789    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 16:16:03.540989 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.540975    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 16:16:03.543643 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.543624    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 16:16:03.543735 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.543664    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 16:16:03.544766 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.544753    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 16:16:03.544840 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.544775    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 16:16:03.544840 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.544783    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 16:16:03.544840 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.544792    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 16:16:03.544840 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.544805    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 16:16:03.544840 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.544813    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 16:16:03.544840 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.544833    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 16:16:03.544840 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.544842    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 16:16:03.545114 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.544852    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 16:16:03.545114 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.544860    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 16:16:03.545114 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.544875    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 16:16:03.545114 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.544883    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 16:16:03.545114 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.544917    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 16:16:03.545114 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.544927    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 16:16:03.548676 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.548660    2566 server.go:1268] "Started kubelet"
Sep 18 16:16:03.548795 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.548749    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 16:16:03.548863 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.548766    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 16:16:03.548863 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.548832    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 16:16:03.549458 ip-10-0-138-142 systemd[1]: Started Kubernetes Kubelet.
Sep 18 16:16:03.550033 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.549475    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 16:16:03.550033 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.549882    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 16:16:03.551122 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.551105    2566 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 16:16:03.552418 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.552398    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-142.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:16:03.552509 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:03.552486    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-142.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:16:03.552563 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:03.552525    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:16:03.553201 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:03.553170    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 16:16:03.555623 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.555608    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 16:16:03.555690 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.555621    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 16:16:03.555823 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:03.555786    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-142.ec2.internal\" not found"
Sep 18 16:16:03.556206 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.556191    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 16:16:03.556297 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.556224    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 16:16:03.556359 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.556191    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 16:16:03.579530 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.579508    2566 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 16:16:03.579653 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.579640    2566 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 16:16:03.579864 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.579845    2566 factory.go:55] Registering systemd factory
Sep 18 16:16:03.579928 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.579868    2566 factory.go:223] Registration of the systemd container factory successfully
Sep 18 16:16:03.579986 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:03.579939    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 16:16:03.580262 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:03.580237    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-142.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 16:16:03.580385 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.580362    2566 factory.go:153] Registering CRI-O factory
Sep 18 16:16:03.580487 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.580393    2566 factory.go:223] Registration of the crio container factory successfully
Sep 18 16:16:03.580487 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.580452    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 16:16:03.580487 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.580475    2566 factory.go:103] Registering Raw factory
Sep 18 16:16:03.580619 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.580493    2566 manager.go:1196] Started watching for new ooms in manager
Sep 18 16:16:03.581394 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.581376    2566 manager.go:349] Starting recovery of all containers
Sep 18 16:16:03.592537 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.592515    2566 manager.go:354] Recovery completed
Sep 18 16:16:03.595462 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:03.594290    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-142.ec2.internal.18d67678e5b952a5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-142.ec2.internal,UID:ip-10-0-138-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-142.ec2.internal,},FirstTimestamp:2026-09-18 16:16:03.548631717 +0000 UTC m=+2.004084786,LastTimestamp:2026-09-18 16:16:03.548631717 +0000 UTC m=+2.004084786,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-142.ec2.internal,}"
Sep 18 16:16:03.596776 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.596760    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:03.598703 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.598678    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:03.598795 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.598714    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:03.598795 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.598728    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-142.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:03.599531 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.599517    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 16:16:03.599531 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.599528    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 16:16:03.599617 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.599542    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 16:16:03.601472 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.601462    2566 policy_none.go:49] "None policy: Start"
Sep 18 16:16:03.601512 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.601477    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 16:16:03.601512 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.601486    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 16:16:03.602699 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.602682    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:03.603382 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.603371    2566 policy_none.go:47] "Start"
Sep 18 16:16:03.603475 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.603459    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:03.603531 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.603492    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:03.603531 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.603509    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-142.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:03.605601 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.605433    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kwxl2"
Sep 18 16:16:03.609753 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:03.609681    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-142.ec2.internal.18d67678e8b5547a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-142.ec2.internal,UID:ip-10-0-138-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-142.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-142.ec2.internal,},FirstTimestamp:2026-09-18 16:16:03.59870169 +0000 UTC m=+2.054154767,LastTimestamp:2026-09-18 16:16:03.59870169 +0000 UTC m=+2.054154767,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-142.ec2.internal,}"
Sep 18 16:16:03.613370 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.613347    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kwxl2"
Sep 18 16:16:03.615838 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:03.615773    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-142.ec2.internal.18d67678e8b59fdf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-142.ec2.internal,UID:ip-10-0-138-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-142.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-142.ec2.internal,},FirstTimestamp:2026-09-18 16:16:03.598720991 +0000 UTC m=+2.054174062,LastTimestamp:2026-09-18 16:16:03.598720991 +0000 UTC m=+2.054174062,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-142.ec2.internal,}"
Sep 18 16:16:03.619850 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:03.619789    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-142.ec2.internal.18d67678e8b5d233  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-142.ec2.internal,UID:ip-10-0-138-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-142.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-142.ec2.internal,},FirstTimestamp:2026-09-18 16:16:03.598733875 +0000 UTC m=+2.054186945,LastTimestamp:2026-09-18 16:16:03.598733875 +0000 UTC m=+2.054186945,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-142.ec2.internal,}"
Sep 18 16:16:03.656255 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.650739    2566 manager.go:337] "Starting Device Plugin manager"
Sep 18 16:16:03.656255 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:03.650780    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 16:16:03.656255 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.650794    2566 server.go:87] "Starting device plugin registration server"
Sep 18 16:16:03.656255 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.651059    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 16:16:03.656255 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.651096    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 16:16:03.656255 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.651189    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 16:16:03.656255 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.651267    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 16:16:03.656255 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.651276    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 16:16:03.656255 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:03.651772    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 16:16:03.656255 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:03.651815    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-142.ec2.internal\" not found"
Sep 18 16:16:03.679330 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.679301    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 16:16:03.680446 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.680432    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 16:16:03.680513 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.680479    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 16:16:03.680513 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.680501    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 16:16:03.680616 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:03.680541    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 16:16:03.682732 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.682714    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:16:03.751720 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.751635    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:03.752727 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.752711    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:03.752818 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.752740    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:03.752818 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.752750    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-142.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:03.752818 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.752772    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-142.ec2.internal"
Sep 18 16:16:03.759242 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.759224    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-142.ec2.internal"
Sep 18 16:16:03.759338 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:03.759246    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-142.ec2.internal\": node \"ip-10-0-138-142.ec2.internal\" not found"
Sep 18 16:16:03.775067 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:03.775040    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-142.ec2.internal\" not found"
Sep 18 16:16:03.781229 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.781207    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-142.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-142.ec2.internal"]
Sep 18 16:16:03.781309 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.781297    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:03.782267 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.782253    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:03.782327 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.782283    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:03.782327 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.782294    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-142.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:03.785624 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.785604    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:03.785739 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.785718    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-142.ec2.internal"
Sep 18 16:16:03.785807 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.785748    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:03.786663 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.786642    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:03.786767 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.786672    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:03.786767 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.786682    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-142.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:03.786767 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.786645    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:03.786767 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.786752    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:03.786947 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.786770    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-142.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:03.788979 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.788962    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-142.ec2.internal"
Sep 18 16:16:03.789068 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.788992    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:03.789770 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.789754    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:03.789840 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.789780    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:03.789840 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.789796    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-142.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:03.822842 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:03.822818    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-142.ec2.internal\" not found" node="ip-10-0-138-142.ec2.internal"
Sep 18 16:16:03.827276 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:03.827258    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-142.ec2.internal\" not found" node="ip-10-0-138-142.ec2.internal"
Sep 18 16:16:03.875359 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:03.875323    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-142.ec2.internal\" not found"
Sep 18 16:16:03.880555 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.880531    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/46cbe6a567db993fd78904be901f7df3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-142.ec2.internal\" (UID: \"46cbe6a567db993fd78904be901f7df3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-142.ec2.internal"
Sep 18 16:16:03.880650 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.880567    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46cbe6a567db993fd78904be901f7df3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-142.ec2.internal\" (UID: \"46cbe6a567db993fd78904be901f7df3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-142.ec2.internal"
Sep 18 16:16:03.880650 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.880590    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/61ad4b4551fb28947d10dc4793af77f5-config\") pod \"kube-apiserver-proxy-ip-10-0-138-142.ec2.internal\" (UID: \"61ad4b4551fb28947d10dc4793af77f5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-142.ec2.internal"
Sep 18 16:16:03.976166 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:03.976136    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-142.ec2.internal\" not found"
Sep 18 16:16:03.981033 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.981015    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/61ad4b4551fb28947d10dc4793af77f5-config\") pod \"kube-apiserver-proxy-ip-10-0-138-142.ec2.internal\" (UID: \"61ad4b4551fb28947d10dc4793af77f5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-142.ec2.internal"
Sep 18 16:16:03.981130 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.981041    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/46cbe6a567db993fd78904be901f7df3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-142.ec2.internal\" (UID: \"46cbe6a567db993fd78904be901f7df3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-142.ec2.internal"
Sep 18 16:16:03.981130 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.981065    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46cbe6a567db993fd78904be901f7df3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-142.ec2.internal\" (UID: \"46cbe6a567db993fd78904be901f7df3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-142.ec2.internal"
Sep 18 16:16:03.981206 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.981129    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46cbe6a567db993fd78904be901f7df3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-142.ec2.internal\" (UID: \"46cbe6a567db993fd78904be901f7df3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-142.ec2.internal"
Sep 18 16:16:03.981206 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.981135    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/61ad4b4551fb28947d10dc4793af77f5-config\") pod \"kube-apiserver-proxy-ip-10-0-138-142.ec2.internal\" (UID: \"61ad4b4551fb28947d10dc4793af77f5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-142.ec2.internal"
Sep 18 16:16:03.981206 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:03.981145    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/46cbe6a567db993fd78904be901f7df3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-142.ec2.internal\" (UID: \"46cbe6a567db993fd78904be901f7df3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-142.ec2.internal"
Sep 18 16:16:04.076516 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:04.076491    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-142.ec2.internal\" not found"
Sep 18 16:16:04.126816 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:04.126785    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-142.ec2.internal"
Sep 18 16:16:04.130522 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:04.130501    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-142.ec2.internal"
Sep 18 16:16:04.176793 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:04.176759    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-142.ec2.internal\" not found"
Sep 18 16:16:04.277390 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:04.277355    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-142.ec2.internal\" not found"
Sep 18 16:16:04.377901 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:04.377819    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-142.ec2.internal\" not found"
Sep 18 16:16:04.403798 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:04.403772    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:16:04.478515 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:04.478478    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-142.ec2.internal\" not found"
Sep 18 16:16:04.511794 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:04.511764    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 16:16:04.512289 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:04.511910    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 16:16:04.512289 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:04.511964    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 16:16:04.579224 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:04.579192    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-142.ec2.internal\" not found"
Sep 18 16:16:04.619115 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:04.619032    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 16:11:03 +0000 UTC" deadline="2028-06-25 06:38:00.363344426 +0000 UTC"
Sep 18 16:16:04.619115 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:04.619105    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15494h21m55.744243149s"
Sep 18 16:16:04.672626 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:04.672540    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:04.673645 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:04.673626    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-142.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:04.673760 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:04.673656    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-142.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:04.673760 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:04.673665    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-142.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:04.679966 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:04.679948    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-142.ec2.internal\" not found"
Sep 18 16:16:04.780166 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:04.780125    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-142.ec2.internal\" not found"
Sep 18 16:16:04.783385 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:04.783367    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:16:04.856702 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:04.856667    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-142.ec2.internal"
Sep 18 16:16:04.870734 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:04.870715    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 16:16:04.872015 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:04.872001    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-142.ec2.internal"
Sep 18 16:16:04.881199 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:04.881178    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 16:16:04.906958 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:04.906929    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ad4b4551fb28947d10dc4793af77f5.slice/crio-5f3f3836ba598131242031255dcb4a2a6ae8709988a4cab8d064b0b81b4479dd WatchSource:0}: Error finding container 5f3f3836ba598131242031255dcb4a2a6ae8709988a4cab8d064b0b81b4479dd: Status 404 returned error can't find the container with id 5f3f3836ba598131242031255dcb4a2a6ae8709988a4cab8d064b0b81b4479dd
Sep 18 16:16:04.911994 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:04.911980    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:16:04.944414 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:04.944389    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46cbe6a567db993fd78904be901f7df3.slice/crio-aa80108d033fa584235727d286e65a100bf75715ffa8739025e57dd0de1edb2e WatchSource:0}: Error finding container aa80108d033fa584235727d286e65a100bf75715ffa8739025e57dd0de1edb2e: Status 404 returned error can't find the container with id aa80108d033fa584235727d286e65a100bf75715ffa8739025e57dd0de1edb2e
Sep 18 16:16:04.945713 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:04.945673    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:16:05.387967 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.387819    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:16:05.540143 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.540112    2566 apiserver.go:52] "Watching apiserver"
Sep 18 16:16:05.548343 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.548315    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 16:16:05.548754 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.548731    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-142.ec2.internal","openshift-multus/multus-additional-cni-plugins-7hjqm","openshift-dns/node-resolver-ptbtj","openshift-multus/multus-jzsfr","openshift-multus/network-metrics-daemon-zzxwf","openshift-network-diagnostics/network-check-target-vnmfs","openshift-network-operator/iptables-alerter-xt2cm","openshift-ovn-kubernetes/ovnkube-node-j92qj","kube-system/global-pull-secret-syncer-gwgv5","kube-system/konnectivity-agent-zcmfd","kube-system/kube-apiserver-proxy-ip-10-0-138-142.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw","openshift-cluster-node-tuning-operator/tuned-v26zp","openshift-image-registry/node-ca-szd7j"]
Sep 18 16:16:05.551319 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.551285    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xt2cm"
Sep 18 16:16:05.553769 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.553694    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.555835 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.554184    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j4q26\""
Sep 18 16:16:05.555835 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.554253    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 16:16:05.555835 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.554411    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 16:16:05.555835 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.554451    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:16:05.557169 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.557140    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-j27z2\""
Sep 18 16:16:05.557270 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.557252    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 16:16:05.557435 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.557417    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ptbtj"
Sep 18 16:16:05.557499 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.557440    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 16:16:05.557690 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.557674    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:16:05.559615 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.559584    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 16:16:05.562130 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.562109    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jsnfb\""
Sep 18 16:16:05.562674 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.562308    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.566876 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.563996    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 16:16:05.566876 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.564862    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 16:16:05.566876 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.565178    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 16:16:05.566876 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.565374    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cvtk6\""
Sep 18 16:16:05.566876 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.565544    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 16:16:05.566876 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.565738    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 16:16:05.566876 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.566385    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:05.566876 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:05.566454    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzxwf" podUID="bbd14141-b0c9-4e6d-aab3-c1091adc5ddf"
Sep 18 16:16:05.566876 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.566735    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:16:05.566876 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:05.566792    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnmfs" podUID="48d628be-136e-4521-a2fa-77f6c20b27d8"
Sep 18 16:16:05.569103 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.569050    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:05.569200 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:05.569124    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwgv5" podUID="94d26f9a-f3f1-417e-91bd-0e74248bbbb4"
Sep 18 16:16:05.569875 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.569820    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 16:16:05.571112 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.571093    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-szd7j"
Sep 18 16:16:05.573592 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.573574    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7hjqm"
Sep 18 16:16:05.575714 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.575695    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 16:16:05.575981 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.575962    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 16:16:05.576067 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.576014    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 16:16:05.576831 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.576330    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j44km\""
Sep 18 16:16:05.576831 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.576536    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 16:16:05.576831 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.576699    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 16:16:05.577452 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.577043    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zcmfd"
Sep 18 16:16:05.577452 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.577276    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pwtd9\""
Sep 18 16:16:05.580041 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.579591    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 16:16:05.580041 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.579774    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-t764l\""
Sep 18 16:16:05.580041 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.579822    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 16:16:05.581029 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.581011    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.583549 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.583061    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw"
Sep 18 16:16:05.584258 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.584238    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 16:16:05.584406 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.584388    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 16:16:05.584514 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.584498    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 16:16:05.584612 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.584583    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 16:16:05.586771 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.586754    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 16:16:05.586896 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.586874    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 16:16:05.586958 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.586815    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wdlx6\""
Sep 18 16:16:05.587099 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.587067    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 16:16:05.587321 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.587306    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 16:16:05.587434 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.587418    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 16:16:05.587491 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.587400    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v7wqf\""
Sep 18 16:16:05.587491 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.587481    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljjb5\" (UniqueName: \"kubernetes.io/projected/044f1a79-aadb-474d-9525-6ccdb4bcbdf0-kube-api-access-ljjb5\") pod \"iptables-alerter-xt2cm\" (UID: \"044f1a79-aadb-474d-9525-6ccdb4bcbdf0\") " pod="openshift-network-operator/iptables-alerter-xt2cm"
Sep 18 16:16:05.587586 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.587511    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-etc-systemd\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.587586 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.587535    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1d77222a-f546-4b33-b6be-fbfa3bf2f6d0-hosts-file\") pod \"node-resolver-ptbtj\" (UID: \"1d77222a-f546-4b33-b6be-fbfa3bf2f6d0\") " pod="openshift-dns/node-resolver-ptbtj"
Sep 18 16:16:05.587586 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.587558    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d77222a-f546-4b33-b6be-fbfa3bf2f6d0-tmp-dir\") pod \"node-resolver-ptbtj\" (UID: \"1d77222a-f546-4b33-b6be-fbfa3bf2f6d0\") " pod="openshift-dns/node-resolver-ptbtj"
Sep 18 16:16:05.587794 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.587589    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-system-cni-dir\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.587794 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.587643    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-host-var-lib-cni-bin\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.587794 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.587682    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-hostroot\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.587794 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.587710    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/044f1a79-aadb-474d-9525-6ccdb4bcbdf0-host-slash\") pod \"iptables-alerter-xt2cm\" (UID: \"044f1a79-aadb-474d-9525-6ccdb4bcbdf0\") " pod="openshift-network-operator/iptables-alerter-xt2cm"
Sep 18 16:16:05.587794 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.587734    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-etc-modprobe-d\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.587794 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.587773    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-multus-socket-dir-parent\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.588096 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.587817    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-multus-conf-dir\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.588096 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.587855    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/044f1a79-aadb-474d-9525-6ccdb4bcbdf0-iptables-alerter-script\") pod \"iptables-alerter-xt2cm\" (UID: \"044f1a79-aadb-474d-9525-6ccdb4bcbdf0\") " pod="openshift-network-operator/iptables-alerter-xt2cm"
Sep 18 16:16:05.588096 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.587892    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-etc-sysctl-conf\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.588096 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.587914    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-run\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.588096 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.587938    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a53312cc-6d62-41f2-b3ef-c7b41da2219c-tmp\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.588096 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.587969    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb932b90-e784-4e7e-8f10-64421f61edaa-cni-binary-copy\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.588096 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.587992    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-host-run-netns\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.588096 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.588029    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-host-var-lib-cni-multus\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.588096 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.588054    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-etc-kubernetes\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.588096 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.588089    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-lib-modules\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.588766 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.588122    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a53312cc-6d62-41f2-b3ef-c7b41da2219c-etc-tuned\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.588766 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.588153    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh58x\" (UniqueName: \"kubernetes.io/projected/1d77222a-f546-4b33-b6be-fbfa3bf2f6d0-kube-api-access-hh58x\") pod \"node-resolver-ptbtj\" (UID: \"1d77222a-f546-4b33-b6be-fbfa3bf2f6d0\") " pod="openshift-dns/node-resolver-ptbtj"
Sep 18 16:16:05.588766 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.588185    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-cnibin\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.588766 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.588215    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-etc-kubernetes\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.588766 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.588237    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-etc-sysctl-d\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.588766 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.588258    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-sys\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.588766 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.588278    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-var-lib-kubelet\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.588766 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.588299    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-host\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.588766 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.588322    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-host-var-lib-kubelet\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.588766 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.588347    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb932b90-e784-4e7e-8f10-64421f61edaa-multus-daemon-config\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.588766 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.588369    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-host-run-multus-certs\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.588766 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.588408    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pxqg\" (UniqueName: \"kubernetes.io/projected/eb932b90-e784-4e7e-8f10-64421f61edaa-kube-api-access-9pxqg\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.588766 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.588431    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-etc-sysconfig\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.588766 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.588454    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9ckh\" (UniqueName: \"kubernetes.io/projected/a53312cc-6d62-41f2-b3ef-c7b41da2219c-kube-api-access-k9ckh\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.588766 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.588477    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-multus-cni-dir\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.588766 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.588499    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-host-run-k8s-cni-cncf-io\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.605905 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.605881    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-np7g6"
Sep 18 16:16:05.613907 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.613351    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-np7g6"
Sep 18 16:16:05.659279 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.659195    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 16:16:05.685614 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.685555    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-142.ec2.internal" event={"ID":"61ad4b4551fb28947d10dc4793af77f5","Type":"ContainerStarted","Data":"5f3f3836ba598131242031255dcb4a2a6ae8709988a4cab8d064b0b81b4479dd"}
Sep 18 16:16:05.687047 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.687018    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-142.ec2.internal" event={"ID":"46cbe6a567db993fd78904be901f7df3","Type":"ContainerStarted","Data":"aa80108d033fa584235727d286e65a100bf75715ffa8739025e57dd0de1edb2e"}
Sep 18 16:16:05.689361 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689339    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-hostroot\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.689468 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689372    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a1fa079-6d52-4844-bded-44bf16212a8a-system-cni-dir\") pod \"multus-additional-cni-plugins-7hjqm\" (UID: \"1a1fa079-6d52-4844-bded-44bf16212a8a\") " pod="openshift-multus/multus-additional-cni-plugins-7hjqm"
Sep 18 16:16:05.689468 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689399    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a1fa079-6d52-4844-bded-44bf16212a8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7hjqm\" (UID: \"1a1fa079-6d52-4844-bded-44bf16212a8a\") " pod="openshift-multus/multus-additional-cni-plugins-7hjqm"
Sep 18 16:16:05.689468 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689423    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8434b53d-830b-4df4-8cf1-e8ed6cf92b9f-konnectivity-ca\") pod \"konnectivity-agent-zcmfd\" (UID: \"8434b53d-830b-4df4-8cf1-e8ed6cf92b9f\") " pod="kube-system/konnectivity-agent-zcmfd"
Sep 18 16:16:05.689468 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689449    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/044f1a79-aadb-474d-9525-6ccdb4bcbdf0-host-slash\") pod \"iptables-alerter-xt2cm\" (UID: \"044f1a79-aadb-474d-9525-6ccdb4bcbdf0\") " pod="openshift-network-operator/iptables-alerter-xt2cm"
Sep 18 16:16:05.689660 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689472    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-etc-modprobe-d\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.689660 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689496    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-run-openvswitch\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.689660 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689509    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-hostroot\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.689660 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689521    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-multus-socket-dir-parent\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.689660 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689551    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9665ba03-1cdc-46c9-84f2-3ca2137b66e2-serviceca\") pod \"node-ca-szd7j\" (UID: \"9665ba03-1cdc-46c9-84f2-3ca2137b66e2\") " pod="openshift-image-registry/node-ca-szd7j"
Sep 18 16:16:05.689660 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689572    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-multus-socket-dir-parent\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.689660 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689578    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cacc3615-823b-4dc2-aa72-a9f0d2f4677b-socket-dir\") pod \"aws-ebs-csi-driver-node-v4wcw\" (UID: \"cacc3615-823b-4dc2-aa72-a9f0d2f4677b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw"
Sep 18 16:16:05.689660 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689601    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cacc3615-823b-4dc2-aa72-a9f0d2f4677b-etc-selinux\") pod \"aws-ebs-csi-driver-node-v4wcw\" (UID: \"cacc3615-823b-4dc2-aa72-a9f0d2f4677b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw"
Sep 18 16:16:05.689660 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689621    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/044f1a79-aadb-474d-9525-6ccdb4bcbdf0-host-slash\") pod \"iptables-alerter-xt2cm\" (UID: \"044f1a79-aadb-474d-9525-6ccdb4bcbdf0\") " pod="openshift-network-operator/iptables-alerter-xt2cm"
Sep 18 16:16:05.689660 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689628    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/044f1a79-aadb-474d-9525-6ccdb4bcbdf0-iptables-alerter-script\") pod \"iptables-alerter-xt2cm\" (UID: \"044f1a79-aadb-474d-9525-6ccdb4bcbdf0\") " pod="openshift-network-operator/iptables-alerter-xt2cm"
Sep 18 16:16:05.689660 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689653    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb932b90-e784-4e7e-8f10-64421f61edaa-cni-binary-copy\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.690226 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689672    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-host-run-netns\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.690226 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689694    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs\") pod \"network-metrics-daemon-zzxwf\" (UID: \"bbd14141-b0c9-4e6d-aab3-c1091adc5ddf\") " pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:05.690226 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689720    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-etc-modprobe-d\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.690226 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689720    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9snn9\" (UniqueName: \"kubernetes.io/projected/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-kube-api-access-9snn9\") pod \"network-metrics-daemon-zzxwf\" (UID: \"bbd14141-b0c9-4e6d-aab3-c1091adc5ddf\") " pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:05.690226 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689813    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/30c9a034-ba9f-4ead-8685-316723e6852a-ovnkube-script-lib\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.690226 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689835    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkqxq\" (UniqueName: \"kubernetes.io/projected/30c9a034-ba9f-4ead-8685-316723e6852a-kube-api-access-pkqxq\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.690226 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689857    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cacc3615-823b-4dc2-aa72-a9f0d2f4677b-device-dir\") pod \"aws-ebs-csi-driver-node-v4wcw\" (UID: \"cacc3615-823b-4dc2-aa72-a9f0d2f4677b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw"
Sep 18 16:16:05.690226 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689880    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zck8g\" (UniqueName: \"kubernetes.io/projected/1a1fa079-6d52-4844-bded-44bf16212a8a-kube-api-access-zck8g\") pod \"multus-additional-cni-plugins-7hjqm\" (UID: \"1a1fa079-6d52-4844-bded-44bf16212a8a\") " pod="openshift-multus/multus-additional-cni-plugins-7hjqm"
Sep 18 16:16:05.690226 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689909    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-etc-kubernetes\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.690226 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689934    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-lib-modules\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.690226 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689958    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-etc-kubernetes\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.690226 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.689983    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.690226 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690009    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-etc-sysctl-d\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.690226 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690037    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-host-run-multus-certs\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.690226 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690061    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a1fa079-6d52-4844-bded-44bf16212a8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-7hjqm\" (UID: \"1a1fa079-6d52-4844-bded-44bf16212a8a\") " pod="openshift-multus/multus-additional-cni-plugins-7hjqm"
Sep 18 16:16:05.690226 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690104    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-etc-sysconfig\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.690226 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690130    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-multus-cni-dir\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.691030 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690156    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a1fa079-6d52-4844-bded-44bf16212a8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7hjqm\" (UID: \"1a1fa079-6d52-4844-bded-44bf16212a8a\") " pod="openshift-multus/multus-additional-cni-plugins-7hjqm"
Sep 18 16:16:05.691030 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690190    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-host-slash\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.691030 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690212    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-run-systemd\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.691030 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690239    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljjb5\" (UniqueName: \"kubernetes.io/projected/044f1a79-aadb-474d-9525-6ccdb4bcbdf0-kube-api-access-ljjb5\") pod \"iptables-alerter-xt2cm\" (UID: \"044f1a79-aadb-474d-9525-6ccdb4bcbdf0\") " pod="openshift-network-operator/iptables-alerter-xt2cm"
Sep 18 16:16:05.691030 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690263    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnfnb\" (UniqueName: \"kubernetes.io/projected/48d628be-136e-4521-a2fa-77f6c20b27d8-kube-api-access-rnfnb\") pod \"network-check-target-vnmfs\" (UID: \"48d628be-136e-4521-a2fa-77f6c20b27d8\") " pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:16:05.691030 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690288    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-dbus\") pod \"global-pull-secret-syncer-gwgv5\" (UID: \"94d26f9a-f3f1-417e-91bd-0e74248bbbb4\") " pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:05.691030 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690294    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/044f1a79-aadb-474d-9525-6ccdb4bcbdf0-iptables-alerter-script\") pod \"iptables-alerter-xt2cm\" (UID: \"044f1a79-aadb-474d-9525-6ccdb4bcbdf0\") " pod="openshift-network-operator/iptables-alerter-xt2cm"
Sep 18 16:16:05.691030 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690311    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-host-run-ovn-kubernetes\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.691030 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690336    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d77222a-f546-4b33-b6be-fbfa3bf2f6d0-tmp-dir\") pod \"node-resolver-ptbtj\" (UID: \"1d77222a-f546-4b33-b6be-fbfa3bf2f6d0\") " pod="openshift-dns/node-resolver-ptbtj"
Sep 18 16:16:05.691030 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690359    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-system-cni-dir\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.691030 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690383    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-original-pull-secret\") pod \"global-pull-secret-syncer-gwgv5\" (UID: \"94d26f9a-f3f1-417e-91bd-0e74248bbbb4\") " pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:05.691030 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690412    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-host-kubelet\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.691030 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690443    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-host-run-netns\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.691030 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690466    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-kubelet-config\") pod \"global-pull-secret-syncer-gwgv5\" (UID: \"94d26f9a-f3f1-417e-91bd-0e74248bbbb4\") " pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:05.691030 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690492    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a1fa079-6d52-4844-bded-44bf16212a8a-cnibin\") pod \"multus-additional-cni-plugins-7hjqm\" (UID: \"1a1fa079-6d52-4844-bded-44bf16212a8a\") " pod="openshift-multus/multus-additional-cni-plugins-7hjqm"
Sep 18 16:16:05.691030 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690517    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-log-socket\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.691842 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690540    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cacc3615-823b-4dc2-aa72-a9f0d2f4677b-sys-fs\") pod \"aws-ebs-csi-driver-node-v4wcw\" (UID: \"cacc3615-823b-4dc2-aa72-a9f0d2f4677b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw"
Sep 18 16:16:05.691842 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690565    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-multus-conf-dir\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.691842 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690587    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9665ba03-1cdc-46c9-84f2-3ca2137b66e2-host\") pod \"node-ca-szd7j\" (UID: \"9665ba03-1cdc-46c9-84f2-3ca2137b66e2\") " pod="openshift-image-registry/node-ca-szd7j"
Sep 18 16:16:05.691842 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690616    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/30c9a034-ba9f-4ead-8685-316723e6852a-ovn-node-metrics-cert\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.691842 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690641    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbfdn\" (UniqueName: \"kubernetes.io/projected/cacc3615-823b-4dc2-aa72-a9f0d2f4677b-kube-api-access-zbfdn\") pod \"aws-ebs-csi-driver-node-v4wcw\" (UID: \"cacc3615-823b-4dc2-aa72-a9f0d2f4677b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw"
Sep 18 16:16:05.691842 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690668    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-etc-sysctl-conf\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.691842 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690693    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-run\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.691842 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690717    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a53312cc-6d62-41f2-b3ef-c7b41da2219c-tmp\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.691842 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690748    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-host-var-lib-cni-multus\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.691842 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690772    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-var-lib-openvswitch\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.691842 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690797    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-host-cni-bin\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.691842 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690815    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb932b90-e784-4e7e-8f10-64421f61edaa-cni-binary-copy\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.691842 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690822    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/30c9a034-ba9f-4ead-8685-316723e6852a-ovnkube-config\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.691842 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690848    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a1fa079-6d52-4844-bded-44bf16212a8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7hjqm\" (UID: \"1a1fa079-6d52-4844-bded-44bf16212a8a\") " pod="openshift-multus/multus-additional-cni-plugins-7hjqm"
Sep 18 16:16:05.691842 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690873    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-host-run-netns\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.691842 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690891    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a53312cc-6d62-41f2-b3ef-c7b41da2219c-etc-tuned\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.691842 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690925    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh58x\" (UniqueName: \"kubernetes.io/projected/1d77222a-f546-4b33-b6be-fbfa3bf2f6d0-kube-api-access-hh58x\") pod \"node-resolver-ptbtj\" (UID: \"1d77222a-f546-4b33-b6be-fbfa3bf2f6d0\") " pod="openshift-dns/node-resolver-ptbtj"
Sep 18 16:16:05.692641 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690951    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-cnibin\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.692641 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.690977    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-systemd-units\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.692641 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691000    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-etc-openvswitch\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.692641 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691023    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-node-log\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.692641 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691047    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cacc3615-823b-4dc2-aa72-a9f0d2f4677b-registration-dir\") pod \"aws-ebs-csi-driver-node-v4wcw\" (UID: \"cacc3615-823b-4dc2-aa72-a9f0d2f4677b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw"
Sep 18 16:16:05.692641 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691090    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-sys\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.692641 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691117    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-var-lib-kubelet\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.692641 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691140    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-host\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.692641 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691164    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-host-var-lib-kubelet\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.692641 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691191    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb932b90-e784-4e7e-8f10-64421f61edaa-multus-daemon-config\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.692641 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691187    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 16:16:05.692641 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691221    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pxqg\" (UniqueName: \"kubernetes.io/projected/eb932b90-e784-4e7e-8f10-64421f61edaa-kube-api-access-9pxqg\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.692641 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691247    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qfwl\" (UniqueName: \"kubernetes.io/projected/9665ba03-1cdc-46c9-84f2-3ca2137b66e2-kube-api-access-8qfwl\") pod \"node-ca-szd7j\" (UID: \"9665ba03-1cdc-46c9-84f2-3ca2137b66e2\") " pod="openshift-image-registry/node-ca-szd7j"
Sep 18 16:16:05.692641 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691270    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-run-ovn\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.692641 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691295    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9ckh\" (UniqueName: \"kubernetes.io/projected/a53312cc-6d62-41f2-b3ef-c7b41da2219c-kube-api-access-k9ckh\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.692641 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691337    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-host-run-k8s-cni-cncf-io\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.692641 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691361    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-host-cni-netd\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.693486 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691385    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cacc3615-823b-4dc2-aa72-a9f0d2f4677b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v4wcw\" (UID: \"cacc3615-823b-4dc2-aa72-a9f0d2f4677b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw"
Sep 18 16:16:05.693486 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691421    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-etc-systemd\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.693486 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691464    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1d77222a-f546-4b33-b6be-fbfa3bf2f6d0-hosts-file\") pod \"node-resolver-ptbtj\" (UID: \"1d77222a-f546-4b33-b6be-fbfa3bf2f6d0\") " pod="openshift-dns/node-resolver-ptbtj"
Sep 18 16:16:05.693486 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691530    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-etc-kubernetes\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.693486 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691555    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-var-lib-kubelet\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.693486 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691590    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-etc-kubernetes\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.693486 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691603    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-cnibin\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.693486 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691649    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-sys\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.693486 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691689    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-host\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.693486 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691704    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-lib-modules\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.693486 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691722    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-host-var-lib-kubelet\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.693486 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691748    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-host-run-k8s-cni-cncf-io\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.693486 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691757    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8434b53d-830b-4df4-8cf1-e8ed6cf92b9f-agent-certs\") pod \"konnectivity-agent-zcmfd\" (UID: \"8434b53d-830b-4df4-8cf1-e8ed6cf92b9f\") " pod="kube-system/konnectivity-agent-zcmfd"
Sep 18 16:16:05.693486 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691786    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/30c9a034-ba9f-4ead-8685-316723e6852a-env-overrides\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.693486 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691815    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-host-var-lib-cni-bin\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.693486 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691888    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-host-var-lib-cni-bin\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.693486 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691939    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1d77222a-f546-4b33-b6be-fbfa3bf2f6d0-hosts-file\") pod \"node-resolver-ptbtj\" (UID: \"1d77222a-f546-4b33-b6be-fbfa3bf2f6d0\") " pod="openshift-dns/node-resolver-ptbtj"
Sep 18 16:16:05.693486 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.691977    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-etc-systemd\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.694327 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.692103    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-etc-sysctl-d\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.694327 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.692150    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-host-run-multus-certs\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.694327 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.692179    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb932b90-e784-4e7e-8f10-64421f61edaa-multus-daemon-config\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.694327 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.692205    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-etc-sysconfig\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.694327 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.692499    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-multus-cni-dir\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.694327 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.692627    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d77222a-f546-4b33-b6be-fbfa3bf2f6d0-tmp-dir\") pod \"node-resolver-ptbtj\" (UID: \"1d77222a-f546-4b33-b6be-fbfa3bf2f6d0\") " pod="openshift-dns/node-resolver-ptbtj"
Sep 18 16:16:05.694327 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.692670    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-system-cni-dir\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.694327 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.692714    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-host-var-lib-cni-multus\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.694327 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.692735    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb932b90-e784-4e7e-8f10-64421f61edaa-multus-conf-dir\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.694327 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.692849    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-etc-sysctl-conf\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.694327 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.692893    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a53312cc-6d62-41f2-b3ef-c7b41da2219c-run\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.696001 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.695925    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a53312cc-6d62-41f2-b3ef-c7b41da2219c-etc-tuned\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.696381 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.696340    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a53312cc-6d62-41f2-b3ef-c7b41da2219c-tmp\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.701243 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.701177    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pxqg\" (UniqueName: \"kubernetes.io/projected/eb932b90-e784-4e7e-8f10-64421f61edaa-kube-api-access-9pxqg\") pod \"multus-jzsfr\" (UID: \"eb932b90-e784-4e7e-8f10-64421f61edaa\") " pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.702164 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.701356    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljjb5\" (UniqueName: \"kubernetes.io/projected/044f1a79-aadb-474d-9525-6ccdb4bcbdf0-kube-api-access-ljjb5\") pod \"iptables-alerter-xt2cm\" (UID: \"044f1a79-aadb-474d-9525-6ccdb4bcbdf0\") " pod="openshift-network-operator/iptables-alerter-xt2cm"
Sep 18 16:16:05.702164 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.701452    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh58x\" (UniqueName: \"kubernetes.io/projected/1d77222a-f546-4b33-b6be-fbfa3bf2f6d0-kube-api-access-hh58x\") pod \"node-resolver-ptbtj\" (UID: \"1d77222a-f546-4b33-b6be-fbfa3bf2f6d0\") " pod="openshift-dns/node-resolver-ptbtj"
Sep 18 16:16:05.703618 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.703221    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9ckh\" (UniqueName: \"kubernetes.io/projected/a53312cc-6d62-41f2-b3ef-c7b41da2219c-kube-api-access-k9ckh\") pod \"tuned-v26zp\" (UID: \"a53312cc-6d62-41f2-b3ef-c7b41da2219c\") " pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.792430 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.792394    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-dbus\") pod \"global-pull-secret-syncer-gwgv5\" (UID: \"94d26f9a-f3f1-417e-91bd-0e74248bbbb4\") " pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:05.792611 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.792438    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-host-run-ovn-kubernetes\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.792611 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.792463    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-original-pull-secret\") pod \"global-pull-secret-syncer-gwgv5\" (UID: \"94d26f9a-f3f1-417e-91bd-0e74248bbbb4\") " pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:05.792611 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.792485    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-host-kubelet\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.792611 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.792505    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-host-run-netns\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.792611 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.792527    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-kubelet-config\") pod \"global-pull-secret-syncer-gwgv5\" (UID: \"94d26f9a-f3f1-417e-91bd-0e74248bbbb4\") " pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:05.792611 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.792592    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a1fa079-6d52-4844-bded-44bf16212a8a-cnibin\") pod \"multus-additional-cni-plugins-7hjqm\" (UID: \"1a1fa079-6d52-4844-bded-44bf16212a8a\") " pod="openshift-multus/multus-additional-cni-plugins-7hjqm"
Sep 18 16:16:05.792915 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.792615    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-log-socket\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.792915 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.792636    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cacc3615-823b-4dc2-aa72-a9f0d2f4677b-sys-fs\") pod \"aws-ebs-csi-driver-node-v4wcw\" (UID: \"cacc3615-823b-4dc2-aa72-a9f0d2f4677b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw"
Sep 18 16:16:05.792915 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.792659    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9665ba03-1cdc-46c9-84f2-3ca2137b66e2-host\") pod \"node-ca-szd7j\" (UID: \"9665ba03-1cdc-46c9-84f2-3ca2137b66e2\") " pod="openshift-image-registry/node-ca-szd7j"
Sep 18 16:16:05.792915 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.792684    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/30c9a034-ba9f-4ead-8685-316723e6852a-ovn-node-metrics-cert\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.792915 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.792708    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbfdn\" (UniqueName: \"kubernetes.io/projected/cacc3615-823b-4dc2-aa72-a9f0d2f4677b-kube-api-access-zbfdn\") pod \"aws-ebs-csi-driver-node-v4wcw\" (UID: \"cacc3615-823b-4dc2-aa72-a9f0d2f4677b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw"
Sep 18 16:16:05.792915 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.792738    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-var-lib-openvswitch\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.792915 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.792760    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-host-cni-bin\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.792915 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.792783    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/30c9a034-ba9f-4ead-8685-316723e6852a-ovnkube-config\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.792915 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.792807    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a1fa079-6d52-4844-bded-44bf16212a8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7hjqm\" (UID: \"1a1fa079-6d52-4844-bded-44bf16212a8a\") " pod="openshift-multus/multus-additional-cni-plugins-7hjqm"
Sep 18 16:16:05.792915 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.792873    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-systemd-units\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.792915 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.792899    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-etc-openvswitch\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.793451 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.792923    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-node-log\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.793451 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.792946    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cacc3615-823b-4dc2-aa72-a9f0d2f4677b-registration-dir\") pod \"aws-ebs-csi-driver-node-v4wcw\" (UID: \"cacc3615-823b-4dc2-aa72-a9f0d2f4677b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw"
Sep 18 16:16:05.793451 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.792978    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qfwl\" (UniqueName: \"kubernetes.io/projected/9665ba03-1cdc-46c9-84f2-3ca2137b66e2-kube-api-access-8qfwl\") pod \"node-ca-szd7j\" (UID: \"9665ba03-1cdc-46c9-84f2-3ca2137b66e2\") " pod="openshift-image-registry/node-ca-szd7j"
Sep 18 16:16:05.793451 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.793004    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-run-ovn\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.793451 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.793029    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-host-cni-netd\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.793451 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.793055    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cacc3615-823b-4dc2-aa72-a9f0d2f4677b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v4wcw\" (UID: \"cacc3615-823b-4dc2-aa72-a9f0d2f4677b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw"
Sep 18 16:16:05.793451 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.793133    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8434b53d-830b-4df4-8cf1-e8ed6cf92b9f-agent-certs\") pod \"konnectivity-agent-zcmfd\" (UID: \"8434b53d-830b-4df4-8cf1-e8ed6cf92b9f\") " pod="kube-system/konnectivity-agent-zcmfd"
Sep 18 16:16:05.793451 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.793158    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/30c9a034-ba9f-4ead-8685-316723e6852a-env-overrides\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.793451 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.793185    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a1fa079-6d52-4844-bded-44bf16212a8a-system-cni-dir\") pod \"multus-additional-cni-plugins-7hjqm\" (UID: \"1a1fa079-6d52-4844-bded-44bf16212a8a\") " pod="openshift-multus/multus-additional-cni-plugins-7hjqm"
Sep 18 16:16:05.793451 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.793212    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a1fa079-6d52-4844-bded-44bf16212a8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7hjqm\" (UID: \"1a1fa079-6d52-4844-bded-44bf16212a8a\") " pod="openshift-multus/multus-additional-cni-plugins-7hjqm"
Sep 18 16:16:05.793451 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.793238    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8434b53d-830b-4df4-8cf1-e8ed6cf92b9f-konnectivity-ca\") pod \"konnectivity-agent-zcmfd\" (UID: \"8434b53d-830b-4df4-8cf1-e8ed6cf92b9f\") " pod="kube-system/konnectivity-agent-zcmfd"
Sep 18 16:16:05.793451 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.793266    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-run-openvswitch\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.793451 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.793291    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9665ba03-1cdc-46c9-84f2-3ca2137b66e2-serviceca\") pod \"node-ca-szd7j\" (UID: \"9665ba03-1cdc-46c9-84f2-3ca2137b66e2\") " pod="openshift-image-registry/node-ca-szd7j"
Sep 18 16:16:05.793451 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.793315    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cacc3615-823b-4dc2-aa72-a9f0d2f4677b-socket-dir\") pod \"aws-ebs-csi-driver-node-v4wcw\" (UID: \"cacc3615-823b-4dc2-aa72-a9f0d2f4677b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw"
Sep 18 16:16:05.793451 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.793367    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cacc3615-823b-4dc2-aa72-a9f0d2f4677b-etc-selinux\") pod \"aws-ebs-csi-driver-node-v4wcw\" (UID: \"cacc3615-823b-4dc2-aa72-a9f0d2f4677b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw"
Sep 18 16:16:05.793451 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.793397    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs\") pod \"network-metrics-daemon-zzxwf\" (UID: \"bbd14141-b0c9-4e6d-aab3-c1091adc5ddf\") " pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:05.793451 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.793421    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9snn9\" (UniqueName: \"kubernetes.io/projected/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-kube-api-access-9snn9\") pod \"network-metrics-daemon-zzxwf\" (UID: \"bbd14141-b0c9-4e6d-aab3-c1091adc5ddf\") " pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:05.794201 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.793446    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/30c9a034-ba9f-4ead-8685-316723e6852a-ovnkube-script-lib\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.794201 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.793471    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkqxq\" (UniqueName: \"kubernetes.io/projected/30c9a034-ba9f-4ead-8685-316723e6852a-kube-api-access-pkqxq\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.794201 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.793496    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cacc3615-823b-4dc2-aa72-a9f0d2f4677b-device-dir\") pod \"aws-ebs-csi-driver-node-v4wcw\" (UID: \"cacc3615-823b-4dc2-aa72-a9f0d2f4677b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw"
Sep 18 16:16:05.794201 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.793523    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zck8g\" (UniqueName: \"kubernetes.io/projected/1a1fa079-6d52-4844-bded-44bf16212a8a-kube-api-access-zck8g\") pod \"multus-additional-cni-plugins-7hjqm\" (UID: \"1a1fa079-6d52-4844-bded-44bf16212a8a\") " pod="openshift-multus/multus-additional-cni-plugins-7hjqm"
Sep 18 16:16:05.794201 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.793554    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.794201 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.793607    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a1fa079-6d52-4844-bded-44bf16212a8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-7hjqm\" (UID: \"1a1fa079-6d52-4844-bded-44bf16212a8a\") " pod="openshift-multus/multus-additional-cni-plugins-7hjqm"
Sep 18 16:16:05.794201 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.793637    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a1fa079-6d52-4844-bded-44bf16212a8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7hjqm\" (UID: \"1a1fa079-6d52-4844-bded-44bf16212a8a\") " pod="openshift-multus/multus-additional-cni-plugins-7hjqm"
Sep 18 16:16:05.794201 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.793661    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-host-slash\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.794201 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.793685    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-run-systemd\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.794201 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.793714    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnfnb\" (UniqueName: \"kubernetes.io/projected/48d628be-136e-4521-a2fa-77f6c20b27d8-kube-api-access-rnfnb\") pod \"network-check-target-vnmfs\" (UID: \"48d628be-136e-4521-a2fa-77f6c20b27d8\") " pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:16:05.794201 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.794159    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-dbus\") pod \"global-pull-secret-syncer-gwgv5\" (UID: \"94d26f9a-f3f1-417e-91bd-0e74248bbbb4\") " pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:05.794697 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.794220    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-host-run-ovn-kubernetes\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.794697 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:05.794306    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:05.794697 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:05.794390    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-original-pull-secret podName:94d26f9a-f3f1-417e-91bd-0e74248bbbb4 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:06.294357369 +0000 UTC m=+4.749810445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-original-pull-secret") pod "global-pull-secret-syncer-gwgv5" (UID: "94d26f9a-f3f1-417e-91bd-0e74248bbbb4") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:05.794697 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.794648    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-host-kubelet\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.794904 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.794697    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-host-run-netns\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.794904 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.794747    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-kubelet-config\") pod \"global-pull-secret-syncer-gwgv5\" (UID: \"94d26f9a-f3f1-417e-91bd-0e74248bbbb4\") " pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:05.794904 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.794790    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a1fa079-6d52-4844-bded-44bf16212a8a-cnibin\") pod \"multus-additional-cni-plugins-7hjqm\" (UID: \"1a1fa079-6d52-4844-bded-44bf16212a8a\") " pod="openshift-multus/multus-additional-cni-plugins-7hjqm"
Sep 18 16:16:05.794904 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.794832    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-log-socket\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.794904 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.794882    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cacc3615-823b-4dc2-aa72-a9f0d2f4677b-sys-fs\") pod \"aws-ebs-csi-driver-node-v4wcw\" (UID: \"cacc3615-823b-4dc2-aa72-a9f0d2f4677b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw"
Sep 18 16:16:05.795145 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.794923    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9665ba03-1cdc-46c9-84f2-3ca2137b66e2-host\") pod \"node-ca-szd7j\" (UID: \"9665ba03-1cdc-46c9-84f2-3ca2137b66e2\") " pod="openshift-image-registry/node-ca-szd7j"
Sep 18 16:16:05.795663 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.795637    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-var-lib-openvswitch\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.795784 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.795697    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-host-cni-bin\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.795784 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.795716    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-run-openvswitch\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.796084 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.796053    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a1fa079-6d52-4844-bded-44bf16212a8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-7hjqm\" (UID: \"1a1fa079-6d52-4844-bded-44bf16212a8a\") " pod="openshift-multus/multus-additional-cni-plugins-7hjqm"
Sep 18 16:16:05.796182 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.796154    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9665ba03-1cdc-46c9-84f2-3ca2137b66e2-serviceca\") pod \"node-ca-szd7j\" (UID: \"9665ba03-1cdc-46c9-84f2-3ca2137b66e2\") " pod="openshift-image-registry/node-ca-szd7j"
Sep 18 16:16:05.796224 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.796198    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.796264 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.796251    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-host-slash\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.796405 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.796386    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8434b53d-830b-4df4-8cf1-e8ed6cf92b9f-konnectivity-ca\") pod \"konnectivity-agent-zcmfd\" (UID: \"8434b53d-830b-4df4-8cf1-e8ed6cf92b9f\") " pod="kube-system/konnectivity-agent-zcmfd"
Sep 18 16:16:05.796439 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.796387    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a1fa079-6d52-4844-bded-44bf16212a8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7hjqm\" (UID: \"1a1fa079-6d52-4844-bded-44bf16212a8a\") " pod="openshift-multus/multus-additional-cni-plugins-7hjqm"
Sep 18 16:16:05.796439 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.796424    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-run-systemd\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.796513 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.796464    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cacc3615-823b-4dc2-aa72-a9f0d2f4677b-socket-dir\") pod \"aws-ebs-csi-driver-node-v4wcw\" (UID: \"cacc3615-823b-4dc2-aa72-a9f0d2f4677b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw"
Sep 18 16:16:05.796583 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.796567    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cacc3615-823b-4dc2-aa72-a9f0d2f4677b-etc-selinux\") pod \"aws-ebs-csi-driver-node-v4wcw\" (UID: \"cacc3615-823b-4dc2-aa72-a9f0d2f4677b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw"
Sep 18 16:16:05.796682 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:05.796669    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:05.796759 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:05.796749    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs podName:bbd14141-b0c9-4e6d-aab3-c1091adc5ddf nodeName:}" failed. No retries permitted until 2026-09-18 16:16:06.296721972 +0000 UTC m=+4.752175029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs") pod "network-metrics-daemon-zzxwf" (UID: "bbd14141-b0c9-4e6d-aab3-c1091adc5ddf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:05.796900 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.796882    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a1fa079-6d52-4844-bded-44bf16212a8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7hjqm\" (UID: \"1a1fa079-6d52-4844-bded-44bf16212a8a\") " pod="openshift-multus/multus-additional-cni-plugins-7hjqm"
Sep 18 16:16:05.796959 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.796945    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-systemd-units\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.796992 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.796156    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cacc3615-823b-4dc2-aa72-a9f0d2f4677b-device-dir\") pod \"aws-ebs-csi-driver-node-v4wcw\" (UID: \"cacc3615-823b-4dc2-aa72-a9f0d2f4677b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw"
Sep 18 16:16:05.797055 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.797037    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/30c9a034-ba9f-4ead-8685-316723e6852a-ovnkube-config\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.797103 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.797047    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cacc3615-823b-4dc2-aa72-a9f0d2f4677b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v4wcw\" (UID: \"cacc3615-823b-4dc2-aa72-a9f0d2f4677b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw"
Sep 18 16:16:05.797103 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.797098    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-run-ovn\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.797171 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.797143    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-etc-openvswitch\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.797205 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.797188    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-host-cni-netd\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.797245 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.797231    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/30c9a034-ba9f-4ead-8685-316723e6852a-node-log\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.797304 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.797291    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cacc3615-823b-4dc2-aa72-a9f0d2f4677b-registration-dir\") pod \"aws-ebs-csi-driver-node-v4wcw\" (UID: \"cacc3615-823b-4dc2-aa72-a9f0d2f4677b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw"
Sep 18 16:16:05.797352 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.797339    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a1fa079-6d52-4844-bded-44bf16212a8a-system-cni-dir\") pod \"multus-additional-cni-plugins-7hjqm\" (UID: \"1a1fa079-6d52-4844-bded-44bf16212a8a\") " pod="openshift-multus/multus-additional-cni-plugins-7hjqm"
Sep 18 16:16:05.797507 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.797489    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/30c9a034-ba9f-4ead-8685-316723e6852a-ovnkube-script-lib\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.797714 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.797695    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/30c9a034-ba9f-4ead-8685-316723e6852a-env-overrides\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.797759 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.797698    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/30c9a034-ba9f-4ead-8685-316723e6852a-ovn-node-metrics-cert\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.797972 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.797937    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a1fa079-6d52-4844-bded-44bf16212a8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7hjqm\" (UID: \"1a1fa079-6d52-4844-bded-44bf16212a8a\") " pod="openshift-multus/multus-additional-cni-plugins-7hjqm"
Sep 18 16:16:05.799756 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.799732    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8434b53d-830b-4df4-8cf1-e8ed6cf92b9f-agent-certs\") pod \"konnectivity-agent-zcmfd\" (UID: \"8434b53d-830b-4df4-8cf1-e8ed6cf92b9f\") " pod="kube-system/konnectivity-agent-zcmfd"
Sep 18 16:16:05.802931 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:05.802909    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:05.803043 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:05.802938    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:05.803043 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:05.802952    2566 projected.go:196] Error preparing data for projected volume kube-api-access-rnfnb for pod openshift-network-diagnostics/network-check-target-vnmfs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:05.803043 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:05.803024    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48d628be-136e-4521-a2fa-77f6c20b27d8-kube-api-access-rnfnb podName:48d628be-136e-4521-a2fa-77f6c20b27d8 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:06.303004237 +0000 UTC m=+4.758457296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rnfnb" (UniqueName: "kubernetes.io/projected/48d628be-136e-4521-a2fa-77f6c20b27d8-kube-api-access-rnfnb") pod "network-check-target-vnmfs" (UID: "48d628be-136e-4521-a2fa-77f6c20b27d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:05.808559 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.808483    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qfwl\" (UniqueName: \"kubernetes.io/projected/9665ba03-1cdc-46c9-84f2-3ca2137b66e2-kube-api-access-8qfwl\") pod \"node-ca-szd7j\" (UID: \"9665ba03-1cdc-46c9-84f2-3ca2137b66e2\") " pod="openshift-image-registry/node-ca-szd7j"
Sep 18 16:16:05.808674 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.808606    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbfdn\" (UniqueName: \"kubernetes.io/projected/cacc3615-823b-4dc2-aa72-a9f0d2f4677b-kube-api-access-zbfdn\") pod \"aws-ebs-csi-driver-node-v4wcw\" (UID: \"cacc3615-823b-4dc2-aa72-a9f0d2f4677b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw"
Sep 18 16:16:05.808912 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.808888    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkqxq\" (UniqueName: \"kubernetes.io/projected/30c9a034-ba9f-4ead-8685-316723e6852a-kube-api-access-pkqxq\") pod \"ovnkube-node-j92qj\" (UID: \"30c9a034-ba9f-4ead-8685-316723e6852a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.810112 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.810050    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9snn9\" (UniqueName: \"kubernetes.io/projected/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-kube-api-access-9snn9\") pod \"network-metrics-daemon-zzxwf\" (UID: \"bbd14141-b0c9-4e6d-aab3-c1091adc5ddf\") " pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:05.810112 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.810071    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zck8g\" (UniqueName: \"kubernetes.io/projected/1a1fa079-6d52-4844-bded-44bf16212a8a-kube-api-access-zck8g\") pod \"multus-additional-cni-plugins-7hjqm\" (UID: \"1a1fa079-6d52-4844-bded-44bf16212a8a\") " pod="openshift-multus/multus-additional-cni-plugins-7hjqm"
Sep 18 16:16:05.865612 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.865580    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xt2cm"
Sep 18 16:16:05.873471 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.873445    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v26zp"
Sep 18 16:16:05.885124 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.885100    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ptbtj"
Sep 18 16:16:05.892864 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.892842    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jzsfr"
Sep 18 16:16:05.899577 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.899557    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-szd7j"
Sep 18 16:16:05.909321 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.909262    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7hjqm"
Sep 18 16:16:05.917192 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.917162    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zcmfd"
Sep 18 16:16:05.925099 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.925065    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:05.932835 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:05.932809    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw"
Sep 18 16:16:06.296366 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:06.296280    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-original-pull-secret\") pod \"global-pull-secret-syncer-gwgv5\" (UID: \"94d26f9a-f3f1-417e-91bd-0e74248bbbb4\") " pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:06.296494 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:06.296405    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:06.296494 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:06.296457    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-original-pull-secret podName:94d26f9a-f3f1-417e-91bd-0e74248bbbb4 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:07.296445422 +0000 UTC m=+5.751898477 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-original-pull-secret") pod "global-pull-secret-syncer-gwgv5" (UID: "94d26f9a-f3f1-417e-91bd-0e74248bbbb4") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:06.396981 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:06.396942    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs\") pod \"network-metrics-daemon-zzxwf\" (UID: \"bbd14141-b0c9-4e6d-aab3-c1091adc5ddf\") " pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:06.397180 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:06.397009    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnfnb\" (UniqueName: \"kubernetes.io/projected/48d628be-136e-4521-a2fa-77f6c20b27d8-kube-api-access-rnfnb\") pod \"network-check-target-vnmfs\" (UID: \"48d628be-136e-4521-a2fa-77f6c20b27d8\") " pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:16:06.397718 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:06.397674    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:06.397835 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:06.397745    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:06.397835 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:06.397774    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:06.397835 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:06.397789    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs podName:bbd14141-b0c9-4e6d-aab3-c1091adc5ddf nodeName:}" failed. No retries permitted until 2026-09-18 16:16:07.397757766 +0000 UTC m=+5.853210837 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs") pod "network-metrics-daemon-zzxwf" (UID: "bbd14141-b0c9-4e6d-aab3-c1091adc5ddf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:06.397835 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:06.397791    2566 projected.go:196] Error preparing data for projected volume kube-api-access-rnfnb for pod openshift-network-diagnostics/network-check-target-vnmfs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:06.398028 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:06.397857    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48d628be-136e-4521-a2fa-77f6c20b27d8-kube-api-access-rnfnb podName:48d628be-136e-4521-a2fa-77f6c20b27d8 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:07.397840146 +0000 UTC m=+5.853293215 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rnfnb" (UniqueName: "kubernetes.io/projected/48d628be-136e-4521-a2fa-77f6c20b27d8-kube-api-access-rnfnb") pod "network-check-target-vnmfs" (UID: "48d628be-136e-4521-a2fa-77f6c20b27d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:06.616756 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:06.616692    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 16:11:05 +0000 UTC" deadline="2028-04-08 06:17:40.071549345 +0000 UTC"
Sep 18 16:16:06.616756 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:06.616746    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13622h1m33.454806617s"
Sep 18 16:16:06.783606 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:06.783576    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d77222a_f546_4b33_b6be_fbfa3bf2f6d0.slice/crio-1cf6d00b5218f9c5b40522424f7ed43549968f253d6eb6d2d90629b1183635bf WatchSource:0}: Error finding container 1cf6d00b5218f9c5b40522424f7ed43549968f253d6eb6d2d90629b1183635bf: Status 404 returned error can't find the container with id 1cf6d00b5218f9c5b40522424f7ed43549968f253d6eb6d2d90629b1183635bf
Sep 18 16:16:06.784836 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:06.784817    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a1fa079_6d52_4844_bded_44bf16212a8a.slice/crio-562410bde47dde665b8d00b5b991171694e04781eae0850475b9ae0031b2f2cb WatchSource:0}: Error finding container 562410bde47dde665b8d00b5b991171694e04781eae0850475b9ae0031b2f2cb: Status 404 returned error can't find the container with id 562410bde47dde665b8d00b5b991171694e04781eae0850475b9ae0031b2f2cb
Sep 18 16:16:06.786361 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:06.786249    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c9a034_ba9f_4ead_8685_316723e6852a.slice/crio-aa7325d1ddad7ba01e56aceb899d3d4e7cde7d9631d95d8f0beb86de1ac6baa9 WatchSource:0}: Error finding container aa7325d1ddad7ba01e56aceb899d3d4e7cde7d9631d95d8f0beb86de1ac6baa9: Status 404 returned error can't find the container with id aa7325d1ddad7ba01e56aceb899d3d4e7cde7d9631d95d8f0beb86de1ac6baa9
Sep 18 16:16:06.789922 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:06.789605    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8434b53d_830b_4df4_8cf1_e8ed6cf92b9f.slice/crio-d4e8e455f080b1958b8049fbcbd690f23a50b7c9aec6bdcd349d8a15506c6504 WatchSource:0}: Error finding container d4e8e455f080b1958b8049fbcbd690f23a50b7c9aec6bdcd349d8a15506c6504: Status 404 returned error can't find the container with id d4e8e455f080b1958b8049fbcbd690f23a50b7c9aec6bdcd349d8a15506c6504
Sep 18 16:16:06.790436 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:06.790408    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcacc3615_823b_4dc2_aa72_a9f0d2f4677b.slice/crio-8b3ab7c98d9da88bb6169278bc7037aeb8cd0952bb6a82e525dfe1a4008142c8 WatchSource:0}: Error finding container 8b3ab7c98d9da88bb6169278bc7037aeb8cd0952bb6a82e525dfe1a4008142c8: Status 404 returned error can't find the container with id 8b3ab7c98d9da88bb6169278bc7037aeb8cd0952bb6a82e525dfe1a4008142c8
Sep 18 16:16:06.791983 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:06.791828    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb932b90_e784_4e7e_8f10_64421f61edaa.slice/crio-42a247b8ba7902d2702b719525e71e9fe59a1d578df669ab40fb0e31b27cc678 WatchSource:0}: Error finding container 42a247b8ba7902d2702b719525e71e9fe59a1d578df669ab40fb0e31b27cc678: Status 404 returned error can't find the container with id 42a247b8ba7902d2702b719525e71e9fe59a1d578df669ab40fb0e31b27cc678
Sep 18 16:16:06.794955 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:06.793688    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044f1a79_aadb_474d_9525_6ccdb4bcbdf0.slice/crio-15b65b54a415b748b16e3ef56944daedc3d57ed1cce0e5d7b1d9956e64378e82 WatchSource:0}: Error finding container 15b65b54a415b748b16e3ef56944daedc3d57ed1cce0e5d7b1d9956e64378e82: Status 404 returned error can't find the container with id 15b65b54a415b748b16e3ef56944daedc3d57ed1cce0e5d7b1d9956e64378e82
Sep 18 16:16:07.304217 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:07.304026    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-original-pull-secret\") pod \"global-pull-secret-syncer-gwgv5\" (UID: \"94d26f9a-f3f1-417e-91bd-0e74248bbbb4\") " pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:07.304380 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:07.304210    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:07.304380 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:07.304290    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-original-pull-secret podName:94d26f9a-f3f1-417e-91bd-0e74248bbbb4 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:09.304269333 +0000 UTC m=+7.759722396 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-original-pull-secret") pod "global-pull-secret-syncer-gwgv5" (UID: "94d26f9a-f3f1-417e-91bd-0e74248bbbb4") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:07.331352 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:07.331324    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:16:07.404856 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:07.404815    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs\") pod \"network-metrics-daemon-zzxwf\" (UID: \"bbd14141-b0c9-4e6d-aab3-c1091adc5ddf\") " pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:07.405034 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:07.404891    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnfnb\" (UniqueName: \"kubernetes.io/projected/48d628be-136e-4521-a2fa-77f6c20b27d8-kube-api-access-rnfnb\") pod \"network-check-target-vnmfs\" (UID: \"48d628be-136e-4521-a2fa-77f6c20b27d8\") " pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:16:07.405132 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:07.405093    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:07.405132 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:07.405113    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:07.405132 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:07.405125    2566 projected.go:196] Error preparing data for projected volume kube-api-access-rnfnb for pod openshift-network-diagnostics/network-check-target-vnmfs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:07.405288 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:07.405182    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48d628be-136e-4521-a2fa-77f6c20b27d8-kube-api-access-rnfnb podName:48d628be-136e-4521-a2fa-77f6c20b27d8 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:09.405164556 +0000 UTC m=+7.860617612 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rnfnb" (UniqueName: "kubernetes.io/projected/48d628be-136e-4521-a2fa-77f6c20b27d8-kube-api-access-rnfnb") pod "network-check-target-vnmfs" (UID: "48d628be-136e-4521-a2fa-77f6c20b27d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:07.405579 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:07.405558    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:07.405662 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:07.405611    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs podName:bbd14141-b0c9-4e6d-aab3-c1091adc5ddf nodeName:}" failed. No retries permitted until 2026-09-18 16:16:09.405597377 +0000 UTC m=+7.861050437 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs") pod "network-metrics-daemon-zzxwf" (UID: "bbd14141-b0c9-4e6d-aab3-c1091adc5ddf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:07.617632 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:07.617590    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 16:11:05 +0000 UTC" deadline="2028-02-17 03:02:30.618840385 +0000 UTC"
Sep 18 16:16:07.617632 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:07.617630    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12394h46m23.001214938s"
Sep 18 16:16:07.684817 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:07.683915    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:07.684817 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:07.684045    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwgv5" podUID="94d26f9a-f3f1-417e-91bd-0e74248bbbb4"
Sep 18 16:16:07.684817 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:07.684483    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:07.684817 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:07.684597    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzxwf" podUID="bbd14141-b0c9-4e6d-aab3-c1091adc5ddf"
Sep 18 16:16:07.684817 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:07.684672    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:16:07.684817 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:07.684757    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnmfs" podUID="48d628be-136e-4521-a2fa-77f6c20b27d8"
Sep 18 16:16:07.698542 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:07.698467    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v26zp" event={"ID":"a53312cc-6d62-41f2-b3ef-c7b41da2219c","Type":"ContainerStarted","Data":"35ca65b6308d42ee8162d1f3db7785ee937a5f8e6076c10c8c0436f35a83514e"}
Sep 18 16:16:07.703307 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:07.703273    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7hjqm" event={"ID":"1a1fa079-6d52-4844-bded-44bf16212a8a","Type":"ContainerStarted","Data":"562410bde47dde665b8d00b5b991171694e04781eae0850475b9ae0031b2f2cb"}
Sep 18 16:16:07.708307 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:07.708264    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ptbtj" event={"ID":"1d77222a-f546-4b33-b6be-fbfa3bf2f6d0","Type":"ContainerStarted","Data":"1cf6d00b5218f9c5b40522424f7ed43549968f253d6eb6d2d90629b1183635bf"}
Sep 18 16:16:07.711783 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:07.711755    2566 generic.go:356] "Generic (PLEG): container finished" podID="46cbe6a567db993fd78904be901f7df3" containerID="061b1c3cf3b0e78b1ccb0cd40c776bc9129b591bb746d2ffe59d84fc5ceacaab" exitCode=0
Sep 18 16:16:07.711895 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:07.711830    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-142.ec2.internal" event={"ID":"46cbe6a567db993fd78904be901f7df3","Type":"ContainerDied","Data":"061b1c3cf3b0e78b1ccb0cd40c776bc9129b591bb746d2ffe59d84fc5ceacaab"}
Sep 18 16:16:07.720961 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:07.720932    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-szd7j" event={"ID":"9665ba03-1cdc-46c9-84f2-3ca2137b66e2","Type":"ContainerStarted","Data":"72de454162f2b5278faafda1f15266c97eb1ad249e83b97f60e657fbd8e5180f"}
Sep 18 16:16:07.725329 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:07.725302    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xt2cm" event={"ID":"044f1a79-aadb-474d-9525-6ccdb4bcbdf0","Type":"ContainerStarted","Data":"15b65b54a415b748b16e3ef56944daedc3d57ed1cce0e5d7b1d9956e64378e82"}
Sep 18 16:16:07.735324 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:07.735294    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jzsfr" event={"ID":"eb932b90-e784-4e7e-8f10-64421f61edaa","Type":"ContainerStarted","Data":"42a247b8ba7902d2702b719525e71e9fe59a1d578df669ab40fb0e31b27cc678"}
Sep 18 16:16:07.776159 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:07.776122    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw" event={"ID":"cacc3615-823b-4dc2-aa72-a9f0d2f4677b","Type":"ContainerStarted","Data":"8b3ab7c98d9da88bb6169278bc7037aeb8cd0952bb6a82e525dfe1a4008142c8"}
Sep 18 16:16:07.784244 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:07.784210    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zcmfd" event={"ID":"8434b53d-830b-4df4-8cf1-e8ed6cf92b9f","Type":"ContainerStarted","Data":"d4e8e455f080b1958b8049fbcbd690f23a50b7c9aec6bdcd349d8a15506c6504"}
Sep 18 16:16:07.790010 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:07.789978    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j92qj" event={"ID":"30c9a034-ba9f-4ead-8685-316723e6852a","Type":"ContainerStarted","Data":"aa7325d1ddad7ba01e56aceb899d3d4e7cde7d9631d95d8f0beb86de1ac6baa9"}
Sep 18 16:16:07.794451 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:07.794414    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-142.ec2.internal" event={"ID":"61ad4b4551fb28947d10dc4793af77f5","Type":"ContainerStarted","Data":"85f539128eb4ac604bf6568da68da1689f58162e1ba09c11576987b50562d92d"}
Sep 18 16:16:08.811848 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:08.811724    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-142.ec2.internal" event={"ID":"46cbe6a567db993fd78904be901f7df3","Type":"ContainerStarted","Data":"ba00d48126aaacea4fcc360bd91ee739cdbd56e9b4f32c0a6d44d94b567dfaa7"}
Sep 18 16:16:08.828653 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:08.828210    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-142.ec2.internal" podStartSLOduration=4.828186839 podStartE2EDuration="4.828186839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:07.808422359 +0000 UTC m=+6.263875436" watchObservedRunningTime="2026-09-18 16:16:08.828186839 +0000 UTC m=+7.283639919"
Sep 18 16:16:09.327278 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:09.327233    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-original-pull-secret\") pod \"global-pull-secret-syncer-gwgv5\" (UID: \"94d26f9a-f3f1-417e-91bd-0e74248bbbb4\") " pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:09.327470 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:09.327402    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:09.327470 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:09.327465    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-original-pull-secret podName:94d26f9a-f3f1-417e-91bd-0e74248bbbb4 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:13.327446773 +0000 UTC m=+11.782899833 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-original-pull-secret") pod "global-pull-secret-syncer-gwgv5" (UID: "94d26f9a-f3f1-417e-91bd-0e74248bbbb4") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:09.428068 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:09.427979    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs\") pod \"network-metrics-daemon-zzxwf\" (UID: \"bbd14141-b0c9-4e6d-aab3-c1091adc5ddf\") " pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:09.428068 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:09.428044    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnfnb\" (UniqueName: \"kubernetes.io/projected/48d628be-136e-4521-a2fa-77f6c20b27d8-kube-api-access-rnfnb\") pod \"network-check-target-vnmfs\" (UID: \"48d628be-136e-4521-a2fa-77f6c20b27d8\") " pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:16:09.428316 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:09.428236    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:09.428316 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:09.428258    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:09.428316 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:09.428270    2566 projected.go:196] Error preparing data for projected volume kube-api-access-rnfnb for pod openshift-network-diagnostics/network-check-target-vnmfs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:09.428477 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:09.428328    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48d628be-136e-4521-a2fa-77f6c20b27d8-kube-api-access-rnfnb podName:48d628be-136e-4521-a2fa-77f6c20b27d8 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:13.428310423 +0000 UTC m=+11.883763495 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rnfnb" (UniqueName: "kubernetes.io/projected/48d628be-136e-4521-a2fa-77f6c20b27d8-kube-api-access-rnfnb") pod "network-check-target-vnmfs" (UID: "48d628be-136e-4521-a2fa-77f6c20b27d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:09.428748 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:09.428731    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:09.428836 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:09.428780    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs podName:bbd14141-b0c9-4e6d-aab3-c1091adc5ddf nodeName:}" failed. No retries permitted until 2026-09-18 16:16:13.428765347 +0000 UTC m=+11.884218406 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs") pod "network-metrics-daemon-zzxwf" (UID: "bbd14141-b0c9-4e6d-aab3-c1091adc5ddf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:09.682067 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:09.681554    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:16:09.682067 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:09.681554    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:09.682067 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:09.681819    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzxwf" podUID="bbd14141-b0c9-4e6d-aab3-c1091adc5ddf"
Sep 18 16:16:09.682067 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:09.681691    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnmfs" podUID="48d628be-136e-4521-a2fa-77f6c20b27d8"
Sep 18 16:16:09.682067 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:09.681884    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:09.682067 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:09.681955    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwgv5" podUID="94d26f9a-f3f1-417e-91bd-0e74248bbbb4"
Sep 18 16:16:11.682664 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:11.682627    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:16:11.683119 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:11.682672    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:11.683119 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:11.682635    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:11.683119 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:11.682771    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnmfs" podUID="48d628be-136e-4521-a2fa-77f6c20b27d8"
Sep 18 16:16:11.683119 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:11.682896    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzxwf" podUID="bbd14141-b0c9-4e6d-aab3-c1091adc5ddf"
Sep 18 16:16:11.683119 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:11.682962    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwgv5" podUID="94d26f9a-f3f1-417e-91bd-0e74248bbbb4"
Sep 18 16:16:13.363557 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:13.363515    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-original-pull-secret\") pod \"global-pull-secret-syncer-gwgv5\" (UID: \"94d26f9a-f3f1-417e-91bd-0e74248bbbb4\") " pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:13.363972 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:13.363649    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:13.363972 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:13.363728    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-original-pull-secret podName:94d26f9a-f3f1-417e-91bd-0e74248bbbb4 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:21.363708273 +0000 UTC m=+19.819161332 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-original-pull-secret") pod "global-pull-secret-syncer-gwgv5" (UID: "94d26f9a-f3f1-417e-91bd-0e74248bbbb4") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:13.464112 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:13.463993    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs\") pod \"network-metrics-daemon-zzxwf\" (UID: \"bbd14141-b0c9-4e6d-aab3-c1091adc5ddf\") " pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:13.464112 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:13.464055    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnfnb\" (UniqueName: \"kubernetes.io/projected/48d628be-136e-4521-a2fa-77f6c20b27d8-kube-api-access-rnfnb\") pod \"network-check-target-vnmfs\" (UID: \"48d628be-136e-4521-a2fa-77f6c20b27d8\") " pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:16:13.464440 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:13.464141    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:13.464440 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:13.464199    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:13.464440 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:13.464212    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs podName:bbd14141-b0c9-4e6d-aab3-c1091adc5ddf nodeName:}" failed. No retries permitted until 2026-09-18 16:16:21.464194132 +0000 UTC m=+19.919647203 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs") pod "network-metrics-daemon-zzxwf" (UID: "bbd14141-b0c9-4e6d-aab3-c1091adc5ddf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:13.464440 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:13.464222    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:13.464440 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:13.464236    2566 projected.go:196] Error preparing data for projected volume kube-api-access-rnfnb for pod openshift-network-diagnostics/network-check-target-vnmfs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:13.464440 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:13.464289    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48d628be-136e-4521-a2fa-77f6c20b27d8-kube-api-access-rnfnb podName:48d628be-136e-4521-a2fa-77f6c20b27d8 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:21.464274217 +0000 UTC m=+19.919727292 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rnfnb" (UniqueName: "kubernetes.io/projected/48d628be-136e-4521-a2fa-77f6c20b27d8-kube-api-access-rnfnb") pod "network-check-target-vnmfs" (UID: "48d628be-136e-4521-a2fa-77f6c20b27d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:13.682379 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:13.682298    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:13.682535 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:13.682403    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:16:13.682535 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:13.682409    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzxwf" podUID="bbd14141-b0c9-4e6d-aab3-c1091adc5ddf"
Sep 18 16:16:13.682535 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:13.682481    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnmfs" podUID="48d628be-136e-4521-a2fa-77f6c20b27d8"
Sep 18 16:16:13.682535 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:13.682509    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:13.682711 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:13.682648    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwgv5" podUID="94d26f9a-f3f1-417e-91bd-0e74248bbbb4"
Sep 18 16:16:15.681530 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:15.681437    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:15.681530 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:15.681486    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:15.682055 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:15.681577    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzxwf" podUID="bbd14141-b0c9-4e6d-aab3-c1091adc5ddf"
Sep 18 16:16:15.682055 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:15.681876    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwgv5" podUID="94d26f9a-f3f1-417e-91bd-0e74248bbbb4"
Sep 18 16:16:15.682055 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:15.681437    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:16:15.682055 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:15.681968    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnmfs" podUID="48d628be-136e-4521-a2fa-77f6c20b27d8"
Sep 18 16:16:17.680817 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:17.680780    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:17.681265 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:17.680780    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:16:17.681265 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:17.680912    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzxwf" podUID="bbd14141-b0c9-4e6d-aab3-c1091adc5ddf"
Sep 18 16:16:17.681265 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:17.680988    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnmfs" podUID="48d628be-136e-4521-a2fa-77f6c20b27d8"
Sep 18 16:16:17.681265 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:17.680788    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:17.681265 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:17.681090    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwgv5" podUID="94d26f9a-f3f1-417e-91bd-0e74248bbbb4"
Sep 18 16:16:19.681107 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:19.681062    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:19.681554 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:19.681062    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:16:19.681554 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:19.681192    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzxwf" podUID="bbd14141-b0c9-4e6d-aab3-c1091adc5ddf"
Sep 18 16:16:19.681554 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:19.681276    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnmfs" podUID="48d628be-136e-4521-a2fa-77f6c20b27d8"
Sep 18 16:16:19.681554 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:19.681067    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:19.681554 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:19.681375    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwgv5" podUID="94d26f9a-f3f1-417e-91bd-0e74248bbbb4"
Sep 18 16:16:21.422910 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:21.422861    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-original-pull-secret\") pod \"global-pull-secret-syncer-gwgv5\" (UID: \"94d26f9a-f3f1-417e-91bd-0e74248bbbb4\") " pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:21.423333 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:21.422993    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:21.423333 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:21.423054    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-original-pull-secret podName:94d26f9a-f3f1-417e-91bd-0e74248bbbb4 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:37.423035634 +0000 UTC m=+35.878488709 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-original-pull-secret") pod "global-pull-secret-syncer-gwgv5" (UID: "94d26f9a-f3f1-417e-91bd-0e74248bbbb4") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:21.523385 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:21.523349    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs\") pod \"network-metrics-daemon-zzxwf\" (UID: \"bbd14141-b0c9-4e6d-aab3-c1091adc5ddf\") " pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:21.523385 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:21.523402    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnfnb\" (UniqueName: \"kubernetes.io/projected/48d628be-136e-4521-a2fa-77f6c20b27d8-kube-api-access-rnfnb\") pod \"network-check-target-vnmfs\" (UID: \"48d628be-136e-4521-a2fa-77f6c20b27d8\") " pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:16:21.523618 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:21.523529    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:21.523618 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:21.523562    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:21.523618 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:21.523576    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:21.523618 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:21.523585    2566 projected.go:196] Error preparing data for projected volume kube-api-access-rnfnb for pod openshift-network-diagnostics/network-check-target-vnmfs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:21.523618 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:21.523599    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs podName:bbd14141-b0c9-4e6d-aab3-c1091adc5ddf nodeName:}" failed. No retries permitted until 2026-09-18 16:16:37.523583307 +0000 UTC m=+35.979036366 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs") pod "network-metrics-daemon-zzxwf" (UID: "bbd14141-b0c9-4e6d-aab3-c1091adc5ddf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:21.523811 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:21.523647    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48d628be-136e-4521-a2fa-77f6c20b27d8-kube-api-access-rnfnb podName:48d628be-136e-4521-a2fa-77f6c20b27d8 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:37.523631153 +0000 UTC m=+35.979084209 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rnfnb" (UniqueName: "kubernetes.io/projected/48d628be-136e-4521-a2fa-77f6c20b27d8-kube-api-access-rnfnb") pod "network-check-target-vnmfs" (UID: "48d628be-136e-4521-a2fa-77f6c20b27d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:21.681533 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:21.681454    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:16:21.681688 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:21.681455    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:21.681688 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:21.681586    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnmfs" podUID="48d628be-136e-4521-a2fa-77f6c20b27d8"
Sep 18 16:16:21.681797 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:21.681689    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzxwf" podUID="bbd14141-b0c9-4e6d-aab3-c1091adc5ddf"
Sep 18 16:16:21.681797 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:21.681455    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:21.681797 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:21.681794    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwgv5" podUID="94d26f9a-f3f1-417e-91bd-0e74248bbbb4"
Sep 18 16:16:23.681383 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:23.681348    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:23.681810 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:23.681461    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzxwf" podUID="bbd14141-b0c9-4e6d-aab3-c1091adc5ddf"
Sep 18 16:16:23.681810 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:23.681518    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:16:23.681810 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:23.681593    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnmfs" podUID="48d628be-136e-4521-a2fa-77f6c20b27d8"
Sep 18 16:16:23.681810 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:23.681626    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:23.681810 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:23.681742    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwgv5" podUID="94d26f9a-f3f1-417e-91bd-0e74248bbbb4"
Sep 18 16:16:24.844925 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:24.844615    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v26zp" event={"ID":"a53312cc-6d62-41f2-b3ef-c7b41da2219c","Type":"ContainerStarted","Data":"73aad33ce6ecf1627ba07cf52882734c670669e6725959b229bf519ad5c76b6e"}
Sep 18 16:16:24.846482 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:24.846453    2566 generic.go:356] "Generic (PLEG): container finished" podID="1a1fa079-6d52-4844-bded-44bf16212a8a" containerID="d895d534e9e6e98afddaaf233bd3cc5c7786bfe5adbcf71103f29ab5ee03464d" exitCode=0
Sep 18 16:16:24.846590 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:24.846540    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7hjqm" event={"ID":"1a1fa079-6d52-4844-bded-44bf16212a8a","Type":"ContainerDied","Data":"d895d534e9e6e98afddaaf233bd3cc5c7786bfe5adbcf71103f29ab5ee03464d"}
Sep 18 16:16:24.848164 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:24.848146    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ptbtj" event={"ID":"1d77222a-f546-4b33-b6be-fbfa3bf2f6d0","Type":"ContainerStarted","Data":"eb118163d348b2f09952750b1d081855878c1bfbca8d2b42aabb86984434d471"}
Sep 18 16:16:24.849663 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:24.849638    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-szd7j" event={"ID":"9665ba03-1cdc-46c9-84f2-3ca2137b66e2","Type":"ContainerStarted","Data":"e9e03aed1224c519622c6a04baf3cbf03f04804fd67fc074377eb2a405e2506a"}
Sep 18 16:16:24.852256 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:24.852230    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jzsfr" event={"ID":"eb932b90-e784-4e7e-8f10-64421f61edaa","Type":"ContainerStarted","Data":"02a524b37a6986d818fda8a4d48162dcaef72e05ec816d063f14e078f11ee059"}
Sep 18 16:16:24.853953 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:24.853930    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw" event={"ID":"cacc3615-823b-4dc2-aa72-a9f0d2f4677b","Type":"ContainerStarted","Data":"27d2d9d1a96d91032c41171d69e1b4441c281822a02b1d9a3c3071461c6a37e2"}
Sep 18 16:16:24.855410 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:24.855387    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zcmfd" event={"ID":"8434b53d-830b-4df4-8cf1-e8ed6cf92b9f","Type":"ContainerStarted","Data":"7a46b2d9495f3e1a9d8be348a003b4f4c1c908e25841c9b107de99eabd277aee"}
Sep 18 16:16:24.858398 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:24.858373    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j92qj" event={"ID":"30c9a034-ba9f-4ead-8685-316723e6852a","Type":"ContainerStarted","Data":"2e31324049fa9d70152846e567f94a83ccfdf76828e24c95e62d16499c4c9e34"}
Sep 18 16:16:24.858492 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:24.858433    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j92qj" event={"ID":"30c9a034-ba9f-4ead-8685-316723e6852a","Type":"ContainerStarted","Data":"492b15b3036a147e79c9d77ef8cc52bc09bd755c1056710b9ea7afbdfa76a218"}
Sep 18 16:16:24.858492 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:24.858464    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j92qj" event={"ID":"30c9a034-ba9f-4ead-8685-316723e6852a","Type":"ContainerStarted","Data":"590b5f5d70e89fc37cc9c548a0da10509020266865b987926be4ed945053337b"}
Sep 18 16:16:24.858492 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:24.858491    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j92qj" event={"ID":"30c9a034-ba9f-4ead-8685-316723e6852a","Type":"ContainerStarted","Data":"ef9596014aa5d971d216cfc409d90aa3ece9a32dc1950182107f1aab7b46a353"}
Sep 18 16:16:24.858651 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:24.858517    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j92qj" event={"ID":"30c9a034-ba9f-4ead-8685-316723e6852a","Type":"ContainerStarted","Data":"4d8049f9048b673ade1fe50379530c93ffde0e6d136c0cd5f8c89c3205f015df"}
Sep 18 16:16:24.858651 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:24.858546    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j92qj" event={"ID":"30c9a034-ba9f-4ead-8685-316723e6852a","Type":"ContainerStarted","Data":"ad138e04eabc0865062be05a608430689af3398a073838b598a6ac4a58e4b676"}
Sep 18 16:16:24.863954 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:24.863909    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-v26zp" podStartSLOduration=4.58858667 podStartE2EDuration="21.863894012s" totalImagesPullingTime="17.275307342s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:24.862672317 +0000 UTC m=+23.318125395" watchObservedRunningTime="2026-09-18 16:16:24.863894012 +0000 UTC m=+23.319347090"
Sep 18 16:16:24.864286 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:24.864248    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-142.ec2.internal" podStartSLOduration=20.864236693 podStartE2EDuration="20.864236693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:08.828638327 +0000 UTC m=+7.284091405" watchObservedRunningTime="2026-09-18 16:16:24.864236693 +0000 UTC m=+23.319689774"
Sep 18 16:16:24.877882 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:24.877819    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-szd7j" podStartSLOduration=4.633725813 podStartE2EDuration="21.87780081s" totalImagesPullingTime="17.244074997s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:24.877710497 +0000 UTC m=+23.333163573" watchObservedRunningTime="2026-09-18 16:16:24.87780081 +0000 UTC m=+23.333253887"
Sep 18 16:16:24.919370 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:24.919316    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zcmfd" podStartSLOduration=8.874194863 podStartE2EDuration="21.919299648s" totalImagesPullingTime="13.045104785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:24.91905089 +0000 UTC m=+23.374503968" watchObservedRunningTime="2026-09-18 16:16:24.919299648 +0000 UTC m=+23.374752722"
Sep 18 16:16:24.934318 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:24.934261    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ptbtj" podStartSLOduration=4.651745303 podStartE2EDuration="21.934243439s" totalImagesPullingTime="17.282498136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:24.933828475 +0000 UTC m=+23.389281553" watchObservedRunningTime="2026-09-18 16:16:24.934243439 +0000 UTC m=+23.389696517"
Sep 18 16:16:24.982473 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:24.982440    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zcmfd_8434b53d-830b-4df4-8cf1-e8ed6cf92b9f/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:25.058449 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:25.058249    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zcmfd_8434b53d-830b-4df4-8cf1-e8ed6cf92b9f/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:25.136048 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:25.136016    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 16:16:25.158251 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:25.158221    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zcmfd_8434b53d-830b-4df4-8cf1-e8ed6cf92b9f/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:25.261289 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:25.261245    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zcmfd_8434b53d-830b-4df4-8cf1-e8ed6cf92b9f/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:25.665341 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:25.665224    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T16:16:25.136034018Z","UUID":"e2ddb644-f6fc-4f47-87fa-302a34648567","Handler":null,"Name":"","Endpoint":""}
Sep 18 16:16:25.666854 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:25.666817    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 16:16:25.666854 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:25.666858    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 16:16:25.681302 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:25.681274    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:16:25.681477 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:25.681274    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:25.681477 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:25.681405    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnmfs" podUID="48d628be-136e-4521-a2fa-77f6c20b27d8"
Sep 18 16:16:25.681477 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:25.681423    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:25.681633 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:25.681520    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzxwf" podUID="bbd14141-b0c9-4e6d-aab3-c1091adc5ddf"
Sep 18 16:16:25.681633 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:25.681598    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwgv5" podUID="94d26f9a-f3f1-417e-91bd-0e74248bbbb4"
Sep 18 16:16:25.863397 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:25.863360    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xt2cm" event={"ID":"044f1a79-aadb-474d-9525-6ccdb4bcbdf0","Type":"ContainerStarted","Data":"c59415ecb9262d1877275b6e6791d744dcff63829fb759fba1fce6ca048a8e93"}
Sep 18 16:16:25.866438 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:25.866406    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw" event={"ID":"cacc3615-823b-4dc2-aa72-a9f0d2f4677b","Type":"ContainerStarted","Data":"2a9e0076223295491b1679f81849ba58f42e20aa980307ae0fd4312090107a75"}
Sep 18 16:16:25.880246 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:25.880193    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jzsfr" podStartSLOduration=5.557027779 podStartE2EDuration="22.880176767s" totalImagesPullingTime="17.323148988s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:24.953025793 +0000 UTC m=+23.408478871" watchObservedRunningTime="2026-09-18 16:16:25.880176767 +0000 UTC m=+24.335629844"
Sep 18 16:16:25.880415 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:25.880380    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xt2cm" podStartSLOduration=5.61007634 podStartE2EDuration="22.880374682s" totalImagesPullingTime="17.270298342s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:25.880115533 +0000 UTC m=+24.335568630" watchObservedRunningTime="2026-09-18 16:16:25.880374682 +0000 UTC m=+24.335827760"
Sep 18 16:16:26.040500 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:26.040465    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zcmfd_8434b53d-830b-4df4-8cf1-e8ed6cf92b9f/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:26.201507 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:26.201418    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zcmfd_8434b53d-830b-4df4-8cf1-e8ed6cf92b9f/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:26.871185 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:26.870913    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw" event={"ID":"cacc3615-823b-4dc2-aa72-a9f0d2f4677b","Type":"ContainerStarted","Data":"06633aadda2f9b0504971f6027d66037f7966826fbb995f2ac83b1bd259a6106"}
Sep 18 16:16:26.875476 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:26.875446    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j92qj" event={"ID":"30c9a034-ba9f-4ead-8685-316723e6852a","Type":"ContainerStarted","Data":"08ee38f6c0fcfcba8c9282f353e48faa86edd4cf4ff8aa001fcce2364786c724"}
Sep 18 16:16:26.895961 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:26.895904    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4wcw" podStartSLOduration=5.12240714 podStartE2EDuration="23.895888791s" totalImagesPullingTime="18.773481651s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:26.895410994 +0000 UTC m=+25.350864073" watchObservedRunningTime="2026-09-18 16:16:26.895888791 +0000 UTC m=+25.351341865"
Sep 18 16:16:27.681322 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:27.681280    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:27.681495 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:27.681414    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:16:27.681495 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:27.681419    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzxwf" podUID="bbd14141-b0c9-4e6d-aab3-c1091adc5ddf"
Sep 18 16:16:27.681596 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:27.681516    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnmfs" podUID="48d628be-136e-4521-a2fa-77f6c20b27d8"
Sep 18 16:16:27.681596 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:27.681564    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:27.681691 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:27.681636    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwgv5" podUID="94d26f9a-f3f1-417e-91bd-0e74248bbbb4"
Sep 18 16:16:29.681893 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:29.681674    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:16:29.682434 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:29.681674    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:29.682434 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:29.681925    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnmfs" podUID="48d628be-136e-4521-a2fa-77f6c20b27d8"
Sep 18 16:16:29.682434 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:29.681728    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:29.682434 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:29.682011    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzxwf" podUID="bbd14141-b0c9-4e6d-aab3-c1091adc5ddf"
Sep 18 16:16:29.682434 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:29.682068    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwgv5" podUID="94d26f9a-f3f1-417e-91bd-0e74248bbbb4"
Sep 18 16:16:29.714226 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:29.714194    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zcmfd"
Sep 18 16:16:29.714758 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:29.714741    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zcmfd"
Sep 18 16:16:29.885260 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:29.885232    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j92qj" event={"ID":"30c9a034-ba9f-4ead-8685-316723e6852a","Type":"ContainerStarted","Data":"5a8ad1ba4397e98959b08191c002490ef74ee0f0790e904cd5e803f6983cf854"}
Sep 18 16:16:29.885536 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:29.885517    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:29.887233 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:29.887205    2566 generic.go:356] "Generic (PLEG): container finished" podID="1a1fa079-6d52-4844-bded-44bf16212a8a" containerID="7a83b84edbb6d699fba2ff394bd4889c5b86a124ab3bdb31ec00ffdc66af6254" exitCode=0
Sep 18 16:16:29.887331 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:29.887307    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7hjqm" event={"ID":"1a1fa079-6d52-4844-bded-44bf16212a8a","Type":"ContainerDied","Data":"7a83b84edbb6d699fba2ff394bd4889c5b86a124ab3bdb31ec00ffdc66af6254"}
Sep 18 16:16:29.887494 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:29.887472    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zcmfd"
Sep 18 16:16:29.888467 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:29.887900    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zcmfd"
Sep 18 16:16:29.903258 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:29.903231    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:29.915804 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:29.915758    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j92qj" podStartSLOduration=9.617865733 podStartE2EDuration="26.91574307s" totalImagesPullingTime="17.297877337s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:29.914098083 +0000 UTC m=+28.369551160" watchObservedRunningTime="2026-09-18 16:16:29.91574307 +0000 UTC m=+28.371196147"
Sep 18 16:16:30.004581 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:30.004495    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zcmfd_8434b53d-830b-4df4-8cf1-e8ed6cf92b9f/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:30.385709 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:30.385664    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zcmfd_8434b53d-830b-4df4-8cf1-e8ed6cf92b9f/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:30.474613 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:30.474577    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zcmfd_8434b53d-830b-4df4-8cf1-e8ed6cf92b9f/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:30.560398 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:30.560352    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zcmfd_8434b53d-830b-4df4-8cf1-e8ed6cf92b9f/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:30.664582 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:30.664518    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zcmfd_8434b53d-830b-4df4-8cf1-e8ed6cf92b9f/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:30.761609 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:30.761571    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zcmfd_8434b53d-830b-4df4-8cf1-e8ed6cf92b9f/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:30.894127 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:30.894024    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7hjqm" event={"ID":"1a1fa079-6d52-4844-bded-44bf16212a8a","Type":"ContainerStarted","Data":"90ac6e2dcc38209507703815d77764dfa9011026f0189aef79a92b0b0544919b"}
Sep 18 16:16:30.894282 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:30.894265    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:30.895100 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:30.894793    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:30.910127 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:30.910101    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:16:31.083395 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:31.083365    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vnmfs"]
Sep 18 16:16:31.083560 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:31.083484    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:16:31.083605 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:31.083571    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnmfs" podUID="48d628be-136e-4521-a2fa-77f6c20b27d8"
Sep 18 16:16:31.086817 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:31.086791    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gwgv5"]
Sep 18 16:16:31.086959 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:31.086903    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:31.087063 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:31.087041    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwgv5" podUID="94d26f9a-f3f1-417e-91bd-0e74248bbbb4"
Sep 18 16:16:31.087531 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:31.087507    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zzxwf"]
Sep 18 16:16:31.087631 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:31.087613    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:31.087751 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:31.087716    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzxwf" podUID="bbd14141-b0c9-4e6d-aab3-c1091adc5ddf"
Sep 18 16:16:31.354257 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:31.354170    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zcmfd_8434b53d-830b-4df4-8cf1-e8ed6cf92b9f/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:31.414196 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:31.414162    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zcmfd_8434b53d-830b-4df4-8cf1-e8ed6cf92b9f/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:31.738362 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:31.738239    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zcmfd_8434b53d-830b-4df4-8cf1-e8ed6cf92b9f/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:31.843400 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:31.843364    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zcmfd_8434b53d-830b-4df4-8cf1-e8ed6cf92b9f/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:31.898262 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:31.898229    2566 generic.go:356] "Generic (PLEG): container finished" podID="1a1fa079-6d52-4844-bded-44bf16212a8a" containerID="90ac6e2dcc38209507703815d77764dfa9011026f0189aef79a92b0b0544919b" exitCode=0
Sep 18 16:16:31.898262 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:31.898258    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7hjqm" event={"ID":"1a1fa079-6d52-4844-bded-44bf16212a8a","Type":"ContainerDied","Data":"90ac6e2dcc38209507703815d77764dfa9011026f0189aef79a92b0b0544919b"}
Sep 18 16:16:32.681265 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:32.681233    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:32.681449 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:32.681285    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:16:32.681449 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:32.681381    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzxwf" podUID="bbd14141-b0c9-4e6d-aab3-c1091adc5ddf"
Sep 18 16:16:32.681449 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:32.681392    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:32.681449 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:32.681441    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnmfs" podUID="48d628be-136e-4521-a2fa-77f6c20b27d8"
Sep 18 16:16:32.681626 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:32.681527    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwgv5" podUID="94d26f9a-f3f1-417e-91bd-0e74248bbbb4"
Sep 18 16:16:32.752612 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:32.752577    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zcmfd_8434b53d-830b-4df4-8cf1-e8ed6cf92b9f/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:32.853800 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:32.853773    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zcmfd_8434b53d-830b-4df4-8cf1-e8ed6cf92b9f/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:33.906055 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:33.906021    2566 generic.go:356] "Generic (PLEG): container finished" podID="1a1fa079-6d52-4844-bded-44bf16212a8a" containerID="fdc6177ab5ee5bb0ec4df9b968e5630e4bb07917a873cccb0bfad77221707691" exitCode=0
Sep 18 16:16:33.906432 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:33.906069    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7hjqm" event={"ID":"1a1fa079-6d52-4844-bded-44bf16212a8a","Type":"ContainerDied","Data":"fdc6177ab5ee5bb0ec4df9b968e5630e4bb07917a873cccb0bfad77221707691"}
Sep 18 16:16:34.681232 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:34.681202    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:16:34.681381 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:34.681202    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:34.681381 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:34.681325    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnmfs" podUID="48d628be-136e-4521-a2fa-77f6c20b27d8"
Sep 18 16:16:34.681484 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:34.681417    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwgv5" podUID="94d26f9a-f3f1-417e-91bd-0e74248bbbb4"
Sep 18 16:16:34.681484 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:34.681202    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:34.681595 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:34.681528    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzxwf" podUID="bbd14141-b0c9-4e6d-aab3-c1091adc5ddf"
Sep 18 16:16:36.681342 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:36.681305    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:36.681786 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:36.681355    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:16:36.681786 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:36.681320    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:36.681786 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:36.681443    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwgv5" podUID="94d26f9a-f3f1-417e-91bd-0e74248bbbb4"
Sep 18 16:16:36.681786 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:36.681544    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzxwf" podUID="bbd14141-b0c9-4e6d-aab3-c1091adc5ddf"
Sep 18 16:16:36.681786 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:36.681620    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vnmfs" podUID="48d628be-136e-4521-a2fa-77f6c20b27d8"
Sep 18 16:16:36.827520 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:36.827486    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-142.ec2.internal" event="NodeReady"
Sep 18 16:16:36.827687 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:36.827645    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 16:16:36.876824 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:36.876796    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-787hc"]
Sep 18 16:16:36.895473 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:36.895432    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2wd2m"]
Sep 18 16:16:36.895983 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:36.895564    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-787hc"
Sep 18 16:16:36.898801 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:36.898650    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 16:16:36.898801 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:36.898671    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 16:16:36.898801 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:36.898656    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ccb8q\""
Sep 18 16:16:36.915556 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:36.915534    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2wd2m"]
Sep 18 16:16:36.915556 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:36.915562    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-787hc"]
Sep 18 16:16:36.915720 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:36.915663    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2wd2m"
Sep 18 16:16:36.918878 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:36.918852    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 16:16:36.919015 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:36.918930    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qbbmj\""
Sep 18 16:16:36.919101 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:36.918860    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 16:16:36.919216 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:36.919198    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 16:16:37.030750 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:37.030713    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zcmfd_8434b53d-830b-4df4-8cf1-e8ed6cf92b9f/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:37.041742 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:37.041708    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/83e8eb33-e386-4b60-98c6-a95b93c408cd-config-volume\") pod \"dns-default-787hc\" (UID: \"83e8eb33-e386-4b60-98c6-a95b93c408cd\") " pod="openshift-dns/dns-default-787hc"
Sep 18 16:16:37.041912 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:37.041752    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/711861f8-db0f-4e93-a894-0cedf9efdf89-cert\") pod \"ingress-canary-2wd2m\" (UID: \"711861f8-db0f-4e93-a894-0cedf9efdf89\") " pod="openshift-ingress-canary/ingress-canary-2wd2m"
Sep 18 16:16:37.041912 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:37.041832    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcw7v\" (UniqueName: \"kubernetes.io/projected/711861f8-db0f-4e93-a894-0cedf9efdf89-kube-api-access-fcw7v\") pod \"ingress-canary-2wd2m\" (UID: \"711861f8-db0f-4e93-a894-0cedf9efdf89\") " pod="openshift-ingress-canary/ingress-canary-2wd2m"
Sep 18 16:16:37.041912 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:37.041881    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls\") pod \"dns-default-787hc\" (UID: \"83e8eb33-e386-4b60-98c6-a95b93c408cd\") " pod="openshift-dns/dns-default-787hc"
Sep 18 16:16:37.041912 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:37.041904    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83e8eb33-e386-4b60-98c6-a95b93c408cd-tmp-dir\") pod \"dns-default-787hc\" (UID: \"83e8eb33-e386-4b60-98c6-a95b93c408cd\") " pod="openshift-dns/dns-default-787hc"
Sep 18 16:16:37.042113 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:37.041936    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdcpc\" (UniqueName: \"kubernetes.io/projected/83e8eb33-e386-4b60-98c6-a95b93c408cd-kube-api-access-bdcpc\") pod \"dns-default-787hc\" (UID: \"83e8eb33-e386-4b60-98c6-a95b93c408cd\") " pod="openshift-dns/dns-default-787hc"
Sep 18 16:16:37.114542 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:37.114504    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zcmfd_8434b53d-830b-4df4-8cf1-e8ed6cf92b9f/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:37.143098 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:37.143047    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcw7v\" (UniqueName: \"kubernetes.io/projected/711861f8-db0f-4e93-a894-0cedf9efdf89-kube-api-access-fcw7v\") pod \"ingress-canary-2wd2m\" (UID: \"711861f8-db0f-4e93-a894-0cedf9efdf89\") " pod="openshift-ingress-canary/ingress-canary-2wd2m"
Sep 18 16:16:37.143293 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:37.143134    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls\") pod \"dns-default-787hc\" (UID: \"83e8eb33-e386-4b60-98c6-a95b93c408cd\") " pod="openshift-dns/dns-default-787hc"
Sep 18 16:16:37.143293 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:37.143160    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83e8eb33-e386-4b60-98c6-a95b93c408cd-tmp-dir\") pod \"dns-default-787hc\" (UID: \"83e8eb33-e386-4b60-98c6-a95b93c408cd\") " pod="openshift-dns/dns-default-787hc"
Sep 18 16:16:37.143293 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:37.143190    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdcpc\" (UniqueName: \"kubernetes.io/projected/83e8eb33-e386-4b60-98c6-a95b93c408cd-kube-api-access-bdcpc\") pod \"dns-default-787hc\" (UID: \"83e8eb33-e386-4b60-98c6-a95b93c408cd\") " pod="openshift-dns/dns-default-787hc"
Sep 18 16:16:37.143293 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:37.143227    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/83e8eb33-e386-4b60-98c6-a95b93c408cd-config-volume\") pod \"dns-default-787hc\" (UID: \"83e8eb33-e386-4b60-98c6-a95b93c408cd\") " pod="openshift-dns/dns-default-787hc"
Sep 18 16:16:37.143293 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:37.143248    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/711861f8-db0f-4e93-a894-0cedf9efdf89-cert\") pod \"ingress-canary-2wd2m\" (UID: \"711861f8-db0f-4e93-a894-0cedf9efdf89\") " pod="openshift-ingress-canary/ingress-canary-2wd2m"
Sep 18 16:16:37.143293 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:37.143263    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:16:37.143596 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:37.143346    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls podName:83e8eb33-e386-4b60-98c6-a95b93c408cd nodeName:}" failed. No retries permitted until 2026-09-18 16:16:37.643329552 +0000 UTC m=+36.098782607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls") pod "dns-default-787hc" (UID: "83e8eb33-e386-4b60-98c6-a95b93c408cd") : secret "dns-default-metrics-tls" not found
Sep 18 16:16:37.143713 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:37.143687    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83e8eb33-e386-4b60-98c6-a95b93c408cd-tmp-dir\") pod \"dns-default-787hc\" (UID: \"83e8eb33-e386-4b60-98c6-a95b93c408cd\") " pod="openshift-dns/dns-default-787hc"
Sep 18 16:16:37.143903 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:37.143879    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/83e8eb33-e386-4b60-98c6-a95b93c408cd-config-volume\") pod \"dns-default-787hc\" (UID: \"83e8eb33-e386-4b60-98c6-a95b93c408cd\") " pod="openshift-dns/dns-default-787hc"
Sep 18 16:16:37.148657 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:37.148634    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/711861f8-db0f-4e93-a894-0cedf9efdf89-cert\") pod \"ingress-canary-2wd2m\" (UID: \"711861f8-db0f-4e93-a894-0cedf9efdf89\") " pod="openshift-ingress-canary/ingress-canary-2wd2m"
Sep 18 16:16:37.152017 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:37.151990    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdcpc\" (UniqueName: \"kubernetes.io/projected/83e8eb33-e386-4b60-98c6-a95b93c408cd-kube-api-access-bdcpc\") pod \"dns-default-787hc\" (UID: \"83e8eb33-e386-4b60-98c6-a95b93c408cd\") " pod="openshift-dns/dns-default-787hc"
Sep 18 16:16:37.152252 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:37.152207    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcw7v\" (UniqueName: \"kubernetes.io/projected/711861f8-db0f-4e93-a894-0cedf9efdf89-kube-api-access-fcw7v\") pod \"ingress-canary-2wd2m\" (UID: \"711861f8-db0f-4e93-a894-0cedf9efdf89\") " pod="openshift-ingress-canary/ingress-canary-2wd2m"
Sep 18 16:16:37.229254 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:37.229176    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2wd2m"
Sep 18 16:16:37.445630 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:37.445583    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-original-pull-secret\") pod \"global-pull-secret-syncer-gwgv5\" (UID: \"94d26f9a-f3f1-417e-91bd-0e74248bbbb4\") " pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:37.445811 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:37.445760    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:37.445857 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:37.445823    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-original-pull-secret podName:94d26f9a-f3f1-417e-91bd-0e74248bbbb4 nodeName:}" failed. No retries permitted until 2026-09-18 16:17:09.445806551 +0000 UTC m=+67.901259612 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-original-pull-secret") pod "global-pull-secret-syncer-gwgv5" (UID: "94d26f9a-f3f1-417e-91bd-0e74248bbbb4") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:37.546407 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:37.546310    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs\") pod \"network-metrics-daemon-zzxwf\" (UID: \"bbd14141-b0c9-4e6d-aab3-c1091adc5ddf\") " pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:37.546407 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:37.546385    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnfnb\" (UniqueName: \"kubernetes.io/projected/48d628be-136e-4521-a2fa-77f6c20b27d8-kube-api-access-rnfnb\") pod \"network-check-target-vnmfs\" (UID: \"48d628be-136e-4521-a2fa-77f6c20b27d8\") " pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:16:37.546634 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:37.546474    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:37.546634 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:37.546527    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:37.546634 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:37.546544    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:37.546634 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:37.546552    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs podName:bbd14141-b0c9-4e6d-aab3-c1091adc5ddf nodeName:}" failed. No retries permitted until 2026-09-18 16:17:09.54653295 +0000 UTC m=+68.001986006 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs") pod "network-metrics-daemon-zzxwf" (UID: "bbd14141-b0c9-4e6d-aab3-c1091adc5ddf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:37.546634 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:37.546557    2566 projected.go:196] Error preparing data for projected volume kube-api-access-rnfnb for pod openshift-network-diagnostics/network-check-target-vnmfs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:37.546634 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:37.546602    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48d628be-136e-4521-a2fa-77f6c20b27d8-kube-api-access-rnfnb podName:48d628be-136e-4521-a2fa-77f6c20b27d8 nodeName:}" failed. No retries permitted until 2026-09-18 16:17:09.546592177 +0000 UTC m=+68.002045237 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rnfnb" (UniqueName: "kubernetes.io/projected/48d628be-136e-4521-a2fa-77f6c20b27d8-kube-api-access-rnfnb") pod "network-check-target-vnmfs" (UID: "48d628be-136e-4521-a2fa-77f6c20b27d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:37.646819 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:37.646785    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls\") pod \"dns-default-787hc\" (UID: \"83e8eb33-e386-4b60-98c6-a95b93c408cd\") " pod="openshift-dns/dns-default-787hc"
Sep 18 16:16:37.646987 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:37.646935    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:16:37.647043 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:37.647010    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls podName:83e8eb33-e386-4b60-98c6-a95b93c408cd nodeName:}" failed. No retries permitted until 2026-09-18 16:16:38.646992173 +0000 UTC m=+37.102445248 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls") pod "dns-default-787hc" (UID: "83e8eb33-e386-4b60-98c6-a95b93c408cd") : secret "dns-default-metrics-tls" not found
Sep 18 16:16:38.627807 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:38.627571    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2wd2m"]
Sep 18 16:16:38.636342 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:16:38.636308    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711861f8_db0f_4e93_a894_0cedf9efdf89.slice/crio-36e4d78fe2c9cf58bebdc0920a652c895d7470275001643d5eb9283956700ee6 WatchSource:0}: Error finding container 36e4d78fe2c9cf58bebdc0920a652c895d7470275001643d5eb9283956700ee6: Status 404 returned error can't find the container with id 36e4d78fe2c9cf58bebdc0920a652c895d7470275001643d5eb9283956700ee6
Sep 18 16:16:38.653783 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:38.653751    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls\") pod \"dns-default-787hc\" (UID: \"83e8eb33-e386-4b60-98c6-a95b93c408cd\") " pod="openshift-dns/dns-default-787hc"
Sep 18 16:16:38.653906 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:38.653875    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:16:38.653969 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:38.653946    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls podName:83e8eb33-e386-4b60-98c6-a95b93c408cd nodeName:}" failed. No retries permitted until 2026-09-18 16:16:40.653927219 +0000 UTC m=+39.109380290 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls") pod "dns-default-787hc" (UID: "83e8eb33-e386-4b60-98c6-a95b93c408cd") : secret "dns-default-metrics-tls" not found
Sep 18 16:16:38.680911 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:38.680804    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:16:38.680911 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:38.680824    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:16:38.680911 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:38.680826    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:16:38.683688 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:38.683667    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 16:16:38.683779 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:38.683701    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 16:16:38.683779 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:38.683707    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ctnzv\""
Sep 18 16:16:38.684584 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:38.684564    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 16:16:38.684677 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:38.684604    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 16:16:38.684677 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:38.684566    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-75jt8\""
Sep 18 16:16:38.921373 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:38.921340    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7hjqm" event={"ID":"1a1fa079-6d52-4844-bded-44bf16212a8a","Type":"ContainerStarted","Data":"20b6d2463de2dba8eff1c653b2ae77a376aac4d5fb4644521c54338dd83d2aa9"}
Sep 18 16:16:38.922749 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:38.922715    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2wd2m" event={"ID":"711861f8-db0f-4e93-a894-0cedf9efdf89","Type":"ContainerStarted","Data":"36e4d78fe2c9cf58bebdc0920a652c895d7470275001643d5eb9283956700ee6"}
Sep 18 16:16:39.928270 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:39.928139    2566 generic.go:356] "Generic (PLEG): container finished" podID="1a1fa079-6d52-4844-bded-44bf16212a8a" containerID="20b6d2463de2dba8eff1c653b2ae77a376aac4d5fb4644521c54338dd83d2aa9" exitCode=0
Sep 18 16:16:39.928270 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:39.928206    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7hjqm" event={"ID":"1a1fa079-6d52-4844-bded-44bf16212a8a","Type":"ContainerDied","Data":"20b6d2463de2dba8eff1c653b2ae77a376aac4d5fb4644521c54338dd83d2aa9"}
Sep 18 16:16:40.669879 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:40.669836    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls\") pod \"dns-default-787hc\" (UID: \"83e8eb33-e386-4b60-98c6-a95b93c408cd\") " pod="openshift-dns/dns-default-787hc"
Sep 18 16:16:40.670067 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:40.669994    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:16:40.670154 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:40.670068    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls podName:83e8eb33-e386-4b60-98c6-a95b93c408cd nodeName:}" failed. No retries permitted until 2026-09-18 16:16:44.670045793 +0000 UTC m=+43.125498848 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls") pod "dns-default-787hc" (UID: "83e8eb33-e386-4b60-98c6-a95b93c408cd") : secret "dns-default-metrics-tls" not found
Sep 18 16:16:40.933660 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:40.933579    2566 generic.go:356] "Generic (PLEG): container finished" podID="1a1fa079-6d52-4844-bded-44bf16212a8a" containerID="746a58302cef0e22c0f1f4ad5f6038aec13acb9e5bd7b97e4997ca22ec8f7d6f" exitCode=0
Sep 18 16:16:40.934063 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:40.933666    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7hjqm" event={"ID":"1a1fa079-6d52-4844-bded-44bf16212a8a","Type":"ContainerDied","Data":"746a58302cef0e22c0f1f4ad5f6038aec13acb9e5bd7b97e4997ca22ec8f7d6f"}
Sep 18 16:16:41.942490 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:41.942294    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7hjqm" event={"ID":"1a1fa079-6d52-4844-bded-44bf16212a8a","Type":"ContainerStarted","Data":"60098adb71331a64847b4773612c52d3ddd4b51d3c759cd13cdfca1a26350aa0"}
Sep 18 16:16:41.943868 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:41.943843    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2wd2m" event={"ID":"711861f8-db0f-4e93-a894-0cedf9efdf89","Type":"ContainerStarted","Data":"ce53fe69a1376e113abe1976731d68ba9e8fc8609c0f71afa030f55c576aa708"}
Sep 18 16:16:41.963087 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:41.963034    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7hjqm" podStartSLOduration=10.610701683 podStartE2EDuration="38.96301911s" totalImagesPullingTime="28.352317427s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:41.961571182 +0000 UTC m=+40.417024259" watchObservedRunningTime="2026-09-18 16:16:41.96301911 +0000 UTC m=+40.418472237"
Sep 18 16:16:41.975839 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:41.975796    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2wd2m" podStartSLOduration=3.4165781969999998 podStartE2EDuration="5.975784352s" totalImagesPullingTime="2.559206155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:41.975591392 +0000 UTC m=+40.431044470" watchObservedRunningTime="2026-09-18 16:16:41.975784352 +0000 UTC m=+40.431237428"
Sep 18 16:16:42.090317 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:42.090284    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zcmfd_8434b53d-830b-4df4-8cf1-e8ed6cf92b9f/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:42.142893 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:42.142847    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zcmfd_8434b53d-830b-4df4-8cf1-e8ed6cf92b9f/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:42.597420 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:42.597382    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zcmfd_8434b53d-830b-4df4-8cf1-e8ed6cf92b9f/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:42.668966 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:42.668931    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zcmfd_8434b53d-830b-4df4-8cf1-e8ed6cf92b9f/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:43.359885 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:43.359849    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zcmfd_8434b53d-830b-4df4-8cf1-e8ed6cf92b9f/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:43.418827 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:43.418798    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zcmfd_8434b53d-830b-4df4-8cf1-e8ed6cf92b9f/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:44.699063 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:44.699014    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls\") pod \"dns-default-787hc\" (UID: \"83e8eb33-e386-4b60-98c6-a95b93c408cd\") " pod="openshift-dns/dns-default-787hc"
Sep 18 16:16:44.699493 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:44.699169    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:16:44.699493 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:44.699232    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls podName:83e8eb33-e386-4b60-98c6-a95b93c408cd nodeName:}" failed. No retries permitted until 2026-09-18 16:16:52.699215724 +0000 UTC m=+51.154668780 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls") pod "dns-default-787hc" (UID: "83e8eb33-e386-4b60-98c6-a95b93c408cd") : secret "dns-default-metrics-tls" not found
Sep 18 16:16:52.742149 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:16:52.742102    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls\") pod \"dns-default-787hc\" (UID: \"83e8eb33-e386-4b60-98c6-a95b93c408cd\") " pod="openshift-dns/dns-default-787hc"
Sep 18 16:16:52.742546 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:52.742215    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:16:52.742546 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:16:52.742278    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls podName:83e8eb33-e386-4b60-98c6-a95b93c408cd nodeName:}" failed. No retries permitted until 2026-09-18 16:17:08.742264639 +0000 UTC m=+67.197717694 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls") pod "dns-default-787hc" (UID: "83e8eb33-e386-4b60-98c6-a95b93c408cd") : secret "dns-default-metrics-tls" not found
Sep 18 16:17:02.919904 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:17:02.919875    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j92qj"
Sep 18 16:17:08.750353 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:17:08.750311    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls\") pod \"dns-default-787hc\" (UID: \"83e8eb33-e386-4b60-98c6-a95b93c408cd\") " pod="openshift-dns/dns-default-787hc"
Sep 18 16:17:08.750946 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:17:08.750485    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:17:08.750946 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:17:08.750571    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls podName:83e8eb33-e386-4b60-98c6-a95b93c408cd nodeName:}" failed. No retries permitted until 2026-09-18 16:17:40.750550444 +0000 UTC m=+99.206003499 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls") pod "dns-default-787hc" (UID: "83e8eb33-e386-4b60-98c6-a95b93c408cd") : secret "dns-default-metrics-tls" not found
Sep 18 16:17:09.455006 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:17:09.454963    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-original-pull-secret\") pod \"global-pull-secret-syncer-gwgv5\" (UID: \"94d26f9a-f3f1-417e-91bd-0e74248bbbb4\") " pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:17:09.458111 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:17:09.458091    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 16:17:09.468883 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:17:09.468852    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94d26f9a-f3f1-417e-91bd-0e74248bbbb4-original-pull-secret\") pod \"global-pull-secret-syncer-gwgv5\" (UID: \"94d26f9a-f3f1-417e-91bd-0e74248bbbb4\") " pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:17:09.555563 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:17:09.555521    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnfnb\" (UniqueName: \"kubernetes.io/projected/48d628be-136e-4521-a2fa-77f6c20b27d8-kube-api-access-rnfnb\") pod \"network-check-target-vnmfs\" (UID: \"48d628be-136e-4521-a2fa-77f6c20b27d8\") " pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:17:09.555735 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:17:09.555590    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs\") pod \"network-metrics-daemon-zzxwf\" (UID: \"bbd14141-b0c9-4e6d-aab3-c1091adc5ddf\") " pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:17:09.558665 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:17:09.558644    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 16:17:09.558728 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:17:09.558654    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 16:17:09.566320 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:17:09.566296    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 16:17:09.566440 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:17:09.566383    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs podName:bbd14141-b0c9-4e6d-aab3-c1091adc5ddf nodeName:}" failed. No retries permitted until 2026-09-18 16:18:13.566354513 +0000 UTC m=+132.021807581 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs") pod "network-metrics-daemon-zzxwf" (UID: "bbd14141-b0c9-4e6d-aab3-c1091adc5ddf") : secret "metrics-daemon-secret" not found
Sep 18 16:17:09.568999 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:17:09.568978    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 16:17:09.579354 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:17:09.579333    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnfnb\" (UniqueName: \"kubernetes.io/projected/48d628be-136e-4521-a2fa-77f6c20b27d8-kube-api-access-rnfnb\") pod \"network-check-target-vnmfs\" (UID: \"48d628be-136e-4521-a2fa-77f6c20b27d8\") " pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:17:09.591244 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:17:09.591220    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwgv5"
Sep 18 16:17:09.604269 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:17:09.604247    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-75jt8\""
Sep 18 16:17:09.611920 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:17:09.611901    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:17:09.891428 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:17:09.891398    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gwgv5"]
Sep 18 16:17:09.895548 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:17:09.895523    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d26f9a_f3f1_417e_91bd_0e74248bbbb4.slice/crio-a650ba55cafcc780a2b2b5477098a113597e8d61934dbee6bf95cfd626c35921 WatchSource:0}: Error finding container a650ba55cafcc780a2b2b5477098a113597e8d61934dbee6bf95cfd626c35921: Status 404 returned error can't find the container with id a650ba55cafcc780a2b2b5477098a113597e8d61934dbee6bf95cfd626c35921
Sep 18 16:17:09.907790 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:17:09.907763    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vnmfs"]
Sep 18 16:17:09.910784 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:17:09.910760    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d628be_136e_4521_a2fa_77f6c20b27d8.slice/crio-bee1b476be282241f9e2493300e57a53a485fc307cec0d6f61465ac8c3a58794 WatchSource:0}: Error finding container bee1b476be282241f9e2493300e57a53a485fc307cec0d6f61465ac8c3a58794: Status 404 returned error can't find the container with id bee1b476be282241f9e2493300e57a53a485fc307cec0d6f61465ac8c3a58794
Sep 18 16:17:10.024582 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:17:10.024539    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vnmfs" event={"ID":"48d628be-136e-4521-a2fa-77f6c20b27d8","Type":"ContainerStarted","Data":"bee1b476be282241f9e2493300e57a53a485fc307cec0d6f61465ac8c3a58794"}
Sep 18 16:17:10.025554 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:17:10.025534    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gwgv5" event={"ID":"94d26f9a-f3f1-417e-91bd-0e74248bbbb4","Type":"ContainerStarted","Data":"a650ba55cafcc780a2b2b5477098a113597e8d61934dbee6bf95cfd626c35921"}
Sep 18 16:17:15.041144 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:17:15.041104    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vnmfs" event={"ID":"48d628be-136e-4521-a2fa-77f6c20b27d8","Type":"ContainerStarted","Data":"d58a91c4352c56e012dface997515d12f9bd7cac5f739c6eb65c25fa0fd2426d"}
Sep 18 16:17:15.041641 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:17:15.041209    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:17:15.042490 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:17:15.042469    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gwgv5" event={"ID":"94d26f9a-f3f1-417e-91bd-0e74248bbbb4","Type":"ContainerStarted","Data":"3a10edec211d2b13468c067714a45fcfd22b141d3084f0a434f525a469eeea4e"}
Sep 18 16:17:15.094714 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:17:15.094658    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vnmfs" podStartSLOduration=67.827980965 podStartE2EDuration="1m12.094641529s" totalImagesPullingTime="4.266660564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:17:15.064979503 +0000 UTC m=+73.520432580" watchObservedRunningTime="2026-09-18 16:17:15.094641529 +0000 UTC m=+73.550094603"
Sep 18 16:17:15.094871 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:17:15.094769    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gwgv5" podStartSLOduration=67.809448931 podStartE2EDuration="1m12.094763113s" totalImagesPullingTime="4.285314182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:17:15.093257603 +0000 UTC m=+73.548710680" watchObservedRunningTime="2026-09-18 16:17:15.094763113 +0000 UTC m=+73.550216190"
Sep 18 16:17:40.767138 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:17:40.767099    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls\") pod \"dns-default-787hc\" (UID: \"83e8eb33-e386-4b60-98c6-a95b93c408cd\") " pod="openshift-dns/dns-default-787hc"
Sep 18 16:17:40.767483 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:17:40.767248    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:17:40.767483 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:17:40.767317    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls podName:83e8eb33-e386-4b60-98c6-a95b93c408cd nodeName:}" failed. No retries permitted until 2026-09-18 16:18:44.767300943 +0000 UTC m=+163.222754003 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls") pod "dns-default-787hc" (UID: "83e8eb33-e386-4b60-98c6-a95b93c408cd") : secret "dns-default-metrics-tls" not found
Sep 18 16:17:46.047465 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:17:46.047435    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vnmfs"
Sep 18 16:18:13.593250 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:18:13.593207    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs\") pod \"network-metrics-daemon-zzxwf\" (UID: \"bbd14141-b0c9-4e6d-aab3-c1091adc5ddf\") " pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:18:13.593722 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:18:13.593334    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 16:18:13.593722 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:18:13.593394    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs podName:bbd14141-b0c9-4e6d-aab3-c1091adc5ddf nodeName:}" failed. No retries permitted until 2026-09-18 16:20:15.59337657 +0000 UTC m=+254.048829627 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs") pod "network-metrics-daemon-zzxwf" (UID: "bbd14141-b0c9-4e6d-aab3-c1091adc5ddf") : secret "metrics-daemon-secret" not found
Sep 18 16:18:39.908272 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:18:39.908230    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-787hc" podUID="83e8eb33-e386-4b60-98c6-a95b93c408cd"
Sep 18 16:18:40.235276 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:18:40.235202    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-787hc"
Sep 18 16:18:41.696495 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:18:41.696457    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-zzxwf" podUID="bbd14141-b0c9-4e6d-aab3-c1091adc5ddf"
Sep 18 16:18:44.794272 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:18:44.794204    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls\") pod \"dns-default-787hc\" (UID: \"83e8eb33-e386-4b60-98c6-a95b93c408cd\") " pod="openshift-dns/dns-default-787hc"
Sep 18 16:18:44.794758 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:18:44.794374    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:18:44.794758 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:18:44.794482    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls podName:83e8eb33-e386-4b60-98c6-a95b93c408cd nodeName:}" failed. No retries permitted until 2026-09-18 16:20:46.794461742 +0000 UTC m=+285.249914802 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls") pod "dns-default-787hc" (UID: "83e8eb33-e386-4b60-98c6-a95b93c408cd") : secret "dns-default-metrics-tls" not found
Sep 18 16:18:54.680821 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:18:54.680778    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:20:13.300553 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.300514    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jrgv9"]
Sep 18 16:20:13.302443 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.302426    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-vfmdv"]
Sep 18 16:20:13.302572 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.302554    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jrgv9"
Sep 18 16:20:13.304601 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.304577    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-vfmdv"
Sep 18 16:20:13.305471 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.305448    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 18 16:20:13.305715 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.305699    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-qgsk9\""
Sep 18 16:20:13.305797 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.305749    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:20:13.305797 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.305757    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 18 16:20:13.306995 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.306981    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 18 16:20:13.307232 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.307218    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-dktk2\""
Sep 18 16:20:13.307417 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.307403    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 18 16:20:13.307591 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.307577    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:20:13.308123 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.308105    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 18 16:20:13.308341 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.308328    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 18 16:20:13.312915 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.312896    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 18 16:20:13.329910 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.329886    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-vfmdv"]
Sep 18 16:20:13.330579 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.330562    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jrgv9"]
Sep 18 16:20:13.384089 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.384054    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-v2fwq"]
Sep 18 16:20:13.385807 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.385786    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-v2fwq"
Sep 18 16:20:13.388534 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.388508    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f6db6c45f-672rr"]
Sep 18 16:20:13.391361 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.391339    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:13.392140 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.392119    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw2fz\" (UniqueName: \"kubernetes.io/projected/dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5-kube-api-access-vw2fz\") pod \"kube-storage-version-migrator-operator-55889f7b47-jrgv9\" (UID: \"dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jrgv9"
Sep 18 16:20:13.392236 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.392149    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e3ca1ec-6856-4afb-850e-85e3fa968b3f-trusted-ca\") pod \"console-operator-fd4bb46c-vfmdv\" (UID: \"1e3ca1ec-6856-4afb-850e-85e3fa968b3f\") " pod="openshift-console-operator/console-operator-fd4bb46c-vfmdv"
Sep 18 16:20:13.392236 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.392196    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-jrgv9\" (UID: \"dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jrgv9"
Sep 18 16:20:13.392236 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.392223    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e3ca1ec-6856-4afb-850e-85e3fa968b3f-serving-cert\") pod \"console-operator-fd4bb46c-vfmdv\" (UID: \"1e3ca1ec-6856-4afb-850e-85e3fa968b3f\") " pod="openshift-console-operator/console-operator-fd4bb46c-vfmdv"
Sep 18 16:20:13.392375 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.392264    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e3ca1ec-6856-4afb-850e-85e3fa968b3f-config\") pod \"console-operator-fd4bb46c-vfmdv\" (UID: \"1e3ca1ec-6856-4afb-850e-85e3fa968b3f\") " pod="openshift-console-operator/console-operator-fd4bb46c-vfmdv"
Sep 18 16:20:13.392375 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.392289    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdd25\" (UniqueName: \"kubernetes.io/projected/1e3ca1ec-6856-4afb-850e-85e3fa968b3f-kube-api-access-xdd25\") pod \"console-operator-fd4bb46c-vfmdv\" (UID: \"1e3ca1ec-6856-4afb-850e-85e3fa968b3f\") " pod="openshift-console-operator/console-operator-fd4bb46c-vfmdv"
Sep 18 16:20:13.392375 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.392312    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-jrgv9\" (UID: \"dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jrgv9"
Sep 18 16:20:13.392712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.392696    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 18 16:20:13.392758 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.392718    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-tkf5q\""
Sep 18 16:20:13.398737 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.398720    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 16:20:13.398852 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.398766    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 16:20:13.398982 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.398957    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 16:20:13.399142 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.398992    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 16:20:13.399142 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.398991    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 18 16:20:13.399142 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.399022    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 16:20:13.399459 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.399442    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-czbcv\""
Sep 18 16:20:13.409465 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.409426    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-v2fwq"]
Sep 18 16:20:13.410126 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.410110    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f6db6c45f-672rr"]
Sep 18 16:20:13.411014 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.410997    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 18 16:20:13.412285 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.412269    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 16:20:13.492990 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.492948    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b329cbb-a9d0-4b69-bedb-e754d774d1ac-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-v2fwq\" (UID: \"7b329cbb-a9d0-4b69-bedb-e754d774d1ac\") " pod="openshift-insights/insights-operator-6f9c644dd9-v2fwq"
Sep 18 16:20:13.492990 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.492983    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b329cbb-a9d0-4b69-bedb-e754d774d1ac-serving-cert\") pod \"insights-operator-6f9c644dd9-v2fwq\" (UID: \"7b329cbb-a9d0-4b69-bedb-e754d774d1ac\") " pod="openshift-insights/insights-operator-6f9c644dd9-v2fwq"
Sep 18 16:20:13.493255 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.493044    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b329cbb-a9d0-4b69-bedb-e754d774d1ac-tmp\") pod \"insights-operator-6f9c644dd9-v2fwq\" (UID: \"7b329cbb-a9d0-4b69-bedb-e754d774d1ac\") " pod="openshift-insights/insights-operator-6f9c644dd9-v2fwq"
Sep 18 16:20:13.493255 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.493071    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6phr\" (UniqueName: \"kubernetes.io/projected/7b329cbb-a9d0-4b69-bedb-e754d774d1ac-kube-api-access-p6phr\") pod \"insights-operator-6f9c644dd9-v2fwq\" (UID: \"7b329cbb-a9d0-4b69-bedb-e754d774d1ac\") " pod="openshift-insights/insights-operator-6f9c644dd9-v2fwq"
Sep 18 16:20:13.493255 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.493119    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-jrgv9\" (UID: \"dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jrgv9"
Sep 18 16:20:13.493255 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.493136    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-registry-tls\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:13.493255 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.493153    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-bound-sa-token\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:13.493255 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.493173    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e3ca1ec-6856-4afb-850e-85e3fa968b3f-serving-cert\") pod \"console-operator-fd4bb46c-vfmdv\" (UID: \"1e3ca1ec-6856-4afb-850e-85e3fa968b3f\") " pod="openshift-console-operator/console-operator-fd4bb46c-vfmdv"
Sep 18 16:20:13.493255 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.493189    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-image-registry-private-configuration\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:13.493255 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.493214    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-ca-trust-extracted\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:13.493650 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.493337    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-registry-certificates\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:13.493650 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.493388    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e3ca1ec-6856-4afb-850e-85e3fa968b3f-config\") pod \"console-operator-fd4bb46c-vfmdv\" (UID: \"1e3ca1ec-6856-4afb-850e-85e3fa968b3f\") " pod="openshift-console-operator/console-operator-fd4bb46c-vfmdv"
Sep 18 16:20:13.493650 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.493410    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdd25\" (UniqueName: \"kubernetes.io/projected/1e3ca1ec-6856-4afb-850e-85e3fa968b3f-kube-api-access-xdd25\") pod \"console-operator-fd4bb46c-vfmdv\" (UID: \"1e3ca1ec-6856-4afb-850e-85e3fa968b3f\") " pod="openshift-console-operator/console-operator-fd4bb46c-vfmdv"
Sep 18 16:20:13.493650 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.493426    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-jrgv9\" (UID: \"dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jrgv9"
Sep 18 16:20:13.493650 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.493470    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw2fz\" (UniqueName: \"kubernetes.io/projected/dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5-kube-api-access-vw2fz\") pod \"kube-storage-version-migrator-operator-55889f7b47-jrgv9\" (UID: \"dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jrgv9"
Sep 18 16:20:13.493650 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.493497    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b329cbb-a9d0-4b69-bedb-e754d774d1ac-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-v2fwq\" (UID: \"7b329cbb-a9d0-4b69-bedb-e754d774d1ac\") " pod="openshift-insights/insights-operator-6f9c644dd9-v2fwq"
Sep 18 16:20:13.493650 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.493518    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-installation-pull-secrets\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:13.493650 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.493544    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e3ca1ec-6856-4afb-850e-85e3fa968b3f-trusted-ca\") pod \"console-operator-fd4bb46c-vfmdv\" (UID: \"1e3ca1ec-6856-4afb-850e-85e3fa968b3f\") " pod="openshift-console-operator/console-operator-fd4bb46c-vfmdv"
Sep 18 16:20:13.493650 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.493588    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-trusted-ca\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:13.493650 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.493611    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6twk\" (UniqueName: \"kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-kube-api-access-x6twk\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:13.493650 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.493637    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7b329cbb-a9d0-4b69-bedb-e754d774d1ac-snapshots\") pod \"insights-operator-6f9c644dd9-v2fwq\" (UID: \"7b329cbb-a9d0-4b69-bedb-e754d774d1ac\") " pod="openshift-insights/insights-operator-6f9c644dd9-v2fwq"
Sep 18 16:20:13.494219 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.494163    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-jrgv9\" (UID: \"dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jrgv9"
Sep 18 16:20:13.494219 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.494163    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e3ca1ec-6856-4afb-850e-85e3fa968b3f-config\") pod \"console-operator-fd4bb46c-vfmdv\" (UID: \"1e3ca1ec-6856-4afb-850e-85e3fa968b3f\") " pod="openshift-console-operator/console-operator-fd4bb46c-vfmdv"
Sep 18 16:20:13.494651 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.494625    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e3ca1ec-6856-4afb-850e-85e3fa968b3f-trusted-ca\") pod \"console-operator-fd4bb46c-vfmdv\" (UID: \"1e3ca1ec-6856-4afb-850e-85e3fa968b3f\") " pod="openshift-console-operator/console-operator-fd4bb46c-vfmdv"
Sep 18 16:20:13.495589 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.495567    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-jrgv9\" (UID: \"dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jrgv9"
Sep 18 16:20:13.495589 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.495581    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e3ca1ec-6856-4afb-850e-85e3fa968b3f-serving-cert\") pod \"console-operator-fd4bb46c-vfmdv\" (UID: \"1e3ca1ec-6856-4afb-850e-85e3fa968b3f\") " pod="openshift-console-operator/console-operator-fd4bb46c-vfmdv"
Sep 18 16:20:13.502144 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.502119    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw2fz\" (UniqueName: \"kubernetes.io/projected/dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5-kube-api-access-vw2fz\") pod \"kube-storage-version-migrator-operator-55889f7b47-jrgv9\" (UID: \"dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jrgv9"
Sep 18 16:20:13.502234 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.502189    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdd25\" (UniqueName: \"kubernetes.io/projected/1e3ca1ec-6856-4afb-850e-85e3fa968b3f-kube-api-access-xdd25\") pod \"console-operator-fd4bb46c-vfmdv\" (UID: \"1e3ca1ec-6856-4afb-850e-85e3fa968b3f\") " pod="openshift-console-operator/console-operator-fd4bb46c-vfmdv"
Sep 18 16:20:13.593958 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.593925    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b329cbb-a9d0-4b69-bedb-e754d774d1ac-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-v2fwq\" (UID: \"7b329cbb-a9d0-4b69-bedb-e754d774d1ac\") " pod="openshift-insights/insights-operator-6f9c644dd9-v2fwq"
Sep 18 16:20:13.593958 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.593961    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b329cbb-a9d0-4b69-bedb-e754d774d1ac-serving-cert\") pod \"insights-operator-6f9c644dd9-v2fwq\" (UID: \"7b329cbb-a9d0-4b69-bedb-e754d774d1ac\") " pod="openshift-insights/insights-operator-6f9c644dd9-v2fwq"
Sep 18 16:20:13.594228 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.593991    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b329cbb-a9d0-4b69-bedb-e754d774d1ac-tmp\") pod \"insights-operator-6f9c644dd9-v2fwq\" (UID: \"7b329cbb-a9d0-4b69-bedb-e754d774d1ac\") " pod="openshift-insights/insights-operator-6f9c644dd9-v2fwq"
Sep 18 16:20:13.594228 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.594007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6phr\" (UniqueName: \"kubernetes.io/projected/7b329cbb-a9d0-4b69-bedb-e754d774d1ac-kube-api-access-p6phr\") pod \"insights-operator-6f9c644dd9-v2fwq\" (UID: \"7b329cbb-a9d0-4b69-bedb-e754d774d1ac\") " pod="openshift-insights/insights-operator-6f9c644dd9-v2fwq"
Sep 18 16:20:13.594228 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.594025    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-registry-tls\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:13.594228 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.594102    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-bound-sa-token\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:13.594228 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:13.594135    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 16:20:13.594228 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:13.594149    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f6db6c45f-672rr: secret "image-registry-tls" not found
Sep 18 16:20:13.594228 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:13.594216    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-registry-tls podName:5ada52f1-14ec-41fe-a161-7e8b0fa6a12d nodeName:}" failed. No retries permitted until 2026-09-18 16:20:14.094195873 +0000 UTC m=+252.549648927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-registry-tls") pod "image-registry-f6db6c45f-672rr" (UID: "5ada52f1-14ec-41fe-a161-7e8b0fa6a12d") : secret "image-registry-tls" not found
Sep 18 16:20:13.594576 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.594141    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-image-registry-private-configuration\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:13.594576 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.594293    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-ca-trust-extracted\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:13.594576 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.594344    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-registry-certificates\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:13.594576 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.594405    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b329cbb-a9d0-4b69-bedb-e754d774d1ac-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-v2fwq\" (UID: \"7b329cbb-a9d0-4b69-bedb-e754d774d1ac\") " pod="openshift-insights/insights-operator-6f9c644dd9-v2fwq"
Sep 18 16:20:13.594576 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.594431    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-installation-pull-secrets\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:13.594576 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.594454    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b329cbb-a9d0-4b69-bedb-e754d774d1ac-tmp\") pod \"insights-operator-6f9c644dd9-v2fwq\" (UID: \"7b329cbb-a9d0-4b69-bedb-e754d774d1ac\") " pod="openshift-insights/insights-operator-6f9c644dd9-v2fwq"
Sep 18 16:20:13.594576 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.594460    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-trusted-ca\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:13.594576 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.594523    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6twk\" (UniqueName: \"kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-kube-api-access-x6twk\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:13.594576 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.594561    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7b329cbb-a9d0-4b69-bedb-e754d774d1ac-snapshots\") pod \"insights-operator-6f9c644dd9-v2fwq\" (UID: \"7b329cbb-a9d0-4b69-bedb-e754d774d1ac\") " pod="openshift-insights/insights-operator-6f9c644dd9-v2fwq"
Sep 18 16:20:13.595021 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.594686    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-ca-trust-extracted\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:13.595021 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.594960    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b329cbb-a9d0-4b69-bedb-e754d774d1ac-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-v2fwq\" (UID: \"7b329cbb-a9d0-4b69-bedb-e754d774d1ac\") " pod="openshift-insights/insights-operator-6f9c644dd9-v2fwq"
Sep 18 16:20:13.595021 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.595007    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-registry-certificates\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:13.595184 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.595164    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b329cbb-a9d0-4b69-bedb-e754d774d1ac-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-v2fwq\" (UID: \"7b329cbb-a9d0-4b69-bedb-e754d774d1ac\") " pod="openshift-insights/insights-operator-6f9c644dd9-v2fwq"
Sep 18 16:20:13.595246 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.595182    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7b329cbb-a9d0-4b69-bedb-e754d774d1ac-snapshots\") pod \"insights-operator-6f9c644dd9-v2fwq\" (UID: \"7b329cbb-a9d0-4b69-bedb-e754d774d1ac\") " pod="openshift-insights/insights-operator-6f9c644dd9-v2fwq"
Sep 18 16:20:13.595495 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.595439    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-trusted-ca\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:13.596609 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.596588    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-image-registry-private-configuration\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:13.596879 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.596864    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b329cbb-a9d0-4b69-bedb-e754d774d1ac-serving-cert\") pod \"insights-operator-6f9c644dd9-v2fwq\" (UID: \"7b329cbb-a9d0-4b69-bedb-e754d774d1ac\") " pod="openshift-insights/insights-operator-6f9c644dd9-v2fwq"
Sep 18 16:20:13.596997 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.596981    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-installation-pull-secrets\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:13.603352 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.603327    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-bound-sa-token\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:13.606439 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.606420    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6twk\" (UniqueName: \"kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-kube-api-access-x6twk\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:13.606516 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.606487    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6phr\" (UniqueName: \"kubernetes.io/projected/7b329cbb-a9d0-4b69-bedb-e754d774d1ac-kube-api-access-p6phr\") pod \"insights-operator-6f9c644dd9-v2fwq\" (UID: \"7b329cbb-a9d0-4b69-bedb-e754d774d1ac\") " pod="openshift-insights/insights-operator-6f9c644dd9-v2fwq"
Sep 18 16:20:13.613324 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.613310    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jrgv9"
Sep 18 16:20:13.617861 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.617844    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-vfmdv"
Sep 18 16:20:13.697859 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.697835    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-v2fwq"
Sep 18 16:20:13.888210 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.888122    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jrgv9"]
Sep 18 16:20:13.891440 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:20:13.891415    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee5cfd6_82ac_4c82_8ebc_65c97d7c68f5.slice/crio-54140cd1913f59044f6fe5767b90f97cfe8846637b8046e98a60187dcfea180b WatchSource:0}: Error finding container 54140cd1913f59044f6fe5767b90f97cfe8846637b8046e98a60187dcfea180b: Status 404 returned error can't find the container with id 54140cd1913f59044f6fe5767b90f97cfe8846637b8046e98a60187dcfea180b
Sep 18 16:20:13.891772 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.891751    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-vfmdv"]
Sep 18 16:20:13.893942 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:20:13.893921    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3ca1ec_6856_4afb_850e_85e3fa968b3f.slice/crio-fe367081a58b130ee55f460681787f2b0705640a7ee3a64e1be5ba897efd0682 WatchSource:0}: Error finding container fe367081a58b130ee55f460681787f2b0705640a7ee3a64e1be5ba897efd0682: Status 404 returned error can't find the container with id fe367081a58b130ee55f460681787f2b0705640a7ee3a64e1be5ba897efd0682
Sep 18 16:20:13.966795 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:13.966763    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-v2fwq"]
Sep 18 16:20:13.969311 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:20:13.969282    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b329cbb_a9d0_4b69_bedb_e754d774d1ac.slice/crio-f951352e50ada380ec484f9c3e007ebda18cc751a7a85176555c4c2caf4736f9 WatchSource:0}: Error finding container f951352e50ada380ec484f9c3e007ebda18cc751a7a85176555c4c2caf4736f9: Status 404 returned error can't find the container with id f951352e50ada380ec484f9c3e007ebda18cc751a7a85176555c4c2caf4736f9
Sep 18 16:20:14.098143 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:14.098107    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-registry-tls\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:14.098300 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:14.098228    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 16:20:14.098300 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:14.098240    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f6db6c45f-672rr: secret "image-registry-tls" not found
Sep 18 16:20:14.098300 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:14.098294    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-registry-tls podName:5ada52f1-14ec-41fe-a161-7e8b0fa6a12d nodeName:}" failed. No retries permitted until 2026-09-18 16:20:15.098280773 +0000 UTC m=+253.553733828 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-registry-tls") pod "image-registry-f6db6c45f-672rr" (UID: "5ada52f1-14ec-41fe-a161-7e8b0fa6a12d") : secret "image-registry-tls" not found
Sep 18 16:20:14.458377 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:14.458338    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jrgv9" event={"ID":"dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5","Type":"ContainerStarted","Data":"54140cd1913f59044f6fe5767b90f97cfe8846637b8046e98a60187dcfea180b"}
Sep 18 16:20:14.459383 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:14.459359    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-v2fwq" event={"ID":"7b329cbb-a9d0-4b69-bedb-e754d774d1ac","Type":"ContainerStarted","Data":"f951352e50ada380ec484f9c3e007ebda18cc751a7a85176555c4c2caf4736f9"}
Sep 18 16:20:14.460393 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:14.460372    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-vfmdv" event={"ID":"1e3ca1ec-6856-4afb-850e-85e3fa968b3f","Type":"ContainerStarted","Data":"fe367081a58b130ee55f460681787f2b0705640a7ee3a64e1be5ba897efd0682"}
Sep 18 16:20:15.105410 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:15.104582    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-registry-tls\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:15.105410 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:15.104858    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 16:20:15.105410 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:15.104898    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f6db6c45f-672rr: secret "image-registry-tls" not found
Sep 18 16:20:15.105410 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:15.104975    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-registry-tls podName:5ada52f1-14ec-41fe-a161-7e8b0fa6a12d nodeName:}" failed. No retries permitted until 2026-09-18 16:20:17.104956176 +0000 UTC m=+255.560409234 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-registry-tls") pod "image-registry-f6db6c45f-672rr" (UID: "5ada52f1-14ec-41fe-a161-7e8b0fa6a12d") : secret "image-registry-tls" not found
Sep 18 16:20:15.609094 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:15.609034    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs\") pod \"network-metrics-daemon-zzxwf\" (UID: \"bbd14141-b0c9-4e6d-aab3-c1091adc5ddf\") " pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:20:15.609586 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:15.609295    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 16:20:15.609586 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:15.609367    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs podName:bbd14141-b0c9-4e6d-aab3-c1091adc5ddf nodeName:}" failed. No retries permitted until 2026-09-18 16:22:17.609347556 +0000 UTC m=+376.064800612 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs") pod "network-metrics-daemon-zzxwf" (UID: "bbd14141-b0c9-4e6d-aab3-c1091adc5ddf") : secret "metrics-daemon-secret" not found
Sep 18 16:20:17.120208 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:17.120179    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-registry-tls\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:17.120542 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:17.120351    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 16:20:17.120542 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:17.120374    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f6db6c45f-672rr: secret "image-registry-tls" not found
Sep 18 16:20:17.120542 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:17.120438    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-registry-tls podName:5ada52f1-14ec-41fe-a161-7e8b0fa6a12d nodeName:}" failed. No retries permitted until 2026-09-18 16:20:21.120418695 +0000 UTC m=+259.575871756 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-registry-tls") pod "image-registry-f6db6c45f-672rr" (UID: "5ada52f1-14ec-41fe-a161-7e8b0fa6a12d") : secret "image-registry-tls" not found
Sep 18 16:20:17.470988 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:17.470954    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jrgv9" event={"ID":"dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5","Type":"ContainerStarted","Data":"3d432ca661fbe392cbeafec7f9da8f37ee5df7ef342815f7ff634162cda779be"}
Sep 18 16:20:17.472563 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:17.472532    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-v2fwq" event={"ID":"7b329cbb-a9d0-4b69-bedb-e754d774d1ac","Type":"ContainerStarted","Data":"1a32b2e9732adc484d15b395d8b85e6ef48a198c31f86931b633be94a652f95e"}
Sep 18 16:20:17.474198 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:17.474175    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-vfmdv_1e3ca1ec-6856-4afb-850e-85e3fa968b3f/console-operator/0.log"
Sep 18 16:20:17.474304 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:17.474215    2566 generic.go:356] "Generic (PLEG): container finished" podID="1e3ca1ec-6856-4afb-850e-85e3fa968b3f" containerID="e553a06bc2cc558884f3cac89bcac331e8d74e0ada93da97caad7f1d3ad6d8ba" exitCode=255
Sep 18 16:20:17.474304 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:17.474268    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-vfmdv" event={"ID":"1e3ca1ec-6856-4afb-850e-85e3fa968b3f","Type":"ContainerDied","Data":"e553a06bc2cc558884f3cac89bcac331e8d74e0ada93da97caad7f1d3ad6d8ba"}
Sep 18 16:20:17.474491 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:17.474476    2566 scope.go:117] "RemoveContainer" containerID="e553a06bc2cc558884f3cac89bcac331e8d74e0ada93da97caad7f1d3ad6d8ba"
Sep 18 16:20:17.489008 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:17.488961    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jrgv9" podStartSLOduration=1.421065802 podStartE2EDuration="4.488942218s" totalImagesPullingTime="3.067876416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:17.487873315 +0000 UTC m=+255.943326383" watchObservedRunningTime="2026-09-18 16:20:17.488942218 +0000 UTC m=+255.944395298"
Sep 18 16:20:17.522794 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:17.522731    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-v2fwq" podStartSLOduration=1.5386222429999998 podStartE2EDuration="4.52271092s" totalImagesPullingTime="2.984088677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:17.520878338 +0000 UTC m=+255.976331427" watchObservedRunningTime="2026-09-18 16:20:17.52271092 +0000 UTC m=+255.978163998"
Sep 18 16:20:18.479588 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:18.479563    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-vfmdv_1e3ca1ec-6856-4afb-850e-85e3fa968b3f/console-operator/1.log"
Sep 18 16:20:18.479967 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:18.479953    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-vfmdv_1e3ca1ec-6856-4afb-850e-85e3fa968b3f/console-operator/0.log"
Sep 18 16:20:18.480024 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:18.479986    2566 generic.go:356] "Generic (PLEG): container finished" podID="1e3ca1ec-6856-4afb-850e-85e3fa968b3f" containerID="c86d582410ad7a2cb9d1c0a477404681cf57f7e179bcc7129b400c39f781cdba" exitCode=255
Sep 18 16:20:18.480114 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:18.480069    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-vfmdv" event={"ID":"1e3ca1ec-6856-4afb-850e-85e3fa968b3f","Type":"ContainerDied","Data":"c86d582410ad7a2cb9d1c0a477404681cf57f7e179bcc7129b400c39f781cdba"}
Sep 18 16:20:18.480189 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:18.480143    2566 scope.go:117] "RemoveContainer" containerID="e553a06bc2cc558884f3cac89bcac331e8d74e0ada93da97caad7f1d3ad6d8ba"
Sep 18 16:20:18.480320 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:18.480305    2566 scope.go:117] "RemoveContainer" containerID="c86d582410ad7a2cb9d1c0a477404681cf57f7e179bcc7129b400c39f781cdba"
Sep 18 16:20:18.480543 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:18.480507    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-vfmdv_openshift-console-operator(1e3ca1ec-6856-4afb-850e-85e3fa968b3f)\"" pod="openshift-console-operator/console-operator-fd4bb46c-vfmdv" podUID="1e3ca1ec-6856-4afb-850e-85e3fa968b3f"
Sep 18 16:20:18.546590 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:18.546549    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cxv7j"]
Sep 18 16:20:18.549263 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:18.549236    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cxv7j"
Sep 18 16:20:18.551764 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:18.551738    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rrvrk\""
Sep 18 16:20:18.552032 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:18.552012    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 16:20:18.552202 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:18.552187    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 16:20:18.560390 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:18.560153    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cxv7j"]
Sep 18 16:20:18.631746 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:18.631708    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/14e3d714-83c0-4ef8-8468-4143f8f2852d-crio-socket\") pod \"insights-runtime-extractor-cxv7j\" (UID: \"14e3d714-83c0-4ef8-8468-4143f8f2852d\") " pod="openshift-insights/insights-runtime-extractor-cxv7j"
Sep 18 16:20:18.631746 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:18.631746    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/14e3d714-83c0-4ef8-8468-4143f8f2852d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cxv7j\" (UID: \"14e3d714-83c0-4ef8-8468-4143f8f2852d\") " pod="openshift-insights/insights-runtime-extractor-cxv7j"
Sep 18 16:20:18.632033 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:18.631776    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/14e3d714-83c0-4ef8-8468-4143f8f2852d-data-volume\") pod \"insights-runtime-extractor-cxv7j\" (UID: \"14e3d714-83c0-4ef8-8468-4143f8f2852d\") " pod="openshift-insights/insights-runtime-extractor-cxv7j"
Sep 18 16:20:18.632033 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:18.631936    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/14e3d714-83c0-4ef8-8468-4143f8f2852d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cxv7j\" (UID: \"14e3d714-83c0-4ef8-8468-4143f8f2852d\") " pod="openshift-insights/insights-runtime-extractor-cxv7j"
Sep 18 16:20:18.632166 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:18.632106    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2ffb\" (UniqueName: \"kubernetes.io/projected/14e3d714-83c0-4ef8-8468-4143f8f2852d-kube-api-access-s2ffb\") pod \"insights-runtime-extractor-cxv7j\" (UID: \"14e3d714-83c0-4ef8-8468-4143f8f2852d\") " pod="openshift-insights/insights-runtime-extractor-cxv7j"
Sep 18 16:20:18.733603 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:18.733501    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/14e3d714-83c0-4ef8-8468-4143f8f2852d-crio-socket\") pod \"insights-runtime-extractor-cxv7j\" (UID: \"14e3d714-83c0-4ef8-8468-4143f8f2852d\") " pod="openshift-insights/insights-runtime-extractor-cxv7j"
Sep 18 16:20:18.733603 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:18.733546    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/14e3d714-83c0-4ef8-8468-4143f8f2852d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cxv7j\" (UID: \"14e3d714-83c0-4ef8-8468-4143f8f2852d\") " pod="openshift-insights/insights-runtime-extractor-cxv7j"
Sep 18 16:20:18.733603 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:18.733567    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/14e3d714-83c0-4ef8-8468-4143f8f2852d-data-volume\") pod \"insights-runtime-extractor-cxv7j\" (UID: \"14e3d714-83c0-4ef8-8468-4143f8f2852d\") " pod="openshift-insights/insights-runtime-extractor-cxv7j"
Sep 18 16:20:18.733873 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:18.733646    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:20:18.733873 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:18.733658    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/14e3d714-83c0-4ef8-8468-4143f8f2852d-crio-socket\") pod \"insights-runtime-extractor-cxv7j\" (UID: \"14e3d714-83c0-4ef8-8468-4143f8f2852d\") " pod="openshift-insights/insights-runtime-extractor-cxv7j"
Sep 18 16:20:18.733873 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:18.733691    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/14e3d714-83c0-4ef8-8468-4143f8f2852d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cxv7j\" (UID: \"14e3d714-83c0-4ef8-8468-4143f8f2852d\") " pod="openshift-insights/insights-runtime-extractor-cxv7j"
Sep 18 16:20:18.733873 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:18.733719    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/14e3d714-83c0-4ef8-8468-4143f8f2852d-kube-rbac-proxy-cm podName:14e3d714-83c0-4ef8-8468-4143f8f2852d nodeName:}" failed. No retries permitted until 2026-09-18 16:20:19.233702806 +0000 UTC m=+257.689155866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/14e3d714-83c0-4ef8-8468-4143f8f2852d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cxv7j" (UID: "14e3d714-83c0-4ef8-8468-4143f8f2852d") : configmap "kube-rbac-proxy" not found
Sep 18 16:20:18.733873 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:18.733768    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:18.733873 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:18.733766    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2ffb\" (UniqueName: \"kubernetes.io/projected/14e3d714-83c0-4ef8-8468-4143f8f2852d-kube-api-access-s2ffb\") pod \"insights-runtime-extractor-cxv7j\" (UID: \"14e3d714-83c0-4ef8-8468-4143f8f2852d\") " pod="openshift-insights/insights-runtime-extractor-cxv7j"
Sep 18 16:20:18.733873 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:18.733816    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14e3d714-83c0-4ef8-8468-4143f8f2852d-insights-runtime-extractor-tls podName:14e3d714-83c0-4ef8-8468-4143f8f2852d nodeName:}" failed. No retries permitted until 2026-09-18 16:20:19.233804098 +0000 UTC m=+257.689257153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/14e3d714-83c0-4ef8-8468-4143f8f2852d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cxv7j" (UID: "14e3d714-83c0-4ef8-8468-4143f8f2852d") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:18.733873 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:18.733874    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/14e3d714-83c0-4ef8-8468-4143f8f2852d-data-volume\") pod \"insights-runtime-extractor-cxv7j\" (UID: \"14e3d714-83c0-4ef8-8468-4143f8f2852d\") " pod="openshift-insights/insights-runtime-extractor-cxv7j"
Sep 18 16:20:18.744870 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:18.744846    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2ffb\" (UniqueName: \"kubernetes.io/projected/14e3d714-83c0-4ef8-8468-4143f8f2852d-kube-api-access-s2ffb\") pod \"insights-runtime-extractor-cxv7j\" (UID: \"14e3d714-83c0-4ef8-8468-4143f8f2852d\") " pod="openshift-insights/insights-runtime-extractor-cxv7j"
Sep 18 16:20:19.237143 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:19.237071    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/14e3d714-83c0-4ef8-8468-4143f8f2852d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cxv7j\" (UID: \"14e3d714-83c0-4ef8-8468-4143f8f2852d\") " pod="openshift-insights/insights-runtime-extractor-cxv7j"
Sep 18 16:20:19.237352 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:19.237168    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/14e3d714-83c0-4ef8-8468-4143f8f2852d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cxv7j\" (UID: \"14e3d714-83c0-4ef8-8468-4143f8f2852d\") " pod="openshift-insights/insights-runtime-extractor-cxv7j"
Sep 18 16:20:19.237352 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:19.237226    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:20:19.237352 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:19.237254    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:19.237352 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:19.237309    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14e3d714-83c0-4ef8-8468-4143f8f2852d-insights-runtime-extractor-tls podName:14e3d714-83c0-4ef8-8468-4143f8f2852d nodeName:}" failed. No retries permitted until 2026-09-18 16:20:20.237296451 +0000 UTC m=+258.692749506 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/14e3d714-83c0-4ef8-8468-4143f8f2852d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cxv7j" (UID: "14e3d714-83c0-4ef8-8468-4143f8f2852d") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:19.237352 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:19.237322    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/14e3d714-83c0-4ef8-8468-4143f8f2852d-kube-rbac-proxy-cm podName:14e3d714-83c0-4ef8-8468-4143f8f2852d nodeName:}" failed. No retries permitted until 2026-09-18 16:20:20.237316283 +0000 UTC m=+258.692769338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/14e3d714-83c0-4ef8-8468-4143f8f2852d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cxv7j" (UID: "14e3d714-83c0-4ef8-8468-4143f8f2852d") : configmap "kube-rbac-proxy" not found
Sep 18 16:20:19.484275 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:19.484250    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-vfmdv_1e3ca1ec-6856-4afb-850e-85e3fa968b3f/console-operator/1.log"
Sep 18 16:20:19.484650 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:19.484636    2566 scope.go:117] "RemoveContainer" containerID="c86d582410ad7a2cb9d1c0a477404681cf57f7e179bcc7129b400c39f781cdba"
Sep 18 16:20:19.484845 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:19.484825    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-vfmdv_openshift-console-operator(1e3ca1ec-6856-4afb-850e-85e3fa968b3f)\"" pod="openshift-console-operator/console-operator-fd4bb46c-vfmdv" podUID="1e3ca1ec-6856-4afb-850e-85e3fa968b3f"
Sep 18 16:20:19.751068 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:19.751037    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-t68mp"]
Sep 18 16:20:19.753060 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:19.753044    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-t68mp"
Sep 18 16:20:19.755867 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:19.755848    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 18 16:20:19.756067 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:19.756053    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 18 16:20:19.756152 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:19.756138    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 18 16:20:19.756201 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:19.756152    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-tssgq\""
Sep 18 16:20:19.757085 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:19.757057    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 18 16:20:19.767070 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:19.767048    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-t68mp"]
Sep 18 16:20:19.841486 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:19.841443    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bebf0601-b9c4-44cf-8cdc-ef7664697470-signing-key\") pod \"service-ca-55f6fc7b86-t68mp\" (UID: \"bebf0601-b9c4-44cf-8cdc-ef7664697470\") " pod="openshift-service-ca/service-ca-55f6fc7b86-t68mp"
Sep 18 16:20:19.841677 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:19.841564    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wj6f\" (UniqueName: \"kubernetes.io/projected/bebf0601-b9c4-44cf-8cdc-ef7664697470-kube-api-access-8wj6f\") pod \"service-ca-55f6fc7b86-t68mp\" (UID: \"bebf0601-b9c4-44cf-8cdc-ef7664697470\") " pod="openshift-service-ca/service-ca-55f6fc7b86-t68mp"
Sep 18 16:20:19.841677 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:19.841650    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bebf0601-b9c4-44cf-8cdc-ef7664697470-signing-cabundle\") pod \"service-ca-55f6fc7b86-t68mp\" (UID: \"bebf0601-b9c4-44cf-8cdc-ef7664697470\") " pod="openshift-service-ca/service-ca-55f6fc7b86-t68mp"
Sep 18 16:20:19.942978 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:19.942932    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wj6f\" (UniqueName: \"kubernetes.io/projected/bebf0601-b9c4-44cf-8cdc-ef7664697470-kube-api-access-8wj6f\") pod \"service-ca-55f6fc7b86-t68mp\" (UID: \"bebf0601-b9c4-44cf-8cdc-ef7664697470\") " pod="openshift-service-ca/service-ca-55f6fc7b86-t68mp"
Sep 18 16:20:19.943199 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:19.943038    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bebf0601-b9c4-44cf-8cdc-ef7664697470-signing-cabundle\") pod \"service-ca-55f6fc7b86-t68mp\" (UID: \"bebf0601-b9c4-44cf-8cdc-ef7664697470\") " pod="openshift-service-ca/service-ca-55f6fc7b86-t68mp"
Sep 18 16:20:19.943199 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:19.943098    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bebf0601-b9c4-44cf-8cdc-ef7664697470-signing-key\") pod \"service-ca-55f6fc7b86-t68mp\" (UID: \"bebf0601-b9c4-44cf-8cdc-ef7664697470\") " pod="openshift-service-ca/service-ca-55f6fc7b86-t68mp"
Sep 18 16:20:19.943693 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:19.943666    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bebf0601-b9c4-44cf-8cdc-ef7664697470-signing-cabundle\") pod \"service-ca-55f6fc7b86-t68mp\" (UID: \"bebf0601-b9c4-44cf-8cdc-ef7664697470\") " pod="openshift-service-ca/service-ca-55f6fc7b86-t68mp"
Sep 18 16:20:19.945418 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:19.945397    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bebf0601-b9c4-44cf-8cdc-ef7664697470-signing-key\") pod \"service-ca-55f6fc7b86-t68mp\" (UID: \"bebf0601-b9c4-44cf-8cdc-ef7664697470\") " pod="openshift-service-ca/service-ca-55f6fc7b86-t68mp"
Sep 18 16:20:19.951371 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:19.951351    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wj6f\" (UniqueName: \"kubernetes.io/projected/bebf0601-b9c4-44cf-8cdc-ef7664697470-kube-api-access-8wj6f\") pod \"service-ca-55f6fc7b86-t68mp\" (UID: \"bebf0601-b9c4-44cf-8cdc-ef7664697470\") " pod="openshift-service-ca/service-ca-55f6fc7b86-t68mp"
Sep 18 16:20:20.062010 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:20.061918    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-t68mp"
Sep 18 16:20:20.245301 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:20.245259    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/14e3d714-83c0-4ef8-8468-4143f8f2852d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cxv7j\" (UID: \"14e3d714-83c0-4ef8-8468-4143f8f2852d\") " pod="openshift-insights/insights-runtime-extractor-cxv7j"
Sep 18 16:20:20.245508 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:20.245343    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/14e3d714-83c0-4ef8-8468-4143f8f2852d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cxv7j\" (UID: \"14e3d714-83c0-4ef8-8468-4143f8f2852d\") " pod="openshift-insights/insights-runtime-extractor-cxv7j"
Sep 18 16:20:20.245508 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:20.245426    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:20.245620 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:20.245511    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14e3d714-83c0-4ef8-8468-4143f8f2852d-insights-runtime-extractor-tls podName:14e3d714-83c0-4ef8-8468-4143f8f2852d nodeName:}" failed. No retries permitted until 2026-09-18 16:20:22.245489878 +0000 UTC m=+260.700942951 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/14e3d714-83c0-4ef8-8468-4143f8f2852d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cxv7j" (UID: "14e3d714-83c0-4ef8-8468-4143f8f2852d") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:20.245620 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:20.245438    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:20:20.245620 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:20.245561    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/14e3d714-83c0-4ef8-8468-4143f8f2852d-kube-rbac-proxy-cm podName:14e3d714-83c0-4ef8-8468-4143f8f2852d nodeName:}" failed. No retries permitted until 2026-09-18 16:20:22.245550772 +0000 UTC m=+260.701003828 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/14e3d714-83c0-4ef8-8468-4143f8f2852d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cxv7j" (UID: "14e3d714-83c0-4ef8-8468-4143f8f2852d") : configmap "kube-rbac-proxy" not found
Sep 18 16:20:20.334899 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:20.334818    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-t68mp"]
Sep 18 16:20:20.339325 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:20:20.339296    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbebf0601_b9c4_44cf_8cdc_ef7664697470.slice/crio-6deaea4ba878a28348f32a0c5ca316c78e83bd4c5f07633db957eb2186a9f91d WatchSource:0}: Error finding container 6deaea4ba878a28348f32a0c5ca316c78e83bd4c5f07633db957eb2186a9f91d: Status 404 returned error can't find the container with id 6deaea4ba878a28348f32a0c5ca316c78e83bd4c5f07633db957eb2186a9f91d
Sep 18 16:20:20.488528 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:20.488496    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-t68mp" event={"ID":"bebf0601-b9c4-44cf-8cdc-ef7664697470","Type":"ContainerStarted","Data":"6deaea4ba878a28348f32a0c5ca316c78e83bd4c5f07633db957eb2186a9f91d"}
Sep 18 16:20:21.153798 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:21.153763    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-registry-tls\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:21.153999 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:21.153887    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 16:20:21.153999 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:21.153899    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f6db6c45f-672rr: secret "image-registry-tls" not found
Sep 18 16:20:21.153999 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:21.153949    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-registry-tls podName:5ada52f1-14ec-41fe-a161-7e8b0fa6a12d nodeName:}" failed. No retries permitted until 2026-09-18 16:20:29.153933695 +0000 UTC m=+267.609386750 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-registry-tls") pod "image-registry-f6db6c45f-672rr" (UID: "5ada52f1-14ec-41fe-a161-7e8b0fa6a12d") : secret "image-registry-tls" not found
Sep 18 16:20:21.506319 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:21.506247    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ptbtj_1d77222a-f546-4b33-b6be-fbfa3bf2f6d0/dns-node-resolver/0.log"
Sep 18 16:20:22.263658 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:22.263556    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/14e3d714-83c0-4ef8-8468-4143f8f2852d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cxv7j\" (UID: \"14e3d714-83c0-4ef8-8468-4143f8f2852d\") " pod="openshift-insights/insights-runtime-extractor-cxv7j"
Sep 18 16:20:22.263658 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:22.263620    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/14e3d714-83c0-4ef8-8468-4143f8f2852d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cxv7j\" (UID: \"14e3d714-83c0-4ef8-8468-4143f8f2852d\") " pod="openshift-insights/insights-runtime-extractor-cxv7j"
Sep 18 16:20:22.263883 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:22.263721    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:22.263883 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:22.263776    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14e3d714-83c0-4ef8-8468-4143f8f2852d-insights-runtime-extractor-tls podName:14e3d714-83c0-4ef8-8468-4143f8f2852d nodeName:}" failed. No retries permitted until 2026-09-18 16:20:26.263761009 +0000 UTC m=+264.719214068 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/14e3d714-83c0-4ef8-8468-4143f8f2852d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cxv7j" (UID: "14e3d714-83c0-4ef8-8468-4143f8f2852d") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:22.264117 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:22.264069    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/14e3d714-83c0-4ef8-8468-4143f8f2852d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cxv7j\" (UID: \"14e3d714-83c0-4ef8-8468-4143f8f2852d\") " pod="openshift-insights/insights-runtime-extractor-cxv7j"
Sep 18 16:20:22.495968 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:22.495931    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-t68mp" event={"ID":"bebf0601-b9c4-44cf-8cdc-ef7664697470","Type":"ContainerStarted","Data":"acb79f952d470183e189dd65e5b4fbfcc40fcaa80ab56d04626f32cc9cc1f3d2"}
Sep 18 16:20:22.506604 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:22.506573    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-szd7j_9665ba03-1cdc-46c9-84f2-3ca2137b66e2/node-ca/0.log"
Sep 18 16:20:22.905621 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:22.905590    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2wd2m_711861f8-db0f-4e93-a894-0cedf9efdf89/serve-healthcheck-canary/0.log"
Sep 18 16:20:23.619094 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:23.618986    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-vfmdv"
Sep 18 16:20:23.619466 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:23.619099    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-vfmdv"
Sep 18 16:20:23.619466 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:23.619448    2566 scope.go:117] "RemoveContainer" containerID="c86d582410ad7a2cb9d1c0a477404681cf57f7e179bcc7129b400c39f781cdba"
Sep 18 16:20:23.619617 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:23.619599    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-vfmdv_openshift-console-operator(1e3ca1ec-6856-4afb-850e-85e3fa968b3f)\"" pod="openshift-console-operator/console-operator-fd4bb46c-vfmdv" podUID="1e3ca1ec-6856-4afb-850e-85e3fa968b3f"
Sep 18 16:20:23.907151 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:23.907047    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-jrgv9_dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5/kube-storage-version-migrator-operator/0.log"
Sep 18 16:20:26.297047 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:26.297009    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/14e3d714-83c0-4ef8-8468-4143f8f2852d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cxv7j\" (UID: \"14e3d714-83c0-4ef8-8468-4143f8f2852d\") " pod="openshift-insights/insights-runtime-extractor-cxv7j"
Sep 18 16:20:26.297543 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:26.297154    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:26.297543 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:26.297237    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14e3d714-83c0-4ef8-8468-4143f8f2852d-insights-runtime-extractor-tls podName:14e3d714-83c0-4ef8-8468-4143f8f2852d nodeName:}" failed. No retries permitted until 2026-09-18 16:20:34.297213484 +0000 UTC m=+272.752666556 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/14e3d714-83c0-4ef8-8468-4143f8f2852d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cxv7j" (UID: "14e3d714-83c0-4ef8-8468-4143f8f2852d") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:29.218656 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:29.218617    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-registry-tls\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:29.220962 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:29.220933    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-registry-tls\") pod \"image-registry-f6db6c45f-672rr\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") " pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:29.302775 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:29.302735    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:29.579065 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:29.579004    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-t68mp" podStartSLOduration=8.939419536 podStartE2EDuration="10.578981957s" totalImagesPullingTime="1.639562421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:22.517678031 +0000 UTC m=+260.973131109" watchObservedRunningTime="2026-09-18 16:20:29.578981957 +0000 UTC m=+268.034435035"
Sep 18 16:20:29.579980 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:29.579959    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f6db6c45f-672rr"]
Sep 18 16:20:29.586520 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:20:29.586496    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ada52f1_14ec_41fe_a161_7e8b0fa6a12d.slice/crio-a6a2e439cd43add9405c0fa8de6538874db5fadbcad15658d993def9529b5cd5 WatchSource:0}: Error finding container a6a2e439cd43add9405c0fa8de6538874db5fadbcad15658d993def9529b5cd5: Status 404 returned error can't find the container with id a6a2e439cd43add9405c0fa8de6538874db5fadbcad15658d993def9529b5cd5
Sep 18 16:20:30.525582 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:30.525549    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f6db6c45f-672rr" event={"ID":"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d","Type":"ContainerStarted","Data":"a96c3422aa5307750e844c4e16313c7408654c771bb54c7982eb8c24d5a3575a"}
Sep 18 16:20:30.525945 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:30.525598    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f6db6c45f-672rr" event={"ID":"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d","Type":"ContainerStarted","Data":"a6a2e439cd43add9405c0fa8de6538874db5fadbcad15658d993def9529b5cd5"}
Sep 18 16:20:30.525945 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:30.525655    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:20:30.548796 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:30.548745    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-f6db6c45f-672rr" podStartSLOduration=17.548729942 podStartE2EDuration="17.548729942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:30.547407536 +0000 UTC m=+269.002860613" watchObservedRunningTime="2026-09-18 16:20:30.548729942 +0000 UTC m=+269.004183018"
Sep 18 16:20:34.356185 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:34.356142    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/14e3d714-83c0-4ef8-8468-4143f8f2852d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cxv7j\" (UID: \"14e3d714-83c0-4ef8-8468-4143f8f2852d\") " pod="openshift-insights/insights-runtime-extractor-cxv7j"
Sep 18 16:20:34.358373 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:34.358349    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/14e3d714-83c0-4ef8-8468-4143f8f2852d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cxv7j\" (UID: \"14e3d714-83c0-4ef8-8468-4143f8f2852d\") " pod="openshift-insights/insights-runtime-extractor-cxv7j"
Sep 18 16:20:34.462683 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:34.462644    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cxv7j"
Sep 18 16:20:34.742940 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:34.742851    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cxv7j"]
Sep 18 16:20:34.745631 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:20:34.745601    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e3d714_83c0_4ef8_8468_4143f8f2852d.slice/crio-39b8f6b36058346aa8d98b5817ba6d1f279d7f760e9ae421851d5530cb589484 WatchSource:0}: Error finding container 39b8f6b36058346aa8d98b5817ba6d1f279d7f760e9ae421851d5530cb589484: Status 404 returned error can't find the container with id 39b8f6b36058346aa8d98b5817ba6d1f279d7f760e9ae421851d5530cb589484
Sep 18 16:20:35.543494 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:35.543468    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cxv7j" event={"ID":"14e3d714-83c0-4ef8-8468-4143f8f2852d","Type":"ContainerStarted","Data":"b61e4b12cb0cc3c02f1b52ab3030f395375545f7289e182fdf94ccf0831745b3"}
Sep 18 16:20:35.543765 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:35.543522    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cxv7j" event={"ID":"14e3d714-83c0-4ef8-8468-4143f8f2852d","Type":"ContainerStarted","Data":"39b8f6b36058346aa8d98b5817ba6d1f279d7f760e9ae421851d5530cb589484"}
Sep 18 16:20:35.681904 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:35.681869    2566 scope.go:117] "RemoveContainer" containerID="c86d582410ad7a2cb9d1c0a477404681cf57f7e179bcc7129b400c39f781cdba"
Sep 18 16:20:36.549633 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:36.549594    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cxv7j" event={"ID":"14e3d714-83c0-4ef8-8468-4143f8f2852d","Type":"ContainerStarted","Data":"13bd1d0256457def8ad29eb7d7f56c8c0b64661b7e7d010202ff67c887589482"}
Sep 18 16:20:36.551799 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:36.551779    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-vfmdv_1e3ca1ec-6856-4afb-850e-85e3fa968b3f/console-operator/1.log"
Sep 18 16:20:36.551899 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:36.551838    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-vfmdv" event={"ID":"1e3ca1ec-6856-4afb-850e-85e3fa968b3f","Type":"ContainerStarted","Data":"f94fca1a70218645ee3d48fd08f38046c11f6b7613a1d84627afcea23cba24e4"}
Sep 18 16:20:36.552305 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:36.552282    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-vfmdv"
Sep 18 16:20:36.558036 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:36.557984    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-vfmdv"
Sep 18 16:20:36.574528 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:36.573475    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-vfmdv" podStartSLOduration=20.510300907 podStartE2EDuration="23.573457912s" totalImagesPullingTime="3.063157005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:36.57214611 +0000 UTC m=+275.027599188" watchObservedRunningTime="2026-09-18 16:20:36.573457912 +0000 UTC m=+275.028910995"
Sep 18 16:20:37.557050 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:37.557016    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cxv7j" event={"ID":"14e3d714-83c0-4ef8-8468-4143f8f2852d","Type":"ContainerStarted","Data":"775aa1b8fc7d56ef26c06b1a194820450b3b250862d05a965bea2a535f8b36cc"}
Sep 18 16:20:43.235912 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:43.235863    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-787hc" podUID="83e8eb33-e386-4b60-98c6-a95b93c408cd"
Sep 18 16:20:43.576708 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:43.576680    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-787hc"
Sep 18 16:20:46.851590 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:46.851552    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls\") pod \"dns-default-787hc\" (UID: \"83e8eb33-e386-4b60-98c6-a95b93c408cd\") " pod="openshift-dns/dns-default-787hc"
Sep 18 16:20:46.853800 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:46.853769    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/83e8eb33-e386-4b60-98c6-a95b93c408cd-metrics-tls\") pod \"dns-default-787hc\" (UID: \"83e8eb33-e386-4b60-98c6-a95b93c408cd\") " pod="openshift-dns/dns-default-787hc"
Sep 18 16:20:46.880447 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:46.880421    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ccb8q\""
Sep 18 16:20:46.888482 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:46.888458    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-787hc"
Sep 18 16:20:47.156968 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:47.156878    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cxv7j" podStartSLOduration=27.069275504 podStartE2EDuration="29.156860999s" totalImagesPullingTime="2.087585495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:20:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:37.578095869 +0000 UTC m=+276.033548940" watchObservedRunningTime="2026-09-18 16:20:47.156860999 +0000 UTC m=+285.612314072"
Sep 18 16:20:47.157125 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:47.157066    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-787hc"]
Sep 18 16:20:47.159970 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:20:47.159942    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e8eb33_e386_4b60_98c6_a95b93c408cd.slice/crio-2fc311d47f4ecb4d4d503093e7687582a529bff4ba5f62fe1225d8229797564f WatchSource:0}: Error finding container 2fc311d47f4ecb4d4d503093e7687582a529bff4ba5f62fe1225d8229797564f: Status 404 returned error can't find the container with id 2fc311d47f4ecb4d4d503093e7687582a529bff4ba5f62fe1225d8229797564f
Sep 18 16:20:47.589390 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:47.589349    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-787hc" event={"ID":"83e8eb33-e386-4b60-98c6-a95b93c408cd","Type":"ContainerStarted","Data":"2fc311d47f4ecb4d4d503093e7687582a529bff4ba5f62fe1225d8229797564f"}
Sep 18 16:20:49.307718 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:49.307673    2566 patch_prober.go:28] interesting pod/image-registry-f6db6c45f-672rr container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 16:20:49.308093 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:49.307733    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-f6db6c45f-672rr" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:20:49.597346 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:49.597305    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-787hc" event={"ID":"83e8eb33-e386-4b60-98c6-a95b93c408cd","Type":"ContainerStarted","Data":"65537f451db31745e46381dffc0df93e61f5f24deab0af57d2b9c827b8a368b5"}
Sep 18 16:20:49.597533 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:49.597358    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-787hc" event={"ID":"83e8eb33-e386-4b60-98c6-a95b93c408cd","Type":"ContainerStarted","Data":"959a672e73717f02870ad638701af4359f2bf5751db848e9a5817797170c2d55"}
Sep 18 16:20:49.597533 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:49.597450    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-787hc"
Sep 18 16:20:49.624464 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:49.621557    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-787hc" podStartSLOduration=252.014902629 podStartE2EDuration="4m13.621534518s" totalImagesPullingTime="1.606631889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:49.619537969 +0000 UTC m=+288.074991047" watchObservedRunningTime="2026-09-18 16:20:49.621534518 +0000 UTC m=+288.076987597"
Sep 18 16:20:51.533228 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:51.533196    2566 patch_prober.go:28] interesting pod/image-registry-f6db6c45f-672rr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 16:20:51.533656 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:51.533246    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-f6db6c45f-672rr" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:20:57.682313 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:20:57.682270    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-zzxwf" podUID="bbd14141-b0c9-4e6d-aab3-c1091adc5ddf"
Sep 18 16:20:59.307321 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:59.307289    2566 patch_prober.go:28] interesting pod/image-registry-f6db6c45f-672rr container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 16:20:59.307694 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:59.307346    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-f6db6c45f-672rr" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:20:59.602688 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:20:59.602658    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-787hc"
Sep 18 16:21:01.533319 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:01.533283    2566 patch_prober.go:28] interesting pod/image-registry-f6db6c45f-672rr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 16:21:01.533712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:01.533335    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-f6db6c45f-672rr" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:21:03.563775 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:03.563745    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-vfmdv_1e3ca1ec-6856-4afb-850e-85e3fa968b3f/console-operator/1.log"
Sep 18 16:21:03.573174 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:03.573151    2566 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 16:21:03.588556 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:03.588532    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-vfmdv_1e3ca1ec-6856-4afb-850e-85e3fa968b3f/console-operator/1.log"
Sep 18 16:21:09.306888 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:09.306851    2566 patch_prober.go:28] interesting pod/image-registry-f6db6c45f-672rr container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 16:21:09.308851 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:09.306905    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-f6db6c45f-672rr" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:21:09.308851 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:09.306946    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:21:09.308851 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:09.307369    2566 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"a96c3422aa5307750e844c4e16313c7408654c771bb54c7982eb8c24d5a3575a"} pod="openshift-image-registry/image-registry-f6db6c45f-672rr" containerMessage="Container registry failed liveness probe, will be restarted"
Sep 18 16:21:09.310865 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:09.310844    2566 patch_prober.go:28] interesting pod/image-registry-f6db6c45f-672rr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 16:21:09.310974 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:09.310882    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-f6db6c45f-672rr" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:21:12.681379 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:12.681349    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:21:19.311386 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:19.311352    2566 patch_prober.go:28] interesting pod/image-registry-f6db6c45f-672rr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 16:21:19.311854 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:19.311409    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-f6db6c45f-672rr" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:21:29.311417 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:29.311380    2566 patch_prober.go:28] interesting pod/image-registry-f6db6c45f-672rr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 16:21:29.311777 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:29.311439    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-f6db6c45f-672rr" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:21:34.325453 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:34.325403    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-f6db6c45f-672rr" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerName="registry" containerID="cri-o://a96c3422aa5307750e844c4e16313c7408654c771bb54c7982eb8c24d5a3575a" gracePeriod=30
Sep 18 16:21:34.352654 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:21:34.352622    2566 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ada52f1_14ec_41fe_a161_7e8b0fa6a12d.slice/crio-conmon-a96c3422aa5307750e844c4e16313c7408654c771bb54c7982eb8c24d5a3575a.scope\": RecentStats: unable to find data in memory cache]"
Sep 18 16:21:34.736987 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:34.736903    2566 generic.go:356] "Generic (PLEG): container finished" podID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerID="a96c3422aa5307750e844c4e16313c7408654c771bb54c7982eb8c24d5a3575a" exitCode=0
Sep 18 16:21:34.736987 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:34.736943    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f6db6c45f-672rr" event={"ID":"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d","Type":"ContainerDied","Data":"a96c3422aa5307750e844c4e16313c7408654c771bb54c7982eb8c24d5a3575a"}
Sep 18 16:21:34.737235 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:34.736990    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f6db6c45f-672rr" event={"ID":"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d","Type":"ContainerStarted","Data":"cf3668b4ff18405f632e27195524df14feabb8e69230501e20ec28efb76166a7"}
Sep 18 16:21:34.737235 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:34.737160    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:21:47.778870 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:47.778786    2566 generic.go:356] "Generic (PLEG): container finished" podID="dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5" containerID="3d432ca661fbe392cbeafec7f9da8f37ee5df7ef342815f7ff634162cda779be" exitCode=0
Sep 18 16:21:47.778870 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:47.778825    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jrgv9" event={"ID":"dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5","Type":"ContainerDied","Data":"3d432ca661fbe392cbeafec7f9da8f37ee5df7ef342815f7ff634162cda779be"}
Sep 18 16:21:47.779279 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:47.779113    2566 scope.go:117] "RemoveContainer" containerID="3d432ca661fbe392cbeafec7f9da8f37ee5df7ef342815f7ff634162cda779be"
Sep 18 16:21:48.783784 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:48.783753    2566 generic.go:356] "Generic (PLEG): container finished" podID="7b329cbb-a9d0-4b69-bedb-e754d774d1ac" containerID="1a32b2e9732adc484d15b395d8b85e6ef48a198c31f86931b633be94a652f95e" exitCode=0
Sep 18 16:21:48.784252 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:48.783825    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-v2fwq" event={"ID":"7b329cbb-a9d0-4b69-bedb-e754d774d1ac","Type":"ContainerDied","Data":"1a32b2e9732adc484d15b395d8b85e6ef48a198c31f86931b633be94a652f95e"}
Sep 18 16:21:48.784252 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:48.784198    2566 scope.go:117] "RemoveContainer" containerID="1a32b2e9732adc484d15b395d8b85e6ef48a198c31f86931b633be94a652f95e"
Sep 18 16:21:48.785862 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:48.785842    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jrgv9" event={"ID":"dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5","Type":"ContainerStarted","Data":"cabfe9f8d7fe75ffca83d72e759021a5239ebf750c12205648d9e4003067cbba"}
Sep 18 16:21:49.306875 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:49.306833    2566 patch_prober.go:28] interesting pod/image-registry-f6db6c45f-672rr container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 16:21:49.307037 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:49.306895    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-f6db6c45f-672rr" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:21:49.790561 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:49.790527    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-v2fwq" event={"ID":"7b329cbb-a9d0-4b69-bedb-e754d774d1ac","Type":"ContainerStarted","Data":"73602667d52d0d151a6e12aaa5f0869aab043d8dcc91c90a09aad14076fed44e"}
Sep 18 16:21:52.179501 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:52.179473    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-vfmdv_1e3ca1ec-6856-4afb-850e-85e3fa968b3f/console-operator/1.log"
Sep 18 16:21:52.376539 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:52.376506    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-vfmdv_1e3ca1ec-6856-4afb-850e-85e3fa968b3f/console-operator/2.log"
Sep 18 16:21:52.977913 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:52.977887    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-787hc_83e8eb33-e386-4b60-98c6-a95b93c408cd/dns/0.log"
Sep 18 16:21:53.175005 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:53.174942    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-787hc_83e8eb33-e386-4b60-98c6-a95b93c408cd/kube-rbac-proxy/0.log"
Sep 18 16:21:54.177740 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:54.177707    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ptbtj_1d77222a-f546-4b33-b6be-fbfa3bf2f6d0/dns-node-resolver/0.log"
Sep 18 16:21:54.775812 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:54.775788    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-f6db6c45f-672rr_5ada52f1-14ec-41fe-a161-7e8b0fa6a12d/registry/0.log"
Sep 18 16:21:54.975799 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:54.975771    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-f6db6c45f-672rr_5ada52f1-14ec-41fe-a161-7e8b0fa6a12d/registry/1.log"
Sep 18 16:21:55.574807 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:55.574784    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-szd7j_9665ba03-1cdc-46c9-84f2-3ca2137b66e2/node-ca/0.log"
Sep 18 16:21:55.745376 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:55.745339    2566 patch_prober.go:28] interesting pod/image-registry-f6db6c45f-672rr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 16:21:55.745524 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:55.745399    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-f6db6c45f-672rr" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:21:55.976147 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:55.976054    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2wd2m_711861f8-db0f-4e93-a894-0cedf9efdf89/serve-healthcheck-canary/0.log"
Sep 18 16:21:56.976515 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:56.976482    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-jrgv9_dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5/kube-storage-version-migrator-operator/0.log"
Sep 18 16:21:57.176260 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:57.176229    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-jrgv9_dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5/kube-storage-version-migrator-operator/1.log"
Sep 18 16:21:59.307112 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:59.307058    2566 patch_prober.go:28] interesting pod/image-registry-f6db6c45f-672rr container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 16:21:59.307472 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:21:59.307146    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-f6db6c45f-672rr" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:22:05.745566 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:05.745534    2566 patch_prober.go:28] interesting pod/image-registry-f6db6c45f-672rr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 16:22:05.745936 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:05.745591    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-f6db6c45f-672rr" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:22:09.307624 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:09.307590    2566 patch_prober.go:28] interesting pod/image-registry-f6db6c45f-672rr container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 16:22:09.308056 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:09.307653    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-f6db6c45f-672rr" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:22:09.308056 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:09.307696    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:22:09.308213 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:09.308117    2566 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"cf3668b4ff18405f632e27195524df14feabb8e69230501e20ec28efb76166a7"} pod="openshift-image-registry/image-registry-f6db6c45f-672rr" containerMessage="Container registry failed liveness probe, will be restarted"
Sep 18 16:22:09.311570 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:09.311537    2566 patch_prober.go:28] interesting pod/image-registry-f6db6c45f-672rr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 16:22:09.311680 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:09.311587    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-f6db6c45f-672rr" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:22:17.624138 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:17.624095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs\") pod \"network-metrics-daemon-zzxwf\" (UID: \"bbd14141-b0c9-4e6d-aab3-c1091adc5ddf\") " pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:22:17.626286 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:17.626267    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbd14141-b0c9-4e6d-aab3-c1091adc5ddf-metrics-certs\") pod \"network-metrics-daemon-zzxwf\" (UID: \"bbd14141-b0c9-4e6d-aab3-c1091adc5ddf\") " pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:22:17.784831 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:17.784803    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ctnzv\""
Sep 18 16:22:17.793109 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:17.793057    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzxwf"
Sep 18 16:22:18.062741 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:18.062667    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zzxwf"]
Sep 18 16:22:18.065985 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:22:18.065953    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd14141_b0c9_4e6d_aab3_c1091adc5ddf.slice/crio-8b02ba40e3041d749ec0ecfd8be55d74716cac0d8e5cdd61d3750e30b05cc1e4 WatchSource:0}: Error finding container 8b02ba40e3041d749ec0ecfd8be55d74716cac0d8e5cdd61d3750e30b05cc1e4: Status 404 returned error can't find the container with id 8b02ba40e3041d749ec0ecfd8be55d74716cac0d8e5cdd61d3750e30b05cc1e4
Sep 18 16:22:18.067897 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:18.067880    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:22:18.884189 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:18.884142    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zzxwf" event={"ID":"bbd14141-b0c9-4e6d-aab3-c1091adc5ddf","Type":"ContainerStarted","Data":"8b02ba40e3041d749ec0ecfd8be55d74716cac0d8e5cdd61d3750e30b05cc1e4"}
Sep 18 16:22:19.311822 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:19.311782    2566 patch_prober.go:28] interesting pod/image-registry-f6db6c45f-672rr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 16:22:19.311981 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:19.311838    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-f6db6c45f-672rr" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:22:19.888649 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:19.888608    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zzxwf" event={"ID":"bbd14141-b0c9-4e6d-aab3-c1091adc5ddf","Type":"ContainerStarted","Data":"d17e7e00a99a1f5b17c7c912c0fc064cb13b4ba14fa2a4dd4c42da02dcb5e3c5"}
Sep 18 16:22:19.889118 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:19.888681    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zzxwf" event={"ID":"bbd14141-b0c9-4e6d-aab3-c1091adc5ddf","Type":"ContainerStarted","Data":"c8e4b844b7fb2ab3d491771f19d7a99e9c61b0d1aedd9385ca66e22eae51c6fa"}
Sep 18 16:22:19.906615 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:19.906564    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zzxwf" podStartSLOduration=375.902879105 podStartE2EDuration="6m16.906548964s" totalImagesPullingTime="1.003669859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:22:19.905830708 +0000 UTC m=+378.361283789" watchObservedRunningTime="2026-09-18 16:22:19.906548964 +0000 UTC m=+378.362002041"
Sep 18 16:22:21.201598 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.201562    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f6db6c45f-672rr"]
Sep 18 16:22:21.327158 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.327120    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-689468cdb8-lqfkq"]
Sep 18 16:22:21.329566 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.329548    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:21.364896 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.364837    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-689468cdb8-lqfkq"]
Sep 18 16:22:21.451300 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.451269    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8367995b-3d1a-4762-8445-6cba40de5867-registry-tls\") pod \"image-registry-689468cdb8-lqfkq\" (UID: \"8367995b-3d1a-4762-8445-6cba40de5867\") " pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:21.451479 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.451312    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmwtz\" (UniqueName: \"kubernetes.io/projected/8367995b-3d1a-4762-8445-6cba40de5867-kube-api-access-lmwtz\") pod \"image-registry-689468cdb8-lqfkq\" (UID: \"8367995b-3d1a-4762-8445-6cba40de5867\") " pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:21.451479 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.451329    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8367995b-3d1a-4762-8445-6cba40de5867-registry-certificates\") pod \"image-registry-689468cdb8-lqfkq\" (UID: \"8367995b-3d1a-4762-8445-6cba40de5867\") " pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:21.451479 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.451356    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8367995b-3d1a-4762-8445-6cba40de5867-installation-pull-secrets\") pod \"image-registry-689468cdb8-lqfkq\" (UID: \"8367995b-3d1a-4762-8445-6cba40de5867\") " pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:21.451479 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.451386    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8367995b-3d1a-4762-8445-6cba40de5867-ca-trust-extracted\") pod \"image-registry-689468cdb8-lqfkq\" (UID: \"8367995b-3d1a-4762-8445-6cba40de5867\") " pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:21.451479 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.451411    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8367995b-3d1a-4762-8445-6cba40de5867-bound-sa-token\") pod \"image-registry-689468cdb8-lqfkq\" (UID: \"8367995b-3d1a-4762-8445-6cba40de5867\") " pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:21.451479 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.451433    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8367995b-3d1a-4762-8445-6cba40de5867-trusted-ca\") pod \"image-registry-689468cdb8-lqfkq\" (UID: \"8367995b-3d1a-4762-8445-6cba40de5867\") " pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:21.451707 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.451500    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8367995b-3d1a-4762-8445-6cba40de5867-image-registry-private-configuration\") pod \"image-registry-689468cdb8-lqfkq\" (UID: \"8367995b-3d1a-4762-8445-6cba40de5867\") " pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:21.552661 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.552631    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8367995b-3d1a-4762-8445-6cba40de5867-registry-tls\") pod \"image-registry-689468cdb8-lqfkq\" (UID: \"8367995b-3d1a-4762-8445-6cba40de5867\") " pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:21.552870 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.552670    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmwtz\" (UniqueName: \"kubernetes.io/projected/8367995b-3d1a-4762-8445-6cba40de5867-kube-api-access-lmwtz\") pod \"image-registry-689468cdb8-lqfkq\" (UID: \"8367995b-3d1a-4762-8445-6cba40de5867\") " pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:21.552870 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.552690    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8367995b-3d1a-4762-8445-6cba40de5867-registry-certificates\") pod \"image-registry-689468cdb8-lqfkq\" (UID: \"8367995b-3d1a-4762-8445-6cba40de5867\") " pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:21.552870 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.552711    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8367995b-3d1a-4762-8445-6cba40de5867-installation-pull-secrets\") pod \"image-registry-689468cdb8-lqfkq\" (UID: \"8367995b-3d1a-4762-8445-6cba40de5867\") " pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:21.552870 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.552729    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8367995b-3d1a-4762-8445-6cba40de5867-ca-trust-extracted\") pod \"image-registry-689468cdb8-lqfkq\" (UID: \"8367995b-3d1a-4762-8445-6cba40de5867\") " pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:21.552870 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.552755    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8367995b-3d1a-4762-8445-6cba40de5867-bound-sa-token\") pod \"image-registry-689468cdb8-lqfkq\" (UID: \"8367995b-3d1a-4762-8445-6cba40de5867\") " pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:21.553155 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.552889    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8367995b-3d1a-4762-8445-6cba40de5867-trusted-ca\") pod \"image-registry-689468cdb8-lqfkq\" (UID: \"8367995b-3d1a-4762-8445-6cba40de5867\") " pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:21.553155 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.552963    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8367995b-3d1a-4762-8445-6cba40de5867-image-registry-private-configuration\") pod \"image-registry-689468cdb8-lqfkq\" (UID: \"8367995b-3d1a-4762-8445-6cba40de5867\") " pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:21.553282 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.553261    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8367995b-3d1a-4762-8445-6cba40de5867-ca-trust-extracted\") pod \"image-registry-689468cdb8-lqfkq\" (UID: \"8367995b-3d1a-4762-8445-6cba40de5867\") " pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:21.553632 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.553608    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8367995b-3d1a-4762-8445-6cba40de5867-registry-certificates\") pod \"image-registry-689468cdb8-lqfkq\" (UID: \"8367995b-3d1a-4762-8445-6cba40de5867\") " pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:21.553912 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.553888    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8367995b-3d1a-4762-8445-6cba40de5867-trusted-ca\") pod \"image-registry-689468cdb8-lqfkq\" (UID: \"8367995b-3d1a-4762-8445-6cba40de5867\") " pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:21.555313 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.555285    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8367995b-3d1a-4762-8445-6cba40de5867-registry-tls\") pod \"image-registry-689468cdb8-lqfkq\" (UID: \"8367995b-3d1a-4762-8445-6cba40de5867\") " pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:21.555400 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.555349    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8367995b-3d1a-4762-8445-6cba40de5867-installation-pull-secrets\") pod \"image-registry-689468cdb8-lqfkq\" (UID: \"8367995b-3d1a-4762-8445-6cba40de5867\") " pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:21.555400 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.555352    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8367995b-3d1a-4762-8445-6cba40de5867-image-registry-private-configuration\") pod \"image-registry-689468cdb8-lqfkq\" (UID: \"8367995b-3d1a-4762-8445-6cba40de5867\") " pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:21.567099 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.567060    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmwtz\" (UniqueName: \"kubernetes.io/projected/8367995b-3d1a-4762-8445-6cba40de5867-kube-api-access-lmwtz\") pod \"image-registry-689468cdb8-lqfkq\" (UID: \"8367995b-3d1a-4762-8445-6cba40de5867\") " pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:21.570712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.570689    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8367995b-3d1a-4762-8445-6cba40de5867-bound-sa-token\") pod \"image-registry-689468cdb8-lqfkq\" (UID: \"8367995b-3d1a-4762-8445-6cba40de5867\") " pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:21.638864 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.638826    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:21.913549 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:21.913518    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-689468cdb8-lqfkq"]
Sep 18 16:22:21.917352 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:22:21.917328    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8367995b_3d1a_4762_8445_6cba40de5867.slice/crio-089ff2b8505bc3a74ce69f83dc640af1d2918460051386ab532318bb840e14c2 WatchSource:0}: Error finding container 089ff2b8505bc3a74ce69f83dc640af1d2918460051386ab532318bb840e14c2: Status 404 returned error can't find the container with id 089ff2b8505bc3a74ce69f83dc640af1d2918460051386ab532318bb840e14c2
Sep 18 16:22:22.899648 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:22.899606    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-689468cdb8-lqfkq" event={"ID":"8367995b-3d1a-4762-8445-6cba40de5867","Type":"ContainerStarted","Data":"255b949f01af28b2b1f79bc85e173e6decdc3b9c0a0aba2f6c62510cf38c9da3"}
Sep 18 16:22:22.900041 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:22.899668    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-689468cdb8-lqfkq" event={"ID":"8367995b-3d1a-4762-8445-6cba40de5867","Type":"ContainerStarted","Data":"089ff2b8505bc3a74ce69f83dc640af1d2918460051386ab532318bb840e14c2"}
Sep 18 16:22:22.900041 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:22.899724    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:22.923849 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:22.923801    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-689468cdb8-lqfkq" podStartSLOduration=1.9237869669999998 podStartE2EDuration="1.923786967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:22:22.922752446 +0000 UTC m=+381.378205523" watchObservedRunningTime="2026-09-18 16:22:22.923786967 +0000 UTC m=+381.379240044"
Sep 18 16:22:29.312679 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:29.312650    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:22:30.768363 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:30.768320    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lfh2m"]
Sep 18 16:22:30.772298 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:30.772269    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:30.775595 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:30.775420    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 16:22:30.775595 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:30.775457    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 16:22:30.776135 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:30.775940    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4wvnt\""
Sep 18 16:22:30.776293 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:30.776275    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 16:22:30.776855 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:30.776630    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 16:22:30.776855 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:30.776642    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 16:22:30.777142 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:30.777119    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 16:22:30.922067 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:30.922035    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4-node-exporter-wtmp\") pod \"node-exporter-lfh2m\" (UID: \"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4\") " pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:30.922267 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:30.922170    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4-metrics-client-ca\") pod \"node-exporter-lfh2m\" (UID: \"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4\") " pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:30.922267 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:30.922210    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-lfh2m\" (UID: \"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4\") " pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:30.922267 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:30.922238    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z5sm\" (UniqueName: \"kubernetes.io/projected/1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4-kube-api-access-6z5sm\") pod \"node-exporter-lfh2m\" (UID: \"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4\") " pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:30.922267 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:30.922263    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lfh2m\" (UID: \"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4\") " pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:30.922478 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:30.922298    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4-sys\") pod \"node-exporter-lfh2m\" (UID: \"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4\") " pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:30.922478 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:30.922337    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4-node-exporter-textfile\") pod \"node-exporter-lfh2m\" (UID: \"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4\") " pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:30.922478 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:30.922365    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4-node-exporter-tls\") pod \"node-exporter-lfh2m\" (UID: \"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4\") " pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:30.922478 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:30.922391    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4-root\") pod \"node-exporter-lfh2m\" (UID: \"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4\") " pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:31.023896 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:31.023795    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4-node-exporter-wtmp\") pod \"node-exporter-lfh2m\" (UID: \"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4\") " pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:31.023896 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:31.023847    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4-metrics-client-ca\") pod \"node-exporter-lfh2m\" (UID: \"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4\") " pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:31.023896 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:31.023877    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-lfh2m\" (UID: \"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4\") " pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:31.024214 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:31.023904    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z5sm\" (UniqueName: \"kubernetes.io/projected/1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4-kube-api-access-6z5sm\") pod \"node-exporter-lfh2m\" (UID: \"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4\") " pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:31.024214 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:31.023931    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lfh2m\" (UID: \"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4\") " pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:31.024214 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:31.023973    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4-node-exporter-wtmp\") pod \"node-exporter-lfh2m\" (UID: \"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4\") " pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:31.024214 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:31.024047    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4-sys\") pod \"node-exporter-lfh2m\" (UID: \"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4\") " pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:31.024214 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:31.024109    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4-node-exporter-textfile\") pod \"node-exporter-lfh2m\" (UID: \"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4\") " pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:31.024214 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:31.024133    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4-node-exporter-tls\") pod \"node-exporter-lfh2m\" (UID: \"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4\") " pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:31.024214 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:31.024158    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4-root\") pod \"node-exporter-lfh2m\" (UID: \"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4\") " pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:31.024214 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:31.024171    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4-sys\") pod \"node-exporter-lfh2m\" (UID: \"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4\") " pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:31.024611 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:31.024223    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4-root\") pod \"node-exporter-lfh2m\" (UID: \"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4\") " pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:31.024611 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:31.024409    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4-node-exporter-textfile\") pod \"node-exporter-lfh2m\" (UID: \"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4\") " pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:31.024712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:31.024653    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-lfh2m\" (UID: \"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4\") " pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:31.024746 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:31.024711    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4-metrics-client-ca\") pod \"node-exporter-lfh2m\" (UID: \"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4\") " pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:31.026813 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:31.026790    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lfh2m\" (UID: \"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4\") " pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:31.026911 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:31.026896    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4-node-exporter-tls\") pod \"node-exporter-lfh2m\" (UID: \"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4\") " pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:31.032883 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:31.032863    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z5sm\" (UniqueName: \"kubernetes.io/projected/1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4-kube-api-access-6z5sm\") pod \"node-exporter-lfh2m\" (UID: \"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4\") " pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:31.084087 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:31.084049    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lfh2m"
Sep 18 16:22:31.093277 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:22:31.093245    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1017f4cb_796b_4c6c_9a9a_4e96ef08c9f4.slice/crio-e1c94e650febd62f25aa9c098259242c884a5282f114423aba2a611cdfced26e WatchSource:0}: Error finding container e1c94e650febd62f25aa9c098259242c884a5282f114423aba2a611cdfced26e: Status 404 returned error can't find the container with id e1c94e650febd62f25aa9c098259242c884a5282f114423aba2a611cdfced26e
Sep 18 16:22:31.933163 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:31.933126    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lfh2m" event={"ID":"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4","Type":"ContainerStarted","Data":"e1c94e650febd62f25aa9c098259242c884a5282f114423aba2a611cdfced26e"}
Sep 18 16:22:32.937600 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:32.937562    2566 generic.go:356] "Generic (PLEG): container finished" podID="1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4" containerID="43461feaca75eba8ada64cafee491253361afc3814c94078379c643f200ba7d0" exitCode=0
Sep 18 16:22:32.937997 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:32.937867    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lfh2m" event={"ID":"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4","Type":"ContainerDied","Data":"43461feaca75eba8ada64cafee491253361afc3814c94078379c643f200ba7d0"}
Sep 18 16:22:33.780082 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:33.780036    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-656bb475f9-xkpvm"]
Sep 18 16:22:33.782711 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:33.782695    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:33.787684 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:33.787656    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 18 16:22:33.787923 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:33.787906    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 18 16:22:33.787992 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:33.787963    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 18 16:22:33.789624 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:33.789603    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5wrk5\""
Sep 18 16:22:33.789727 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:33.789631    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-a2uistap7duti\""
Sep 18 16:22:33.789727 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:33.789644    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 18 16:22:33.789727 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:33.789633    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 18 16:22:33.798787 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:33.798763    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-656bb475f9-xkpvm"]
Sep 18 16:22:33.943726 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:33.943690    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lfh2m" event={"ID":"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4","Type":"ContainerStarted","Data":"bee71c3ee3888547f32f05a1caf2d8ebafa5f373d0cec1a4ff09b0f684674070"}
Sep 18 16:22:33.944139 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:33.943751    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lfh2m" event={"ID":"1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4","Type":"ContainerStarted","Data":"8fa7f00c71f0581c459ea569eaca82af1d229a25f4349d73c387d603b7b6040e"}
Sep 18 16:22:33.949149 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:33.949128    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/455c4a22-d4ae-4d97-b6be-33f2b1b264d2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-656bb475f9-xkpvm\" (UID: \"455c4a22-d4ae-4d97-b6be-33f2b1b264d2\") " pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:33.949235 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:33.949161    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/455c4a22-d4ae-4d97-b6be-33f2b1b264d2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-656bb475f9-xkpvm\" (UID: \"455c4a22-d4ae-4d97-b6be-33f2b1b264d2\") " pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:33.949235 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:33.949185    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/455c4a22-d4ae-4d97-b6be-33f2b1b264d2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-656bb475f9-xkpvm\" (UID: \"455c4a22-d4ae-4d97-b6be-33f2b1b264d2\") " pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:33.949310 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:33.949245    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2qsc\" (UniqueName: \"kubernetes.io/projected/455c4a22-d4ae-4d97-b6be-33f2b1b264d2-kube-api-access-f2qsc\") pod \"thanos-querier-656bb475f9-xkpvm\" (UID: \"455c4a22-d4ae-4d97-b6be-33f2b1b264d2\") " pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:33.949310 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:33.949263    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/455c4a22-d4ae-4d97-b6be-33f2b1b264d2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-656bb475f9-xkpvm\" (UID: \"455c4a22-d4ae-4d97-b6be-33f2b1b264d2\") " pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:33.949378 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:33.949316    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/455c4a22-d4ae-4d97-b6be-33f2b1b264d2-secret-grpc-tls\") pod \"thanos-querier-656bb475f9-xkpvm\" (UID: \"455c4a22-d4ae-4d97-b6be-33f2b1b264d2\") " pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:33.949378 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:33.949344    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/455c4a22-d4ae-4d97-b6be-33f2b1b264d2-metrics-client-ca\") pod \"thanos-querier-656bb475f9-xkpvm\" (UID: \"455c4a22-d4ae-4d97-b6be-33f2b1b264d2\") " pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:33.949378 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:33.949367    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/455c4a22-d4ae-4d97-b6be-33f2b1b264d2-secret-thanos-querier-tls\") pod \"thanos-querier-656bb475f9-xkpvm\" (UID: \"455c4a22-d4ae-4d97-b6be-33f2b1b264d2\") " pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:33.966460 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:33.966410    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lfh2m" podStartSLOduration=2.946406221 podStartE2EDuration="3.96639508s" totalImagesPullingTime="957.28418ms" totalInitContainerRuntime="62.704679ms" isStatefulPod=true podCreationTimestamp="2026-09-18 16:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:22:33.964974063 +0000 UTC m=+392.420427158" watchObservedRunningTime="2026-09-18 16:22:33.96639508 +0000 UTC m=+392.421848187"
Sep 18 16:22:34.050132 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:34.050021    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/455c4a22-d4ae-4d97-b6be-33f2b1b264d2-secret-grpc-tls\") pod \"thanos-querier-656bb475f9-xkpvm\" (UID: \"455c4a22-d4ae-4d97-b6be-33f2b1b264d2\") " pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:34.050132 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:34.050089    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/455c4a22-d4ae-4d97-b6be-33f2b1b264d2-metrics-client-ca\") pod \"thanos-querier-656bb475f9-xkpvm\" (UID: \"455c4a22-d4ae-4d97-b6be-33f2b1b264d2\") " pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:34.050322 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:34.050208    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/455c4a22-d4ae-4d97-b6be-33f2b1b264d2-secret-thanos-querier-tls\") pod \"thanos-querier-656bb475f9-xkpvm\" (UID: \"455c4a22-d4ae-4d97-b6be-33f2b1b264d2\") " pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:34.050322 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:34.050245    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/455c4a22-d4ae-4d97-b6be-33f2b1b264d2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-656bb475f9-xkpvm\" (UID: \"455c4a22-d4ae-4d97-b6be-33f2b1b264d2\") " pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:34.050322 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:34.050280    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/455c4a22-d4ae-4d97-b6be-33f2b1b264d2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-656bb475f9-xkpvm\" (UID: \"455c4a22-d4ae-4d97-b6be-33f2b1b264d2\") " pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:34.050322 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:34.050306    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/455c4a22-d4ae-4d97-b6be-33f2b1b264d2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-656bb475f9-xkpvm\" (UID: \"455c4a22-d4ae-4d97-b6be-33f2b1b264d2\") " pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:34.050869 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:34.050847    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/455c4a22-d4ae-4d97-b6be-33f2b1b264d2-metrics-client-ca\") pod \"thanos-querier-656bb475f9-xkpvm\" (UID: \"455c4a22-d4ae-4d97-b6be-33f2b1b264d2\") " pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:34.051243 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:34.051218    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2qsc\" (UniqueName: \"kubernetes.io/projected/455c4a22-d4ae-4d97-b6be-33f2b1b264d2-kube-api-access-f2qsc\") pod \"thanos-querier-656bb475f9-xkpvm\" (UID: \"455c4a22-d4ae-4d97-b6be-33f2b1b264d2\") " pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:34.051420 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:34.051401    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/455c4a22-d4ae-4d97-b6be-33f2b1b264d2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-656bb475f9-xkpvm\" (UID: \"455c4a22-d4ae-4d97-b6be-33f2b1b264d2\") " pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:34.052961 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:34.052933    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/455c4a22-d4ae-4d97-b6be-33f2b1b264d2-secret-thanos-querier-tls\") pod \"thanos-querier-656bb475f9-xkpvm\" (UID: \"455c4a22-d4ae-4d97-b6be-33f2b1b264d2\") " pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:34.053226 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:34.053205    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/455c4a22-d4ae-4d97-b6be-33f2b1b264d2-secret-grpc-tls\") pod \"thanos-querier-656bb475f9-xkpvm\" (UID: \"455c4a22-d4ae-4d97-b6be-33f2b1b264d2\") " pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:34.053284 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:34.053264    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/455c4a22-d4ae-4d97-b6be-33f2b1b264d2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-656bb475f9-xkpvm\" (UID: \"455c4a22-d4ae-4d97-b6be-33f2b1b264d2\") " pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:34.053393 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:34.053374    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/455c4a22-d4ae-4d97-b6be-33f2b1b264d2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-656bb475f9-xkpvm\" (UID: \"455c4a22-d4ae-4d97-b6be-33f2b1b264d2\") " pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:34.053440 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:34.053407    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/455c4a22-d4ae-4d97-b6be-33f2b1b264d2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-656bb475f9-xkpvm\" (UID: \"455c4a22-d4ae-4d97-b6be-33f2b1b264d2\") " pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:34.053637 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:34.053621    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/455c4a22-d4ae-4d97-b6be-33f2b1b264d2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-656bb475f9-xkpvm\" (UID: \"455c4a22-d4ae-4d97-b6be-33f2b1b264d2\") " pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:34.059955 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:34.059931    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2qsc\" (UniqueName: \"kubernetes.io/projected/455c4a22-d4ae-4d97-b6be-33f2b1b264d2-kube-api-access-f2qsc\") pod \"thanos-querier-656bb475f9-xkpvm\" (UID: \"455c4a22-d4ae-4d97-b6be-33f2b1b264d2\") " pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:34.091537 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:34.091507    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:34.326166 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:34.326129    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-f6db6c45f-672rr" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerName="registry" containerID="cri-o://cf3668b4ff18405f632e27195524df14feabb8e69230501e20ec28efb76166a7" gracePeriod=30
Sep 18 16:22:34.372140 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:34.372098    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-656bb475f9-xkpvm"]
Sep 18 16:22:34.392046 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:22:34.392006    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod455c4a22_d4ae_4d97_b6be_33f2b1b264d2.slice/crio-e18291917191b81b85d58f9e55ddf3980ee4f40d53fa3938f7407c6f0a10dc7d WatchSource:0}: Error finding container e18291917191b81b85d58f9e55ddf3980ee4f40d53fa3938f7407c6f0a10dc7d: Status 404 returned error can't find the container with id e18291917191b81b85d58f9e55ddf3980ee4f40d53fa3938f7407c6f0a10dc7d
Sep 18 16:22:34.949042 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:34.949008    2566 generic.go:356] "Generic (PLEG): container finished" podID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerID="cf3668b4ff18405f632e27195524df14feabb8e69230501e20ec28efb76166a7" exitCode=0
Sep 18 16:22:34.949496 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:34.949100    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f6db6c45f-672rr" event={"ID":"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d","Type":"ContainerDied","Data":"cf3668b4ff18405f632e27195524df14feabb8e69230501e20ec28efb76166a7"}
Sep 18 16:22:34.949496 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:34.949138    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f6db6c45f-672rr" event={"ID":"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d","Type":"ContainerStarted","Data":"9a5433a99660fd6cab2d75a1d0a87972d229b323536808813b7b8cfcfc9ac982"}
Sep 18 16:22:34.949496 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:34.949164    2566 scope.go:117] "RemoveContainer" containerID="a96c3422aa5307750e844c4e16313c7408654c771bb54c7982eb8c24d5a3575a"
Sep 18 16:22:34.949496 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:34.949227    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:22:34.950584 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:34.950565    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm" event={"ID":"455c4a22-d4ae-4d97-b6be-33f2b1b264d2","Type":"ContainerStarted","Data":"e18291917191b81b85d58f9e55ddf3980ee4f40d53fa3938f7407c6f0a10dc7d"}
Sep 18 16:22:35.468389 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.468350    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b94d5b846-s7qq9"]
Sep 18 16:22:35.471041 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.471017    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:22:35.474145 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.474097    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 16:22:35.474267 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.474097    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 16:22:35.475228 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.475066    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 16:22:35.475228 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.475121    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 16:22:35.475228 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.475153    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 16:22:35.475228 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.475192    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 16:22:35.475550 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.475537    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wz97g\""
Sep 18 16:22:35.476292 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.476265    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 16:22:35.483671 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.483650    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 16:22:35.488279 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.488256    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b94d5b846-s7qq9"]
Sep 18 16:22:35.564761 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.564728    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-service-ca\") pod \"console-b94d5b846-s7qq9\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") " pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:22:35.564936 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.564777    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-oauth-serving-cert\") pod \"console-b94d5b846-s7qq9\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") " pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:22:35.564936 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.564809    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-console-oauth-config\") pod \"console-b94d5b846-s7qq9\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") " pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:22:35.564936 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.564891    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-console-serving-cert\") pod \"console-b94d5b846-s7qq9\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") " pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:22:35.565145 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.564979    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-trusted-ca-bundle\") pod \"console-b94d5b846-s7qq9\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") " pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:22:35.565145 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.565009    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-console-config\") pod \"console-b94d5b846-s7qq9\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") " pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:22:35.565145 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.565038    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9vpc\" (UniqueName: \"kubernetes.io/projected/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-kube-api-access-r9vpc\") pod \"console-b94d5b846-s7qq9\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") " pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:22:35.665728 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.665689    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-console-serving-cert\") pod \"console-b94d5b846-s7qq9\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") " pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:22:35.665915 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.665755    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-trusted-ca-bundle\") pod \"console-b94d5b846-s7qq9\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") " pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:22:35.665915 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.665782    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-console-config\") pod \"console-b94d5b846-s7qq9\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") " pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:22:35.665915 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.665822    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9vpc\" (UniqueName: \"kubernetes.io/projected/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-kube-api-access-r9vpc\") pod \"console-b94d5b846-s7qq9\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") " pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:22:35.665915 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.665897    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-service-ca\") pod \"console-b94d5b846-s7qq9\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") " pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:22:35.666132 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.666026    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-oauth-serving-cert\") pod \"console-b94d5b846-s7qq9\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") " pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:22:35.666132 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.666065    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-console-oauth-config\") pod \"console-b94d5b846-s7qq9\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") " pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:22:35.666838 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.666809    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-console-config\") pod \"console-b94d5b846-s7qq9\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") " pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:22:35.666838 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.666826    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-oauth-serving-cert\") pod \"console-b94d5b846-s7qq9\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") " pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:22:35.667427 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.667403    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-service-ca\") pod \"console-b94d5b846-s7qq9\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") " pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:22:35.667587 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.667564    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-trusted-ca-bundle\") pod \"console-b94d5b846-s7qq9\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") " pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:22:35.668791 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.668772    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-console-oauth-config\") pod \"console-b94d5b846-s7qq9\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") " pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:22:35.668970 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.668947    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-console-serving-cert\") pod \"console-b94d5b846-s7qq9\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") " pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:22:35.676880 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.676853    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9vpc\" (UniqueName: \"kubernetes.io/projected/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-kube-api-access-r9vpc\") pod \"console-b94d5b846-s7qq9\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") " pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:22:35.783382 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:35.783296    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:22:36.062429 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:36.062395    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b94d5b846-s7qq9"]
Sep 18 16:22:36.064983 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:22:36.064958    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ce2ea5_46c5_4df2_877b_4f78d16cbf36.slice/crio-d1a77b6809681c46dd66fd199c1336121993b0b19b6bf4386d93c3307792497e WatchSource:0}: Error finding container d1a77b6809681c46dd66fd199c1336121993b0b19b6bf4386d93c3307792497e: Status 404 returned error can't find the container with id d1a77b6809681c46dd66fd199c1336121993b0b19b6bf4386d93c3307792497e
Sep 18 16:22:36.978459 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:36.976787    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:22:36.984962 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:36.984831    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm" event={"ID":"455c4a22-d4ae-4d97-b6be-33f2b1b264d2","Type":"ContainerStarted","Data":"6222c47e1d82891eb61fb0823fde9163b97bda15cb984f7ed45b37c8cfa4be84"}
Sep 18 16:22:36.984962 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:36.984900    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm" event={"ID":"455c4a22-d4ae-4d97-b6be-33f2b1b264d2","Type":"ContainerStarted","Data":"bc3154648c372409a63c53e4bcb95ea5458792ef91f47562db0103efa6e89cd8"}
Sep 18 16:22:36.984962 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:36.984941    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b94d5b846-s7qq9" event={"ID":"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36","Type":"ContainerStarted","Data":"d1a77b6809681c46dd66fd199c1336121993b0b19b6bf4386d93c3307792497e"}
Sep 18 16:22:36.985330 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:36.985063    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:36.989679 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:36.988040    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 16:22:36.989679 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:36.988397    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 16:22:36.989679 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:36.988671    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 16:22:36.989679 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:36.988958    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 16:22:36.989679 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:36.989155    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 16:22:36.989679 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:36.989365    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 16:22:36.989679 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:36.989539    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 16:22:36.990457 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:36.990199    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 16:22:36.990457 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:36.990322    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 16:22:36.990605 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:36.990551    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 16:22:36.991702 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:36.990720    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 16:22:36.991702 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:36.990784    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 16:22:36.991702 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:36.990988    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3ri1d1dcnr9dk\""
Sep 18 16:22:36.993922 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:36.993578    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-64ckn\""
Sep 18 16:22:36.993922 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:36.993758    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 16:22:36.999012 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:36.998245    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 16:22:36.999012 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:36.999004    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 16:22:37.000325 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.000290    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:22:37.081254 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.080200    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.081254 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.080246    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.081254 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.080278    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.081254 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.080313    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-config\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.081254 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.080340    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.081254 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.080457    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.081254 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.080548    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.081254 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.080576    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.081254 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.080603    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-web-config\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.081254 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.080633    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.081254 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.080664    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.081254 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.080701    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.081254 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.080728    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.081254 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.080755    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-config-out\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.081254 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.080794    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.081254 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.080837    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.082581 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.080866    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.082581 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.080902    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rplnl\" (UniqueName: \"kubernetes.io/projected/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-kube-api-access-rplnl\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.082581 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.080933    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.082581 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.080963    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.182568 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.182521    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.182568 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.182570    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.182817 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.182608    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rplnl\" (UniqueName: \"kubernetes.io/projected/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-kube-api-access-rplnl\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.182817 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.182637    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.182817 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.182668    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.182817 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.182701    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.182817 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.182725    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.182817 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.182755    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.182817 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.182795    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-config\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.183184 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.182820    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.183184 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.182842    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.183184 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.182886    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.183184 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.182912    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.183184 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.182937    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-web-config\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.183184 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.182965    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.183184 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.183001    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.183184 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.183038    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.183184 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.183066    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.183184 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.183120    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-config-out\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.183184 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.183163    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.183696 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.183668    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.185605 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.184619    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.185605 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.185267    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.188662 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.188631    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.190214 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.189803    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.190214 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.189838    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.190214 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.190105    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.190214 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.190108    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-web-config\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.190678 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.190651    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.191371 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.191332    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.195416 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.195052    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.195416 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.195375    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.195566 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.195500    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.196235 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.195618    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.196235 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.195728    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-config\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.196235 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.195745    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.196235 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.195992    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.197623 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.197584    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.198533 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.198509    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rplnl\" (UniqueName: \"kubernetes.io/projected/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-kube-api-access-rplnl\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.199019 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.198999    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-config-out\") pod \"prometheus-k8s-0\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.309368 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.309333    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:37.610347 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.610237    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:22:37.619252 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:22:37.619195    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd7094e_fea8_43b7_a6e8_d4dc35d17908.slice/crio-3d04e588349f621358122961af19f3aff37490ceac8bf6f2081455cc901d0df5 WatchSource:0}: Error finding container 3d04e588349f621358122961af19f3aff37490ceac8bf6f2081455cc901d0df5: Status 404 returned error can't find the container with id 3d04e588349f621358122961af19f3aff37490ceac8bf6f2081455cc901d0df5
Sep 18 16:22:37.991714 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.991573    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6cd7094e-fea8-43b7-a6e8-d4dc35d17908","Type":"ContainerStarted","Data":"3d04e588349f621358122961af19f3aff37490ceac8bf6f2081455cc901d0df5"}
Sep 18 16:22:37.994395 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:37.994363    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm" event={"ID":"455c4a22-d4ae-4d97-b6be-33f2b1b264d2","Type":"ContainerStarted","Data":"4b4cb45c2afb57f005181edba58f04e1115fabef148ddfeabed2e87d978595c6"}
Sep 18 16:22:41.008975 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:41.008937    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b94d5b846-s7qq9" event={"ID":"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36","Type":"ContainerStarted","Data":"0d87e371184a8e2e4afeccda8fb0a196457d247980da6339727d83b1b76feceb"}
Sep 18 16:22:41.010766 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:41.010740    2566 generic.go:356] "Generic (PLEG): container finished" podID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerID="1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c" exitCode=0
Sep 18 16:22:41.010894 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:41.010827    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6cd7094e-fea8-43b7-a6e8-d4dc35d17908","Type":"ContainerDied","Data":"1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c"}
Sep 18 16:22:41.013706 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:41.013686    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm" event={"ID":"455c4a22-d4ae-4d97-b6be-33f2b1b264d2","Type":"ContainerStarted","Data":"93ce5334b693bb222bfbe755916b2c5d765cd7d8eba291155a5dd0759b6b7dd6"}
Sep 18 16:22:41.039695 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:41.039636    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b94d5b846-s7qq9" podStartSLOduration=1.273083843 podStartE2EDuration="6.039611733s" totalImagesPullingTime="4.76652789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:22:41.038389531 +0000 UTC m=+399.493842609" watchObservedRunningTime="2026-09-18 16:22:41.039611733 +0000 UTC m=+399.495064811"
Sep 18 16:22:42.021778 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:42.021738    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm" event={"ID":"455c4a22-d4ae-4d97-b6be-33f2b1b264d2","Type":"ContainerStarted","Data":"51aeeaa5c42297d4969fc3245ca003719a0a4eac851d49af5fd9ba420a2107c1"}
Sep 18 16:22:42.022246 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:42.021822    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm" event={"ID":"455c4a22-d4ae-4d97-b6be-33f2b1b264d2","Type":"ContainerStarted","Data":"cdd2f4c1fb90281cb5c393571a76932c263fd1ce402c647f0907c3539e5b8512"}
Sep 18 16:22:42.053510 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:42.053454    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm" podStartSLOduration=2.947321981 podStartE2EDuration="9.053426487s" totalImagesPullingTime="6.106104506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:22:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:22:42.051667124 +0000 UTC m=+400.507120241" watchObservedRunningTime="2026-09-18 16:22:42.053426487 +0000 UTC m=+400.508879564"
Sep 18 16:22:43.025941 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:43.025903    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:43.031381 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:43.031361    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-656bb475f9-xkpvm"
Sep 18 16:22:43.908986 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:43.908955    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-689468cdb8-lqfkq"
Sep 18 16:22:45.037491 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:45.037458    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6cd7094e-fea8-43b7-a6e8-d4dc35d17908","Type":"ContainerStarted","Data":"391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018"}
Sep 18 16:22:45.037873 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:45.037526    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6cd7094e-fea8-43b7-a6e8-d4dc35d17908","Type":"ContainerStarted","Data":"c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf"}
Sep 18 16:22:45.037873 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:45.037560    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6cd7094e-fea8-43b7-a6e8-d4dc35d17908","Type":"ContainerStarted","Data":"01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28"}
Sep 18 16:22:45.037873 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:45.037590    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6cd7094e-fea8-43b7-a6e8-d4dc35d17908","Type":"ContainerStarted","Data":"ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665"}
Sep 18 16:22:45.037873 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:45.037613    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6cd7094e-fea8-43b7-a6e8-d4dc35d17908","Type":"ContainerStarted","Data":"5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48"}
Sep 18 16:22:45.037873 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:45.037646    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6cd7094e-fea8-43b7-a6e8-d4dc35d17908","Type":"ContainerStarted","Data":"d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735"}
Sep 18 16:22:45.065942 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:45.065875    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.568552415 podStartE2EDuration="9.065852439s" totalImagesPullingTime="6.465171564s" totalInitContainerRuntime="32.12846ms" isStatefulPod=false podCreationTimestamp="2026-09-18 16:22:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:22:45.064144609 +0000 UTC m=+403.519597687" watchObservedRunningTime="2026-09-18 16:22:45.065852439 +0000 UTC m=+403.521305517"
Sep 18 16:22:45.784257 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:45.784213    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:22:45.784398 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:45.784282    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:22:45.789325 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:45.789299    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:22:46.045350 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:46.045255    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:22:47.309876 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:47.309838    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:22:54.955852 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:54.955822    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:22:59.968016 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:22:59.967969    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-f6db6c45f-672rr" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerName="registry" containerID="cri-o://9a5433a99660fd6cab2d75a1d0a87972d229b323536808813b7b8cfcfc9ac982" gracePeriod=30
Sep 18 16:23:00.099419 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.099384    2566 generic.go:356] "Generic (PLEG): container finished" podID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerID="9a5433a99660fd6cab2d75a1d0a87972d229b323536808813b7b8cfcfc9ac982" exitCode=0
Sep 18 16:23:00.099587 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.099429    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f6db6c45f-672rr" event={"ID":"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d","Type":"ContainerDied","Data":"9a5433a99660fd6cab2d75a1d0a87972d229b323536808813b7b8cfcfc9ac982"}
Sep 18 16:23:00.099587 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.099477    2566 scope.go:117] "RemoveContainer" containerID="cf3668b4ff18405f632e27195524df14feabb8e69230501e20ec28efb76166a7"
Sep 18 16:23:00.218426 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.218370    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:23:00.280132 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.280098    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-registry-tls\") pod \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") "
Sep 18 16:23:00.280316 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.280142    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-registry-certificates\") pod \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") "
Sep 18 16:23:00.280316 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.280173    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-trusted-ca\") pod \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") "
Sep 18 16:23:00.280316 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.280216    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-bound-sa-token\") pod \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") "
Sep 18 16:23:00.280316 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.280241    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-ca-trust-extracted\") pod \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") "
Sep 18 16:23:00.280316 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.280279    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6twk\" (UniqueName: \"kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-kube-api-access-x6twk\") pod \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") "
Sep 18 16:23:00.280565 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.280337    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-installation-pull-secrets\") pod \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") "
Sep 18 16:23:00.280565 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.280387    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-image-registry-private-configuration\") pod \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\" (UID: \"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d\") "
Sep 18 16:23:00.280692 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.280664    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" (UID: "5ada52f1-14ec-41fe-a161-7e8b0fa6a12d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:23:00.281150 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.281122    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" (UID: "5ada52f1-14ec-41fe-a161-7e8b0fa6a12d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:23:00.282926 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.282812    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" (UID: "5ada52f1-14ec-41fe-a161-7e8b0fa6a12d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:23:00.282926 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.282870    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-kube-api-access-x6twk" (OuterVolumeSpecName: "kube-api-access-x6twk") pod "5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" (UID: "5ada52f1-14ec-41fe-a161-7e8b0fa6a12d"). InnerVolumeSpecName "kube-api-access-x6twk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:23:00.282926 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.282887    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" (UID: "5ada52f1-14ec-41fe-a161-7e8b0fa6a12d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:23:00.283129 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.282941    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" (UID: "5ada52f1-14ec-41fe-a161-7e8b0fa6a12d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:23:00.283129 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.282996    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" (UID: "5ada52f1-14ec-41fe-a161-7e8b0fa6a12d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:23:00.288922 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.288897    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" (UID: "5ada52f1-14ec-41fe-a161-7e8b0fa6a12d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:23:00.381568 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.381522    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-installation-pull-secrets\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:00.381568 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.381563    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-image-registry-private-configuration\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:00.381568 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.381573    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-registry-tls\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:00.381761 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.381584    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-registry-certificates\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:00.381761 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.381593    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-trusted-ca\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:00.381761 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.381600    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-bound-sa-token\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:00.381761 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.381608    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-ca-trust-extracted\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:00.381761 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:00.381615    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6twk\" (UniqueName: \"kubernetes.io/projected/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d-kube-api-access-x6twk\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:01.104610 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:01.104573    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f6db6c45f-672rr" event={"ID":"5ada52f1-14ec-41fe-a161-7e8b0fa6a12d","Type":"ContainerDied","Data":"a6a2e439cd43add9405c0fa8de6538874db5fadbcad15658d993def9529b5cd5"}
Sep 18 16:23:01.104998 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:01.104633    2566 scope.go:117] "RemoveContainer" containerID="9a5433a99660fd6cab2d75a1d0a87972d229b323536808813b7b8cfcfc9ac982"
Sep 18 16:23:01.104998 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:01.104585    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f6db6c45f-672rr"
Sep 18 16:23:01.126730 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:01.126704    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f6db6c45f-672rr"]
Sep 18 16:23:01.132124 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:01.132100    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-f6db6c45f-672rr"]
Sep 18 16:23:01.686708 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:01.686669    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" path="/var/lib/kubelet/pods/5ada52f1-14ec-41fe-a161-7e8b0fa6a12d/volumes"
Sep 18 16:23:37.310041 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:37.309993    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:37.329342 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:37.329315    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:38.258281 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:38.258255    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:55.365873 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.365836    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:23:55.366413 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.366176    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerName="prometheus" containerID="cri-o://d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735" gracePeriod=600
Sep 18 16:23:55.366413 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.366199    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerName="kube-rbac-proxy" containerID="cri-o://c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf" gracePeriod=600
Sep 18 16:23:55.366413 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.366244    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerName="kube-rbac-proxy-thanos" containerID="cri-o://391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018" gracePeriod=600
Sep 18 16:23:55.366413 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.366266    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerName="config-reloader" containerID="cri-o://5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48" gracePeriod=600
Sep 18 16:23:55.366413 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.366219    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerName="kube-rbac-proxy-web" containerID="cri-o://01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28" gracePeriod=600
Sep 18 16:23:55.366413 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.366252    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerName="thanos-sidecar" containerID="cri-o://ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665" gracePeriod=600
Sep 18 16:23:55.610293 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.610271    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:55.648471 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.648388    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-config\") pod \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") "
Sep 18 16:23:55.648471 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.648433    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") "
Sep 18 16:23:55.648471 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.648460    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-thanos-prometheus-http-client-file\") pod \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") "
Sep 18 16:23:55.648720 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.648488    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-kube-rbac-proxy\") pod \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") "
Sep 18 16:23:55.648720 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.648508    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-k8s-db\") pod \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") "
Sep 18 16:23:55.648720 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.648527    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-k8s-rulefiles-2\") pod \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") "
Sep 18 16:23:55.648720 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.648549    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-metrics-client-certs\") pod \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") "
Sep 18 16:23:55.648720 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.648574    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-k8s-rulefiles-0\") pod \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") "
Sep 18 16:23:55.648720 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.648616    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-trusted-ca-bundle\") pod \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") "
Sep 18 16:23:55.648720 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.648661    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-configmap-serving-certs-ca-bundle\") pod \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") "
Sep 18 16:23:55.648720 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.648684    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-web-config\") pod \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") "
Sep 18 16:23:55.648720 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.648717    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-grpc-tls\") pod \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") "
Sep 18 16:23:55.649196 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.648748    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-configmap-metrics-client-ca\") pod \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") "
Sep 18 16:23:55.649196 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.648819    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-k8s-rulefiles-1\") pod \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") "
Sep 18 16:23:55.649196 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.648847    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-config-out\") pod \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") "
Sep 18 16:23:55.649196 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.648885    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") "
Sep 18 16:23:55.649196 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.648912    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-prometheus-k8s-tls\") pod \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") "
Sep 18 16:23:55.649196 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.648937    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rplnl\" (UniqueName: \"kubernetes.io/projected/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-kube-api-access-rplnl\") pod \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") "
Sep 18 16:23:55.649196 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.648968    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-configmap-kubelet-serving-ca-bundle\") pod \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") "
Sep 18 16:23:55.649196 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.649002    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-tls-assets\") pod \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\" (UID: \"6cd7094e-fea8-43b7-a6e8-d4dc35d17908\") "
Sep 18 16:23:55.650946 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.649893    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "6cd7094e-fea8-43b7-a6e8-d4dc35d17908" (UID: "6cd7094e-fea8-43b7-a6e8-d4dc35d17908"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:23:55.650946 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.649907    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "6cd7094e-fea8-43b7-a6e8-d4dc35d17908" (UID: "6cd7094e-fea8-43b7-a6e8-d4dc35d17908"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:23:55.650946 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.649998    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "6cd7094e-fea8-43b7-a6e8-d4dc35d17908" (UID: "6cd7094e-fea8-43b7-a6e8-d4dc35d17908"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:23:55.650946 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.650425    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:55.650946 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.650449    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:55.650946 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.650465    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:55.650946 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.650544    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "6cd7094e-fea8-43b7-a6e8-d4dc35d17908" (UID: "6cd7094e-fea8-43b7-a6e8-d4dc35d17908"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:23:55.650946 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.650912    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "6cd7094e-fea8-43b7-a6e8-d4dc35d17908" (UID: "6cd7094e-fea8-43b7-a6e8-d4dc35d17908"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:23:55.651742 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.651686    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "6cd7094e-fea8-43b7-a6e8-d4dc35d17908" (UID: "6cd7094e-fea8-43b7-a6e8-d4dc35d17908"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:23:55.651742 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.651680    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "6cd7094e-fea8-43b7-a6e8-d4dc35d17908" (UID: "6cd7094e-fea8-43b7-a6e8-d4dc35d17908"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:23:55.652019 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.651998    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "6cd7094e-fea8-43b7-a6e8-d4dc35d17908" (UID: "6cd7094e-fea8-43b7-a6e8-d4dc35d17908"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:23:55.652213 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.652185    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "6cd7094e-fea8-43b7-a6e8-d4dc35d17908" (UID: "6cd7094e-fea8-43b7-a6e8-d4dc35d17908"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:23:55.652384 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.652359    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "6cd7094e-fea8-43b7-a6e8-d4dc35d17908" (UID: "6cd7094e-fea8-43b7-a6e8-d4dc35d17908"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:23:55.652634 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.652606    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "6cd7094e-fea8-43b7-a6e8-d4dc35d17908" (UID: "6cd7094e-fea8-43b7-a6e8-d4dc35d17908"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:23:55.653445 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.653419    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-config-out" (OuterVolumeSpecName: "config-out") pod "6cd7094e-fea8-43b7-a6e8-d4dc35d17908" (UID: "6cd7094e-fea8-43b7-a6e8-d4dc35d17908"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:23:55.653538 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.653454    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-kube-api-access-rplnl" (OuterVolumeSpecName: "kube-api-access-rplnl") pod "6cd7094e-fea8-43b7-a6e8-d4dc35d17908" (UID: "6cd7094e-fea8-43b7-a6e8-d4dc35d17908"). InnerVolumeSpecName "kube-api-access-rplnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:23:55.653889 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.653860    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "6cd7094e-fea8-43b7-a6e8-d4dc35d17908" (UID: "6cd7094e-fea8-43b7-a6e8-d4dc35d17908"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:23:55.654393 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.654359    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-config" (OuterVolumeSpecName: "config") pod "6cd7094e-fea8-43b7-a6e8-d4dc35d17908" (UID: "6cd7094e-fea8-43b7-a6e8-d4dc35d17908"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:23:55.654393 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.654373    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6cd7094e-fea8-43b7-a6e8-d4dc35d17908" (UID: "6cd7094e-fea8-43b7-a6e8-d4dc35d17908"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:23:55.655125 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.655054    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "6cd7094e-fea8-43b7-a6e8-d4dc35d17908" (UID: "6cd7094e-fea8-43b7-a6e8-d4dc35d17908"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:23:55.655125 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.655066    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "6cd7094e-fea8-43b7-a6e8-d4dc35d17908" (UID: "6cd7094e-fea8-43b7-a6e8-d4dc35d17908"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:23:55.655391 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.655373    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "6cd7094e-fea8-43b7-a6e8-d4dc35d17908" (UID: "6cd7094e-fea8-43b7-a6e8-d4dc35d17908"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:23:55.667431 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.667404    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-web-config" (OuterVolumeSpecName: "web-config") pod "6cd7094e-fea8-43b7-a6e8-d4dc35d17908" (UID: "6cd7094e-fea8-43b7-a6e8-d4dc35d17908"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:23:55.751250 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.751214    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:55.751250 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.751243    2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-tls-assets\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:55.751250 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.751253    2566 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-config\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:55.751495 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.751261    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:55.751495 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.751271    2566 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:55.751495 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.751281    2566 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-kube-rbac-proxy\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:55.751495 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.751289    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-k8s-db\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:55.751495 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.751298    2566 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-metrics-client-certs\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:55.751495 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.751307    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:55.751495 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.751317    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:55.751495 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.751326    2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-web-config\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:55.751495 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.751334    2566 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-grpc-tls\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:55.751495 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.751342    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-configmap-metrics-client-ca\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:55.751495 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.751352    2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-config-out\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:55.751495 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.751360    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:55.751495 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.751368    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:55.751495 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:55.751376    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rplnl\" (UniqueName: \"kubernetes.io/projected/6cd7094e-fea8-43b7-a6e8-d4dc35d17908-kube-api-access-rplnl\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:23:56.308129 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.308092    2566 generic.go:356] "Generic (PLEG): container finished" podID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerID="391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018" exitCode=0
Sep 18 16:23:56.308129 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.308122    2566 generic.go:356] "Generic (PLEG): container finished" podID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerID="c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf" exitCode=0
Sep 18 16:23:56.308129 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.308130    2566 generic.go:356] "Generic (PLEG): container finished" podID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerID="01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28" exitCode=0
Sep 18 16:23:56.308354 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.308140    2566 generic.go:356] "Generic (PLEG): container finished" podID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerID="ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665" exitCode=0
Sep 18 16:23:56.308354 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.308148    2566 generic.go:356] "Generic (PLEG): container finished" podID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerID="5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48" exitCode=0
Sep 18 16:23:56.308354 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.308157    2566 generic.go:356] "Generic (PLEG): container finished" podID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerID="d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735" exitCode=0
Sep 18 16:23:56.308354 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.308170    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6cd7094e-fea8-43b7-a6e8-d4dc35d17908","Type":"ContainerDied","Data":"391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018"}
Sep 18 16:23:56.308354 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.308185    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.308354 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.308242    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6cd7094e-fea8-43b7-a6e8-d4dc35d17908","Type":"ContainerDied","Data":"c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf"}
Sep 18 16:23:56.308354 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.308270    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6cd7094e-fea8-43b7-a6e8-d4dc35d17908","Type":"ContainerDied","Data":"01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28"}
Sep 18 16:23:56.308354 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.308296    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6cd7094e-fea8-43b7-a6e8-d4dc35d17908","Type":"ContainerDied","Data":"ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665"}
Sep 18 16:23:56.308354 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.308307    2566 scope.go:117] "RemoveContainer" containerID="391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018"
Sep 18 16:23:56.308354 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.308319    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6cd7094e-fea8-43b7-a6e8-d4dc35d17908","Type":"ContainerDied","Data":"5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48"}
Sep 18 16:23:56.308354 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.308344    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6cd7094e-fea8-43b7-a6e8-d4dc35d17908","Type":"ContainerDied","Data":"d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735"}
Sep 18 16:23:56.308784 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.308369    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6cd7094e-fea8-43b7-a6e8-d4dc35d17908","Type":"ContainerDied","Data":"3d04e588349f621358122961af19f3aff37490ceac8bf6f2081455cc901d0df5"}
Sep 18 16:23:56.316020 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.316004    2566 scope.go:117] "RemoveContainer" containerID="c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf"
Sep 18 16:23:56.322909 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.322891    2566 scope.go:117] "RemoveContainer" containerID="01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28"
Sep 18 16:23:56.329280 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.329265    2566 scope.go:117] "RemoveContainer" containerID="ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665"
Sep 18 16:23:56.332607 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.332587    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:23:56.336481 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.336462    2566 scope.go:117] "RemoveContainer" containerID="5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48"
Sep 18 16:23:56.337791 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.337773    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:23:56.342667 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.342650    2566 scope.go:117] "RemoveContainer" containerID="d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735"
Sep 18 16:23:56.349277 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.349260    2566 scope.go:117] "RemoveContainer" containerID="1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c"
Sep 18 16:23:56.355290 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.355275    2566 scope.go:117] "RemoveContainer" containerID="391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018"
Sep 18 16:23:56.355575 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:23:56.355549    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018\": container with ID starting with 391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018 not found: ID does not exist" containerID="391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018"
Sep 18 16:23:56.355619 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.355583    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018"} err="failed to get container status \"391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018\": rpc error: code = NotFound desc = could not find container \"391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018\": container with ID starting with 391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018 not found: ID does not exist"
Sep 18 16:23:56.355619 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.355601    2566 scope.go:117] "RemoveContainer" containerID="c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf"
Sep 18 16:23:56.355853 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:23:56.355836    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf\": container with ID starting with c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf not found: ID does not exist" containerID="c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf"
Sep 18 16:23:56.355905 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.355860    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf"} err="failed to get container status \"c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf\": rpc error: code = NotFound desc = could not find container \"c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf\": container with ID starting with c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf not found: ID does not exist"
Sep 18 16:23:56.355905 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.355880    2566 scope.go:117] "RemoveContainer" containerID="01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28"
Sep 18 16:23:56.356119 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:23:56.356100    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28\": container with ID starting with 01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28 not found: ID does not exist" containerID="01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28"
Sep 18 16:23:56.356162 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.356124    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28"} err="failed to get container status \"01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28\": rpc error: code = NotFound desc = could not find container \"01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28\": container with ID starting with 01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28 not found: ID does not exist"
Sep 18 16:23:56.356162 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.356140    2566 scope.go:117] "RemoveContainer" containerID="ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665"
Sep 18 16:23:56.356367 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:23:56.356349    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665\": container with ID starting with ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665 not found: ID does not exist" containerID="ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665"
Sep 18 16:23:56.356406 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.356371    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665"} err="failed to get container status \"ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665\": rpc error: code = NotFound desc = could not find container \"ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665\": container with ID starting with ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665 not found: ID does not exist"
Sep 18 16:23:56.356406 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.356385    2566 scope.go:117] "RemoveContainer" containerID="5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48"
Sep 18 16:23:56.356612 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:23:56.356598    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48\": container with ID starting with 5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48 not found: ID does not exist" containerID="5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48"
Sep 18 16:23:56.356666 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.356614    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48"} err="failed to get container status \"5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48\": rpc error: code = NotFound desc = could not find container \"5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48\": container with ID starting with 5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48 not found: ID does not exist"
Sep 18 16:23:56.356666 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.356626    2566 scope.go:117] "RemoveContainer" containerID="d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735"
Sep 18 16:23:56.356822 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:23:56.356808    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735\": container with ID starting with d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735 not found: ID does not exist" containerID="d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735"
Sep 18 16:23:56.356860 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.356825    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735"} err="failed to get container status \"d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735\": rpc error: code = NotFound desc = could not find container \"d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735\": container with ID starting with d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735 not found: ID does not exist"
Sep 18 16:23:56.356860 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.356836    2566 scope.go:117] "RemoveContainer" containerID="1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c"
Sep 18 16:23:56.357053 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:23:56.357035    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c\": container with ID starting with 1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c not found: ID does not exist" containerID="1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c"
Sep 18 16:23:56.357126 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.357057    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c"} err="failed to get container status \"1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c\": rpc error: code = NotFound desc = could not find container \"1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c\": container with ID starting with 1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c not found: ID does not exist"
Sep 18 16:23:56.357126 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.357071    2566 scope.go:117] "RemoveContainer" containerID="391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018"
Sep 18 16:23:56.357313 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.357298    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018"} err="failed to get container status \"391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018\": rpc error: code = NotFound desc = could not find container \"391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018\": container with ID starting with 391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018 not found: ID does not exist"
Sep 18 16:23:56.357313 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.357312    2566 scope.go:117] "RemoveContainer" containerID="c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf"
Sep 18 16:23:56.357528 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.357507    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf"} err="failed to get container status \"c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf\": rpc error: code = NotFound desc = could not find container \"c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf\": container with ID starting with c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf not found: ID does not exist"
Sep 18 16:23:56.357571 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.357528    2566 scope.go:117] "RemoveContainer" containerID="01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28"
Sep 18 16:23:56.357736 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.357721    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28"} err="failed to get container status \"01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28\": rpc error: code = NotFound desc = could not find container \"01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28\": container with ID starting with 01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28 not found: ID does not exist"
Sep 18 16:23:56.357736 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.357735    2566 scope.go:117] "RemoveContainer" containerID="ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665"
Sep 18 16:23:56.357937 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.357921    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665"} err="failed to get container status \"ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665\": rpc error: code = NotFound desc = could not find container \"ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665\": container with ID starting with ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665 not found: ID does not exist"
Sep 18 16:23:56.357975 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.357937    2566 scope.go:117] "RemoveContainer" containerID="5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48"
Sep 18 16:23:56.358157 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.358142    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48"} err="failed to get container status \"5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48\": rpc error: code = NotFound desc = could not find container \"5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48\": container with ID starting with 5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48 not found: ID does not exist"
Sep 18 16:23:56.358157 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.358156    2566 scope.go:117] "RemoveContainer" containerID="d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735"
Sep 18 16:23:56.358370 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.358355    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735"} err="failed to get container status \"d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735\": rpc error: code = NotFound desc = could not find container \"d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735\": container with ID starting with d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735 not found: ID does not exist"
Sep 18 16:23:56.358370 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.358370    2566 scope.go:117] "RemoveContainer" containerID="1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c"
Sep 18 16:23:56.358582 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.358564    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c"} err="failed to get container status \"1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c\": rpc error: code = NotFound desc = could not find container \"1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c\": container with ID starting with 1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c not found: ID does not exist"
Sep 18 16:23:56.358638 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.358581    2566 scope.go:117] "RemoveContainer" containerID="391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018"
Sep 18 16:23:56.358777 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.358761    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018"} err="failed to get container status \"391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018\": rpc error: code = NotFound desc = could not find container \"391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018\": container with ID starting with 391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018 not found: ID does not exist"
Sep 18 16:23:56.358818 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.358777    2566 scope.go:117] "RemoveContainer" containerID="c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf"
Sep 18 16:23:56.358953 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.358937    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf"} err="failed to get container status \"c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf\": rpc error: code = NotFound desc = could not find container \"c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf\": container with ID starting with c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf not found: ID does not exist"
Sep 18 16:23:56.358953 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.358951    2566 scope.go:117] "RemoveContainer" containerID="01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28"
Sep 18 16:23:56.359188 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.359171    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28"} err="failed to get container status \"01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28\": rpc error: code = NotFound desc = could not find container \"01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28\": container with ID starting with 01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28 not found: ID does not exist"
Sep 18 16:23:56.359188 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.359187    2566 scope.go:117] "RemoveContainer" containerID="ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665"
Sep 18 16:23:56.359412 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.359397    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665"} err="failed to get container status \"ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665\": rpc error: code = NotFound desc = could not find container \"ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665\": container with ID starting with ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665 not found: ID does not exist"
Sep 18 16:23:56.359450 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.359411    2566 scope.go:117] "RemoveContainer" containerID="5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48"
Sep 18 16:23:56.359640 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.359622    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48"} err="failed to get container status \"5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48\": rpc error: code = NotFound desc = could not find container \"5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48\": container with ID starting with 5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48 not found: ID does not exist"
Sep 18 16:23:56.359703 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.359643    2566 scope.go:117] "RemoveContainer" containerID="d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735"
Sep 18 16:23:56.359856 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.359839    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735"} err="failed to get container status \"d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735\": rpc error: code = NotFound desc = could not find container \"d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735\": container with ID starting with d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735 not found: ID does not exist"
Sep 18 16:23:56.359910 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.359856    2566 scope.go:117] "RemoveContainer" containerID="1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c"
Sep 18 16:23:56.360088 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.360060    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c"} err="failed to get container status \"1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c\": rpc error: code = NotFound desc = could not find container \"1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c\": container with ID starting with 1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c not found: ID does not exist"
Sep 18 16:23:56.360159 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.360107    2566 scope.go:117] "RemoveContainer" containerID="391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018"
Sep 18 16:23:56.360333 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.360316    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018"} err="failed to get container status \"391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018\": rpc error: code = NotFound desc = could not find container \"391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018\": container with ID starting with 391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018 not found: ID does not exist"
Sep 18 16:23:56.360379 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.360333    2566 scope.go:117] "RemoveContainer" containerID="c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf"
Sep 18 16:23:56.360534 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.360518    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf"} err="failed to get container status \"c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf\": rpc error: code = NotFound desc = could not find container \"c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf\": container with ID starting with c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf not found: ID does not exist"
Sep 18 16:23:56.360579 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.360536    2566 scope.go:117] "RemoveContainer" containerID="01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28"
Sep 18 16:23:56.360769 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.360752    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28"} err="failed to get container status \"01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28\": rpc error: code = NotFound desc = could not find container \"01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28\": container with ID starting with 01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28 not found: ID does not exist"
Sep 18 16:23:56.360809 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.360769    2566 scope.go:117] "RemoveContainer" containerID="ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665"
Sep 18 16:23:56.361000 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.360985    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665"} err="failed to get container status \"ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665\": rpc error: code = NotFound desc = could not find container \"ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665\": container with ID starting with ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665 not found: ID does not exist"
Sep 18 16:23:56.361000 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.360999    2566 scope.go:117] "RemoveContainer" containerID="5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48"
Sep 18 16:23:56.361217 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.361200    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48"} err="failed to get container status \"5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48\": rpc error: code = NotFound desc = could not find container \"5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48\": container with ID starting with 5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48 not found: ID does not exist"
Sep 18 16:23:56.361268 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.361218    2566 scope.go:117] "RemoveContainer" containerID="d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735"
Sep 18 16:23:56.361443 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.361425    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735"} err="failed to get container status \"d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735\": rpc error: code = NotFound desc = could not find container \"d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735\": container with ID starting with d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735 not found: ID does not exist"
Sep 18 16:23:56.361493 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.361444    2566 scope.go:117] "RemoveContainer" containerID="1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c"
Sep 18 16:23:56.361680 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.361634    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c"} err="failed to get container status \"1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c\": rpc error: code = NotFound desc = could not find container \"1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c\": container with ID starting with 1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c not found: ID does not exist"
Sep 18 16:23:56.361680 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.361652    2566 scope.go:117] "RemoveContainer" containerID="391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018"
Sep 18 16:23:56.361854 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.361834    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018"} err="failed to get container status \"391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018\": rpc error: code = NotFound desc = could not find container \"391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018\": container with ID starting with 391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018 not found: ID does not exist"
Sep 18 16:23:56.361888 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.361856    2566 scope.go:117] "RemoveContainer" containerID="c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf"
Sep 18 16:23:56.362048 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.362035    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf"} err="failed to get container status \"c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf\": rpc error: code = NotFound desc = could not find container \"c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf\": container with ID starting with c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf not found: ID does not exist"
Sep 18 16:23:56.362048 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.362048    2566 scope.go:117] "RemoveContainer" containerID="01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28"
Sep 18 16:23:56.362267 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.362246    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28"} err="failed to get container status \"01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28\": rpc error: code = NotFound desc = could not find container \"01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28\": container with ID starting with 01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28 not found: ID does not exist"
Sep 18 16:23:56.362321 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.362268    2566 scope.go:117] "RemoveContainer" containerID="ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665"
Sep 18 16:23:56.362466 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.362450    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665"} err="failed to get container status \"ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665\": rpc error: code = NotFound desc = could not find container \"ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665\": container with ID starting with ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665 not found: ID does not exist"
Sep 18 16:23:56.362507 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.362465    2566 scope.go:117] "RemoveContainer" containerID="5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48"
Sep 18 16:23:56.362671 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.362652    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48"} err="failed to get container status \"5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48\": rpc error: code = NotFound desc = could not find container \"5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48\": container with ID starting with 5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48 not found: ID does not exist"
Sep 18 16:23:56.362708 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.362673    2566 scope.go:117] "RemoveContainer" containerID="d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735"
Sep 18 16:23:56.362886 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.362871    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735"} err="failed to get container status \"d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735\": rpc error: code = NotFound desc = could not find container \"d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735\": container with ID starting with d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735 not found: ID does not exist"
Sep 18 16:23:56.362886 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.362884    2566 scope.go:117] "RemoveContainer" containerID="1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c"
Sep 18 16:23:56.363111 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.363093    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c"} err="failed to get container status \"1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c\": rpc error: code = NotFound desc = could not find container \"1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c\": container with ID starting with 1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c not found: ID does not exist"
Sep 18 16:23:56.363111 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.363109    2566 scope.go:117] "RemoveContainer" containerID="391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018"
Sep 18 16:23:56.363321 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.363304    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018"} err="failed to get container status \"391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018\": rpc error: code = NotFound desc = could not find container \"391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018\": container with ID starting with 391d46ed996774017af20cd12dc38c7382c40c36dec38d3a13f11495a031a018 not found: ID does not exist"
Sep 18 16:23:56.363383 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.363322    2566 scope.go:117] "RemoveContainer" containerID="c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf"
Sep 18 16:23:56.363564 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.363547    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf"} err="failed to get container status \"c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf\": rpc error: code = NotFound desc = could not find container \"c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf\": container with ID starting with c7042bbff245d60b48657ee80456442518372afc13d223154027a0af1f8f1edf not found: ID does not exist"
Sep 18 16:23:56.363610 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.363565    2566 scope.go:117] "RemoveContainer" containerID="01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28"
Sep 18 16:23:56.363794 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.363777    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28"} err="failed to get container status \"01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28\": rpc error: code = NotFound desc = could not find container \"01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28\": container with ID starting with 01bf172e3038fe22ecbeb3246ace764ec6074263edcfc277f9d6c87e6bcfec28 not found: ID does not exist"
Sep 18 16:23:56.363844 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.363793    2566 scope.go:117] "RemoveContainer" containerID="ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665"
Sep 18 16:23:56.364013 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.363995    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665"} err="failed to get container status \"ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665\": rpc error: code = NotFound desc = could not find container \"ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665\": container with ID starting with ee4edd927ccceb746430ae8cbc9583658994f143b5c289f73a89d6832e328665 not found: ID does not exist"
Sep 18 16:23:56.364097 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.364013    2566 scope.go:117] "RemoveContainer" containerID="5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48"
Sep 18 16:23:56.364244 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.364227    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48"} err="failed to get container status \"5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48\": rpc error: code = NotFound desc = could not find container \"5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48\": container with ID starting with 5d7bf79004bd6aacf622c3c85b2db4b6d8cc9da0a0cf227ebbdd0d3f06df0e48 not found: ID does not exist"
Sep 18 16:23:56.364309 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.364245    2566 scope.go:117] "RemoveContainer" containerID="d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735"
Sep 18 16:23:56.364489 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.364473    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735"} err="failed to get container status \"d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735\": rpc error: code = NotFound desc = could not find container \"d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735\": container with ID starting with d663aafad37161a1959852ce8ec8d11e7c8731f85e7af6474c73d245cd5db735 not found: ID does not exist"
Sep 18 16:23:56.364531 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.364491    2566 scope.go:117] "RemoveContainer" containerID="1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c"
Sep 18 16:23:56.364716 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.364699    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c"} err="failed to get container status \"1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c\": rpc error: code = NotFound desc = could not find container \"1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c\": container with ID starting with 1d750af6228aec8390c4b54ed3275dbc930fbd5ed32714986929872910dd196c not found: ID does not exist"
Sep 18 16:23:56.377858 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.377840    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:23:56.378167 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378070    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerName="config-reloader"
Sep 18 16:23:56.378167 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378098    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerName="config-reloader"
Sep 18 16:23:56.378167 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378106    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerName="kube-rbac-proxy-thanos"
Sep 18 16:23:56.378167 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378112    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerName="kube-rbac-proxy-thanos"
Sep 18 16:23:56.378167 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378120    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerName="registry"
Sep 18 16:23:56.378167 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378125    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerName="registry"
Sep 18 16:23:56.378167 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378132    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerName="kube-rbac-proxy"
Sep 18 16:23:56.378167 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378137    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerName="kube-rbac-proxy"
Sep 18 16:23:56.378167 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378147    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerName="kube-rbac-proxy-web"
Sep 18 16:23:56.378167 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378152    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerName="kube-rbac-proxy-web"
Sep 18 16:23:56.378167 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378158    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerName="thanos-sidecar"
Sep 18 16:23:56.378167 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378163    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerName="thanos-sidecar"
Sep 18 16:23:56.378167 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378170    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerName="prometheus"
Sep 18 16:23:56.378167 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378175    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerName="prometheus"
Sep 18 16:23:56.378792 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378184    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerName="registry"
Sep 18 16:23:56.378792 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378188    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerName="registry"
Sep 18 16:23:56.378792 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378195    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerName="init-config-reloader"
Sep 18 16:23:56.378792 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378200    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerName="init-config-reloader"
Sep 18 16:23:56.378792 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378244    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerName="registry"
Sep 18 16:23:56.378792 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378251    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerName="kube-rbac-proxy"
Sep 18 16:23:56.378792 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378257    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerName="kube-rbac-proxy-thanos"
Sep 18 16:23:56.378792 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378262    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerName="kube-rbac-proxy-web"
Sep 18 16:23:56.378792 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378269    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerName="registry"
Sep 18 16:23:56.378792 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378274    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerName="config-reloader"
Sep 18 16:23:56.378792 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378279    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerName="prometheus"
Sep 18 16:23:56.378792 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378284    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" containerName="thanos-sidecar"
Sep 18 16:23:56.378792 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378317    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerName="registry"
Sep 18 16:23:56.378792 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378323    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerName="registry"
Sep 18 16:23:56.378792 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.378368    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ada52f1-14ec-41fe-a161-7e8b0fa6a12d" containerName="registry"
Sep 18 16:23:56.382009 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.381992    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.384455 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.384424    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 16:23:56.384557 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.384451    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 16:23:56.384557 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.384501    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3ri1d1dcnr9dk\""
Sep 18 16:23:56.384942 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.384761    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 16:23:56.384942 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.384776    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 16:23:56.384942 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.384799    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 16:23:56.384942 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.384822    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 16:23:56.384942 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.384836    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 16:23:56.384942 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.384870    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-64ckn\""
Sep 18 16:23:56.384942 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.384870    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 16:23:56.385458 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.384963    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 16:23:56.385458 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.385291    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 16:23:56.385458 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.385377    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 16:23:56.385458 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.385411    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 16:23:56.386055 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.386038    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 16:23:56.387652 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.387633    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 16:23:56.391384 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.391367    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 16:23:56.403619 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.403597    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:23:56.456871 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.456840    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/66ab706e-7e5f-4f92-a967-33f822ff4b64-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.456871 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.456873    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66ab706e-7e5f-4f92-a967-33f822ff4b64-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.457059 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.456895    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66ab706e-7e5f-4f92-a967-33f822ff4b64-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.457059 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.456929    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66ab706e-7e5f-4f92-a967-33f822ff4b64-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.457059 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.456959    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66ab706e-7e5f-4f92-a967-33f822ff4b64-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.457059 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.456981    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66ab706e-7e5f-4f92-a967-33f822ff4b64-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.457059 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.457008    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/66ab706e-7e5f-4f92-a967-33f822ff4b64-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.457059 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.457029    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66ab706e-7e5f-4f92-a967-33f822ff4b64-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.457059 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.457047    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66ab706e-7e5f-4f92-a967-33f822ff4b64-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.457295 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.457070    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66ab706e-7e5f-4f92-a967-33f822ff4b64-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.457295 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.457116    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66ab706e-7e5f-4f92-a967-33f822ff4b64-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.457295 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.457139    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66ab706e-7e5f-4f92-a967-33f822ff4b64-config-out\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.457295 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.457153    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/66ab706e-7e5f-4f92-a967-33f822ff4b64-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.457295 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.457169    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66ab706e-7e5f-4f92-a967-33f822ff4b64-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.457295 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.457204    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66ab706e-7e5f-4f92-a967-33f822ff4b64-web-config\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.457295 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.457245    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66ab706e-7e5f-4f92-a967-33f822ff4b64-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.457295 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.457270    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qpm9\" (UniqueName: \"kubernetes.io/projected/66ab706e-7e5f-4f92-a967-33f822ff4b64-kube-api-access-5qpm9\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.457521 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.457297    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/66ab706e-7e5f-4f92-a967-33f822ff4b64-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.457521 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.457323    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/66ab706e-7e5f-4f92-a967-33f822ff4b64-config\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.457521 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.457350    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66ab706e-7e5f-4f92-a967-33f822ff4b64-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.558407 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.558334    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66ab706e-7e5f-4f92-a967-33f822ff4b64-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.558407 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.558367    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66ab706e-7e5f-4f92-a967-33f822ff4b64-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.558407 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.558388    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66ab706e-7e5f-4f92-a967-33f822ff4b64-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.558633 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.558426    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66ab706e-7e5f-4f92-a967-33f822ff4b64-config-out\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.558633 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.558451    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/66ab706e-7e5f-4f92-a967-33f822ff4b64-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.558992 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.558475    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66ab706e-7e5f-4f92-a967-33f822ff4b64-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.559152 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.559024    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66ab706e-7e5f-4f92-a967-33f822ff4b64-web-config\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.559373 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.559297    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66ab706e-7e5f-4f92-a967-33f822ff4b64-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.559599 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.559562    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qpm9\" (UniqueName: \"kubernetes.io/projected/66ab706e-7e5f-4f92-a967-33f822ff4b64-kube-api-access-5qpm9\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.559757 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.559736    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/66ab706e-7e5f-4f92-a967-33f822ff4b64-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.559901 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.559885    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/66ab706e-7e5f-4f92-a967-33f822ff4b64-config\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.560032 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.560016    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66ab706e-7e5f-4f92-a967-33f822ff4b64-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.560199 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.560182    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/66ab706e-7e5f-4f92-a967-33f822ff4b64-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.560343 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.560324    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66ab706e-7e5f-4f92-a967-33f822ff4b64-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.560483 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.560468    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66ab706e-7e5f-4f92-a967-33f822ff4b64-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.560624 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.560610    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66ab706e-7e5f-4f92-a967-33f822ff4b64-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.560769 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.560751    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66ab706e-7e5f-4f92-a967-33f822ff4b64-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.560924 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.560898    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66ab706e-7e5f-4f92-a967-33f822ff4b64-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.561055 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.561038    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/66ab706e-7e5f-4f92-a967-33f822ff4b64-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.563803 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.562980    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66ab706e-7e5f-4f92-a967-33f822ff4b64-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.563803 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.559405    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66ab706e-7e5f-4f92-a967-33f822ff4b64-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.563803 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.559387    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66ab706e-7e5f-4f92-a967-33f822ff4b64-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.563803 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.562231    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66ab706e-7e5f-4f92-a967-33f822ff4b64-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.563803 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.562922    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66ab706e-7e5f-4f92-a967-33f822ff4b64-config-out\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.563803 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.561105    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66ab706e-7e5f-4f92-a967-33f822ff4b64-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.563803 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.563221    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66ab706e-7e5f-4f92-a967-33f822ff4b64-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.563803 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.563714    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66ab706e-7e5f-4f92-a967-33f822ff4b64-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.563803 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.563747    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/66ab706e-7e5f-4f92-a967-33f822ff4b64-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.563803 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.563771    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66ab706e-7e5f-4f92-a967-33f822ff4b64-web-config\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.564397 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.563808    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/66ab706e-7e5f-4f92-a967-33f822ff4b64-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.564397 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.563915    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/66ab706e-7e5f-4f92-a967-33f822ff4b64-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.564397 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.564095    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66ab706e-7e5f-4f92-a967-33f822ff4b64-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.564397 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.564174    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66ab706e-7e5f-4f92-a967-33f822ff4b64-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.564397 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.564281    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/66ab706e-7e5f-4f92-a967-33f822ff4b64-config\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.564637 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.564443    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66ab706e-7e5f-4f92-a967-33f822ff4b64-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.566187 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.565372    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66ab706e-7e5f-4f92-a967-33f822ff4b64-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.566187 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.565707    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66ab706e-7e5f-4f92-a967-33f822ff4b64-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.569391 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.568966    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qpm9\" (UniqueName: \"kubernetes.io/projected/66ab706e-7e5f-4f92-a967-33f822ff4b64-kube-api-access-5qpm9\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.571751 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.571732    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/66ab706e-7e5f-4f92-a967-33f822ff4b64-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.571938 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.571917    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66ab706e-7e5f-4f92-a967-33f822ff4b64-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"66ab706e-7e5f-4f92-a967-33f822ff4b64\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.696261 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.696228    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:23:56.975186 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:56.975164    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:23:56.976827 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:23:56.976805    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ab706e_7e5f_4f92_a967_33f822ff4b64.slice/crio-3af363202fdc6e846c947695122075b277b7aee04bd72c730e1305a7067c4af1 WatchSource:0}: Error finding container 3af363202fdc6e846c947695122075b277b7aee04bd72c730e1305a7067c4af1: Status 404 returned error can't find the container with id 3af363202fdc6e846c947695122075b277b7aee04bd72c730e1305a7067c4af1
Sep 18 16:23:57.314063 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:57.313977    2566 generic.go:356] "Generic (PLEG): container finished" podID="66ab706e-7e5f-4f92-a967-33f822ff4b64" containerID="9944b9eef7bb7da155d8eff0ed6b9abb7703d006cd4ee37c82f33cb4918cd365" exitCode=0
Sep 18 16:23:57.314063 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:57.314021    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66ab706e-7e5f-4f92-a967-33f822ff4b64","Type":"ContainerDied","Data":"9944b9eef7bb7da155d8eff0ed6b9abb7703d006cd4ee37c82f33cb4918cd365"}
Sep 18 16:23:57.314286 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:57.314114    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66ab706e-7e5f-4f92-a967-33f822ff4b64","Type":"ContainerStarted","Data":"3af363202fdc6e846c947695122075b277b7aee04bd72c730e1305a7067c4af1"}
Sep 18 16:23:57.686282 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:57.686256    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cd7094e-fea8-43b7-a6e8-d4dc35d17908" path="/var/lib/kubelet/pods/6cd7094e-fea8-43b7-a6e8-d4dc35d17908/volumes"
Sep 18 16:23:58.320850 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:58.320818    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66ab706e-7e5f-4f92-a967-33f822ff4b64","Type":"ContainerStarted","Data":"bdca288c21d8091a8344f17e7aa826506bf90dcc7eb49cd590f6903ced9e1422"}
Sep 18 16:23:58.321002 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:58.320879    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66ab706e-7e5f-4f92-a967-33f822ff4b64","Type":"ContainerStarted","Data":"cb661ac1ecdbf7e74f724a1779308f27512b43610a7bdd43c9f5a66bc9ee429e"}
Sep 18 16:23:58.321002 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:58.320906    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66ab706e-7e5f-4f92-a967-33f822ff4b64","Type":"ContainerStarted","Data":"2bf312a594daa7e21f2dcca4c186fa39dfd26e57c837b5a58fe7793a0768b717"}
Sep 18 16:23:58.321002 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:58.320931    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66ab706e-7e5f-4f92-a967-33f822ff4b64","Type":"ContainerStarted","Data":"d04160ceaa84194a9004633344a019d18d42ef01111685d6c9ab5bd631fbff85"}
Sep 18 16:23:58.321002 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:58.320956    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66ab706e-7e5f-4f92-a967-33f822ff4b64","Type":"ContainerStarted","Data":"82c32a9499be7283d57cca455cc7928667281dc115d71f47d20c1616a83927d3"}
Sep 18 16:23:58.321002 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:58.320978    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66ab706e-7e5f-4f92-a967-33f822ff4b64","Type":"ContainerStarted","Data":"763f2ba338cf5c27629b04f0f6d37774d8715a271454df5d1de6b4131c7c3f05"}
Sep 18 16:23:58.349654 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:23:58.349601    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.330974338 podStartE2EDuration="2.349582813s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.608475ms" isStatefulPod=false podCreationTimestamp="2026-09-18 16:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:23:58.347709456 +0000 UTC m=+476.803162536" watchObservedRunningTime="2026-09-18 16:23:58.349582813 +0000 UTC m=+476.805035889"
Sep 18 16:24:01.697013 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:01.696964    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:24:08.722920 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:08.722888    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b94d5b846-s7qq9"]
Sep 18 16:24:33.743306 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:33.743252    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b94d5b846-s7qq9" podUID="d1ce2ea5-46c5-4df2-877b-4f78d16cbf36" containerName="console" containerID="cri-o://0d87e371184a8e2e4afeccda8fb0a196457d247980da6339727d83b1b76feceb" gracePeriod=15
Sep 18 16:24:33.988587 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:33.988556    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:24:34.081027 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.080998    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-service-ca\") pod \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") "
Sep 18 16:24:34.081227 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.081040    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-console-serving-cert\") pod \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") "
Sep 18 16:24:34.081227 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.081100    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-console-oauth-config\") pod \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") "
Sep 18 16:24:34.081227 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.081118    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9vpc\" (UniqueName: \"kubernetes.io/projected/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-kube-api-access-r9vpc\") pod \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") "
Sep 18 16:24:34.081227 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.081141    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-trusted-ca-bundle\") pod \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") "
Sep 18 16:24:34.081227 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.081161    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-console-config\") pod \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") "
Sep 18 16:24:34.081480 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.081262    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-oauth-serving-cert\") pod \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\" (UID: \"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36\") "
Sep 18 16:24:34.081582 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.081554    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-service-ca" (OuterVolumeSpecName: "service-ca") pod "d1ce2ea5-46c5-4df2-877b-4f78d16cbf36" (UID: "d1ce2ea5-46c5-4df2-877b-4f78d16cbf36"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:24:34.081716 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.081677    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-console-config" (OuterVolumeSpecName: "console-config") pod "d1ce2ea5-46c5-4df2-877b-4f78d16cbf36" (UID: "d1ce2ea5-46c5-4df2-877b-4f78d16cbf36"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:24:34.081716 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.081685    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d1ce2ea5-46c5-4df2-877b-4f78d16cbf36" (UID: "d1ce2ea5-46c5-4df2-877b-4f78d16cbf36"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:24:34.081814 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.081728    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d1ce2ea5-46c5-4df2-877b-4f78d16cbf36" (UID: "d1ce2ea5-46c5-4df2-877b-4f78d16cbf36"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:24:34.083520 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.083500    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d1ce2ea5-46c5-4df2-877b-4f78d16cbf36" (UID: "d1ce2ea5-46c5-4df2-877b-4f78d16cbf36"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:24:34.083608 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.083536    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d1ce2ea5-46c5-4df2-877b-4f78d16cbf36" (UID: "d1ce2ea5-46c5-4df2-877b-4f78d16cbf36"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:24:34.083608 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.083602    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-kube-api-access-r9vpc" (OuterVolumeSpecName: "kube-api-access-r9vpc") pod "d1ce2ea5-46c5-4df2-877b-4f78d16cbf36" (UID: "d1ce2ea5-46c5-4df2-877b-4f78d16cbf36"). InnerVolumeSpecName "kube-api-access-r9vpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:24:34.182053 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.182017    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-console-oauth-config\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:24:34.182053 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.182047    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9vpc\" (UniqueName: \"kubernetes.io/projected/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-kube-api-access-r9vpc\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:24:34.182053 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.182056    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-trusted-ca-bundle\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:24:34.182053 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.182066    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-console-config\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:24:34.182351 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.182096    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-oauth-serving-cert\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:24:34.182351 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.182105    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-service-ca\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:24:34.182351 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.182115    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36-console-serving-cert\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:24:34.458859 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.458773    2566 generic.go:356] "Generic (PLEG): container finished" podID="d1ce2ea5-46c5-4df2-877b-4f78d16cbf36" containerID="0d87e371184a8e2e4afeccda8fb0a196457d247980da6339727d83b1b76feceb" exitCode=0
Sep 18 16:24:34.458859 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.458813    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b94d5b846-s7qq9"
Sep 18 16:24:34.459032 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.458858    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b94d5b846-s7qq9" event={"ID":"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36","Type":"ContainerDied","Data":"0d87e371184a8e2e4afeccda8fb0a196457d247980da6339727d83b1b76feceb"}
Sep 18 16:24:34.459032 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.458907    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b94d5b846-s7qq9" event={"ID":"d1ce2ea5-46c5-4df2-877b-4f78d16cbf36","Type":"ContainerDied","Data":"d1a77b6809681c46dd66fd199c1336121993b0b19b6bf4386d93c3307792497e"}
Sep 18 16:24:34.459032 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.458930    2566 scope.go:117] "RemoveContainer" containerID="0d87e371184a8e2e4afeccda8fb0a196457d247980da6339727d83b1b76feceb"
Sep 18 16:24:34.467909 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.467885    2566 scope.go:117] "RemoveContainer" containerID="0d87e371184a8e2e4afeccda8fb0a196457d247980da6339727d83b1b76feceb"
Sep 18 16:24:34.468214 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:24:34.468194    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d87e371184a8e2e4afeccda8fb0a196457d247980da6339727d83b1b76feceb\": container with ID starting with 0d87e371184a8e2e4afeccda8fb0a196457d247980da6339727d83b1b76feceb not found: ID does not exist" containerID="0d87e371184a8e2e4afeccda8fb0a196457d247980da6339727d83b1b76feceb"
Sep 18 16:24:34.468274 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.468224    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d87e371184a8e2e4afeccda8fb0a196457d247980da6339727d83b1b76feceb"} err="failed to get container status \"0d87e371184a8e2e4afeccda8fb0a196457d247980da6339727d83b1b76feceb\": rpc error: code = NotFound desc = could not find container \"0d87e371184a8e2e4afeccda8fb0a196457d247980da6339727d83b1b76feceb\": container with ID starting with 0d87e371184a8e2e4afeccda8fb0a196457d247980da6339727d83b1b76feceb not found: ID does not exist"
Sep 18 16:24:34.481128 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.481095    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b94d5b846-s7qq9"]
Sep 18 16:24:34.483332 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:34.483309    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b94d5b846-s7qq9"]
Sep 18 16:24:35.685254 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:35.685223    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1ce2ea5-46c5-4df2-877b-4f78d16cbf36" path="/var/lib/kubelet/pods/d1ce2ea5-46c5-4df2-877b-4f78d16cbf36/volumes"
Sep 18 16:24:56.696621 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:56.696577    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:24:56.711918 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:56.711893    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:24:57.567747 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:24:57.567720    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:26:03.586037 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:26:03.586008    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-vfmdv_1e3ca1ec-6856-4afb-850e-85e3fa968b3f/console-operator/1.log"
Sep 18 16:26:03.611198 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:26:03.611173    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-vfmdv_1e3ca1ec-6856-4afb-850e-85e3fa968b3f/console-operator/1.log"
Sep 18 16:27:41.731644 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:41.731608    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-nkc8p"]
Sep 18 16:27:41.732127 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:41.732009    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1ce2ea5-46c5-4df2-877b-4f78d16cbf36" containerName="console"
Sep 18 16:27:41.732127 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:41.732023    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1ce2ea5-46c5-4df2-877b-4f78d16cbf36" containerName="console"
Sep 18 16:27:41.732127 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:41.732112    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1ce2ea5-46c5-4df2-877b-4f78d16cbf36" containerName="console"
Sep 18 16:27:41.735286 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:41.735270    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nkc8p"
Sep 18 16:27:41.738248 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:41.738223    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 18 16:27:41.738382 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:41.738223    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 18 16:27:41.739128 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:41.739108    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-zjtn5\""
Sep 18 16:27:41.739128 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:41.739125    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 18 16:27:41.739294 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:41.739107    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 18 16:27:41.739294 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:41.739184    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 18 16:27:41.748057 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:41.748031    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-nkc8p"]
Sep 18 16:27:41.846087 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:41.846046    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt6df\" (UniqueName: \"kubernetes.io/projected/367f2576-d500-4d52-a2fa-4a018614bdf4-kube-api-access-nt6df\") pod \"keda-metrics-apiserver-77857b7b98-nkc8p\" (UID: \"367f2576-d500-4d52-a2fa-4a018614bdf4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nkc8p"
Sep 18 16:27:41.846276 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:41.846239    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/367f2576-d500-4d52-a2fa-4a018614bdf4-certificates\") pod \"keda-metrics-apiserver-77857b7b98-nkc8p\" (UID: \"367f2576-d500-4d52-a2fa-4a018614bdf4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nkc8p"
Sep 18 16:27:41.846351 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:41.846318    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/367f2576-d500-4d52-a2fa-4a018614bdf4-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-nkc8p\" (UID: \"367f2576-d500-4d52-a2fa-4a018614bdf4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nkc8p"
Sep 18 16:27:41.946858 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:41.946819    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/367f2576-d500-4d52-a2fa-4a018614bdf4-certificates\") pod \"keda-metrics-apiserver-77857b7b98-nkc8p\" (UID: \"367f2576-d500-4d52-a2fa-4a018614bdf4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nkc8p"
Sep 18 16:27:41.946858 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:41.946860    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/367f2576-d500-4d52-a2fa-4a018614bdf4-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-nkc8p\" (UID: \"367f2576-d500-4d52-a2fa-4a018614bdf4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nkc8p"
Sep 18 16:27:41.947112 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:41.946910    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt6df\" (UniqueName: \"kubernetes.io/projected/367f2576-d500-4d52-a2fa-4a018614bdf4-kube-api-access-nt6df\") pod \"keda-metrics-apiserver-77857b7b98-nkc8p\" (UID: \"367f2576-d500-4d52-a2fa-4a018614bdf4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nkc8p"
Sep 18 16:27:41.947112 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:27:41.946972    2566 secret.go:281] references non-existent secret key: tls.crt
Sep 18 16:27:41.947112 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:27:41.946997    2566 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 18 16:27:41.947112 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:27:41.947021    2566 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-nkc8p: references non-existent secret key: tls.crt
Sep 18 16:27:41.947112 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:27:41.947110    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/367f2576-d500-4d52-a2fa-4a018614bdf4-certificates podName:367f2576-d500-4d52-a2fa-4a018614bdf4 nodeName:}" failed. No retries permitted until 2026-09-18 16:27:42.447064741 +0000 UTC m=+700.902517800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/367f2576-d500-4d52-a2fa-4a018614bdf4-certificates") pod "keda-metrics-apiserver-77857b7b98-nkc8p" (UID: "367f2576-d500-4d52-a2fa-4a018614bdf4") : references non-existent secret key: tls.crt
Sep 18 16:27:41.947399 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:41.947376    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/367f2576-d500-4d52-a2fa-4a018614bdf4-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-nkc8p\" (UID: \"367f2576-d500-4d52-a2fa-4a018614bdf4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nkc8p"
Sep 18 16:27:41.956130 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:41.956102    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt6df\" (UniqueName: \"kubernetes.io/projected/367f2576-d500-4d52-a2fa-4a018614bdf4-kube-api-access-nt6df\") pod \"keda-metrics-apiserver-77857b7b98-nkc8p\" (UID: \"367f2576-d500-4d52-a2fa-4a018614bdf4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nkc8p"
Sep 18 16:27:42.451459 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:42.451421    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/367f2576-d500-4d52-a2fa-4a018614bdf4-certificates\") pod \"keda-metrics-apiserver-77857b7b98-nkc8p\" (UID: \"367f2576-d500-4d52-a2fa-4a018614bdf4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nkc8p"
Sep 18 16:27:42.453984 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:42.453955    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/367f2576-d500-4d52-a2fa-4a018614bdf4-certificates\") pod \"keda-metrics-apiserver-77857b7b98-nkc8p\" (UID: \"367f2576-d500-4d52-a2fa-4a018614bdf4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nkc8p"
Sep 18 16:27:42.645916 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:42.645872    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nkc8p"
Sep 18 16:27:42.921174 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:42.921149    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-nkc8p"]
Sep 18 16:27:42.923568 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:27:42.923537    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod367f2576_d500_4d52_a2fa_4a018614bdf4.slice/crio-880584ba827e58f6a077ea0b59f762417e871f7597ace5d39b56c5b63daec590 WatchSource:0}: Error finding container 880584ba827e58f6a077ea0b59f762417e871f7597ace5d39b56c5b63daec590: Status 404 returned error can't find the container with id 880584ba827e58f6a077ea0b59f762417e871f7597ace5d39b56c5b63daec590
Sep 18 16:27:42.924881 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:42.924864    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:27:43.151425 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:43.151387    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nkc8p" event={"ID":"367f2576-d500-4d52-a2fa-4a018614bdf4","Type":"ContainerStarted","Data":"880584ba827e58f6a077ea0b59f762417e871f7597ace5d39b56c5b63daec590"}
Sep 18 16:27:47.174322 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:47.174278    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nkc8p" event={"ID":"367f2576-d500-4d52-a2fa-4a018614bdf4","Type":"ContainerStarted","Data":"da6b6b3edf989f4cfcb1a6601fc3c3fa773eedc7a1e54f79d77de13218cd3d26"}
Sep 18 16:27:47.174795 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:47.174379    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nkc8p"
Sep 18 16:27:47.192920 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:47.192872    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nkc8p" podStartSLOduration=2.902277783 podStartE2EDuration="6.192855122s" totalImagesPullingTime="3.290577339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:27:47.191838694 +0000 UTC m=+705.647291782" watchObservedRunningTime="2026-09-18 16:27:47.192855122 +0000 UTC m=+705.648308178"
Sep 18 16:27:58.183560 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:27:58.183522    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nkc8p"
Sep 18 16:29:00.470318 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:29:00.470284    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-g2mv4"]
Sep 18 16:29:00.473299 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:29:00.473283    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-g2mv4"
Sep 18 16:29:00.475927 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:29:00.475903    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 18 16:29:00.476048 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:29:00.475967    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 16:29:00.476351 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:29:00.476336    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 16:29:00.477029 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:29:00.477013    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-zzblr\""
Sep 18 16:29:00.483562 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:29:00.483540    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-g2mv4"]
Sep 18 16:29:00.624302 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:29:00.624262    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18591da2-4faa-4c7f-96cf-f9b9e9dce2c6-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-g2mv4\" (UID: \"18591da2-4faa-4c7f-96cf-f9b9e9dce2c6\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-g2mv4"
Sep 18 16:29:00.624473 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:29:00.624320    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78m2l\" (UniqueName: \"kubernetes.io/projected/18591da2-4faa-4c7f-96cf-f9b9e9dce2c6-kube-api-access-78m2l\") pod \"llmisvc-controller-manager-645f9c8fd6-g2mv4\" (UID: \"18591da2-4faa-4c7f-96cf-f9b9e9dce2c6\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-g2mv4"
Sep 18 16:29:00.725258 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:29:00.725165    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18591da2-4faa-4c7f-96cf-f9b9e9dce2c6-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-g2mv4\" (UID: \"18591da2-4faa-4c7f-96cf-f9b9e9dce2c6\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-g2mv4"
Sep 18 16:29:00.725258 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:29:00.725219    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78m2l\" (UniqueName: \"kubernetes.io/projected/18591da2-4faa-4c7f-96cf-f9b9e9dce2c6-kube-api-access-78m2l\") pod \"llmisvc-controller-manager-645f9c8fd6-g2mv4\" (UID: \"18591da2-4faa-4c7f-96cf-f9b9e9dce2c6\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-g2mv4"
Sep 18 16:29:00.727663 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:29:00.727640    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18591da2-4faa-4c7f-96cf-f9b9e9dce2c6-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-g2mv4\" (UID: \"18591da2-4faa-4c7f-96cf-f9b9e9dce2c6\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-g2mv4"
Sep 18 16:29:00.737119 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:29:00.737094    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78m2l\" (UniqueName: \"kubernetes.io/projected/18591da2-4faa-4c7f-96cf-f9b9e9dce2c6-kube-api-access-78m2l\") pod \"llmisvc-controller-manager-645f9c8fd6-g2mv4\" (UID: \"18591da2-4faa-4c7f-96cf-f9b9e9dce2c6\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-g2mv4"
Sep 18 16:29:00.784911 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:29:00.784871    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-g2mv4"
Sep 18 16:29:01.052673 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:29:01.052582    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-g2mv4"]
Sep 18 16:29:01.056047 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:29:01.056023    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod18591da2_4faa_4c7f_96cf_f9b9e9dce2c6.slice/crio-9f7147c62764c13db4456dda265bf599e8142b894acd577d53dea2bae00ab4b3 WatchSource:0}: Error finding container 9f7147c62764c13db4456dda265bf599e8142b894acd577d53dea2bae00ab4b3: Status 404 returned error can't find the container with id 9f7147c62764c13db4456dda265bf599e8142b894acd577d53dea2bae00ab4b3
Sep 18 16:29:01.459660 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:29:01.459622    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-g2mv4" event={"ID":"18591da2-4faa-4c7f-96cf-f9b9e9dce2c6","Type":"ContainerStarted","Data":"9f7147c62764c13db4456dda265bf599e8142b894acd577d53dea2bae00ab4b3"}
Sep 18 16:29:04.473770 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:29:04.473733    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-g2mv4" event={"ID":"18591da2-4faa-4c7f-96cf-f9b9e9dce2c6","Type":"ContainerStarted","Data":"431ea469ea925743be58dd1603cb23f5280ef9504bac2b775bf9fc017c207d14"}
Sep 18 16:29:04.474140 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:29:04.473890    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-g2mv4"
Sep 18 16:29:04.491012 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:29:04.490937    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-g2mv4" podStartSLOduration=2.123391386 podStartE2EDuration="4.490920133s" totalImagesPullingTime="2.367528747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:29:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:29:04.490097155 +0000 UTC m=+782.945550230" watchObservedRunningTime="2026-09-18 16:29:04.490920133 +0000 UTC m=+782.946373209"
Sep 18 16:29:35.480290 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:29:35.480218    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-g2mv4"
Sep 18 16:30:34.101784 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:30:34.101748    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-5lvl8"]
Sep 18 16:30:34.105356 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:30:34.105333    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5lvl8"
Sep 18 16:30:34.108120 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:30:34.108095    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 18 16:30:34.108120 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:30:34.108109    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-dp9b6\""
Sep 18 16:30:34.108297 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:30:34.108141    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 18 16:30:34.112973 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:30:34.112952    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-5lvl8"]
Sep 18 16:30:34.237159 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:30:34.237124    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf2b9\" (UniqueName: \"kubernetes.io/projected/edb62ba8-aecc-40fa-93d7-c865713ab9e8-kube-api-access-rf2b9\") pod \"s3-init-5lvl8\" (UID: \"edb62ba8-aecc-40fa-93d7-c865713ab9e8\") " pod="kserve/s3-init-5lvl8"
Sep 18 16:30:34.237333 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:30:34.237190    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edb62ba8-aecc-40fa-93d7-c865713ab9e8-model-cache\") pod \"s3-init-5lvl8\" (UID: \"edb62ba8-aecc-40fa-93d7-c865713ab9e8\") " pod="kserve/s3-init-5lvl8"
Sep 18 16:30:34.338585 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:30:34.338533    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf2b9\" (UniqueName: \"kubernetes.io/projected/edb62ba8-aecc-40fa-93d7-c865713ab9e8-kube-api-access-rf2b9\") pod \"s3-init-5lvl8\" (UID: \"edb62ba8-aecc-40fa-93d7-c865713ab9e8\") " pod="kserve/s3-init-5lvl8"
Sep 18 16:30:34.338769 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:30:34.338616    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edb62ba8-aecc-40fa-93d7-c865713ab9e8-model-cache\") pod \"s3-init-5lvl8\" (UID: \"edb62ba8-aecc-40fa-93d7-c865713ab9e8\") " pod="kserve/s3-init-5lvl8"
Sep 18 16:30:34.338954 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:30:34.338934    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edb62ba8-aecc-40fa-93d7-c865713ab9e8-model-cache\") pod \"s3-init-5lvl8\" (UID: \"edb62ba8-aecc-40fa-93d7-c865713ab9e8\") " pod="kserve/s3-init-5lvl8"
Sep 18 16:30:34.347553 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:30:34.347526    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf2b9\" (UniqueName: \"kubernetes.io/projected/edb62ba8-aecc-40fa-93d7-c865713ab9e8-kube-api-access-rf2b9\") pod \"s3-init-5lvl8\" (UID: \"edb62ba8-aecc-40fa-93d7-c865713ab9e8\") " pod="kserve/s3-init-5lvl8"
Sep 18 16:30:34.422726 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:30:34.422632    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5lvl8"
Sep 18 16:30:34.694999 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:30:34.694915    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-5lvl8"]
Sep 18 16:30:34.698577 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:30:34.698534    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedb62ba8_aecc_40fa_93d7_c865713ab9e8.slice/crio-3b43c51a7a952b0d1349ee5df0f0f3132a9152d9c785582ffc74a52e11d972a2 WatchSource:0}: Error finding container 3b43c51a7a952b0d1349ee5df0f0f3132a9152d9c785582ffc74a52e11d972a2: Status 404 returned error can't find the container with id 3b43c51a7a952b0d1349ee5df0f0f3132a9152d9c785582ffc74a52e11d972a2
Sep 18 16:30:34.814403 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:30:34.814372    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5lvl8" event={"ID":"edb62ba8-aecc-40fa-93d7-c865713ab9e8","Type":"ContainerStarted","Data":"3b43c51a7a952b0d1349ee5df0f0f3132a9152d9c785582ffc74a52e11d972a2"}
Sep 18 16:30:39.836878 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:30:39.836830    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5lvl8" event={"ID":"edb62ba8-aecc-40fa-93d7-c865713ab9e8","Type":"ContainerStarted","Data":"1b502616ebd5d0e5512af1fc4ecee2e7d619a21cc7f8b987d29c0182dfcd1bb8"}
Sep 18 16:30:41.846850 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:30:41.846817    2566 generic.go:356] "Generic (PLEG): container finished" podID="edb62ba8-aecc-40fa-93d7-c865713ab9e8" containerID="1b502616ebd5d0e5512af1fc4ecee2e7d619a21cc7f8b987d29c0182dfcd1bb8" exitCode=0
Sep 18 16:30:41.847237 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:30:41.846892    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5lvl8" event={"ID":"edb62ba8-aecc-40fa-93d7-c865713ab9e8","Type":"ContainerDied","Data":"1b502616ebd5d0e5512af1fc4ecee2e7d619a21cc7f8b987d29c0182dfcd1bb8"}
Sep 18 16:30:46.870833 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:30:46.870797    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5lvl8" event={"ID":"edb62ba8-aecc-40fa-93d7-c865713ab9e8","Type":"ContainerStarted","Data":"f05344709842209f4aa908d91ca65e480dad8eceb5991a8a68d6826234867c69"}
Sep 18 16:30:46.890871 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:30:46.890813    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-5lvl8" podStartSLOduration=1.681196517 podStartE2EDuration="12.890762173s" totalImagesPullingTime="9.072446071s" totalInitContainerRuntime="2.137119585s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:30:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:30:46.889135625 +0000 UTC m=+885.344588755" watchObservedRunningTime="2026-09-18 16:30:46.890762173 +0000 UTC m=+885.346215252"
Sep 18 16:31:03.614453 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:31:03.614427    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-vfmdv_1e3ca1ec-6856-4afb-850e-85e3fa968b3f/console-operator/1.log"
Sep 18 16:31:03.635797 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:31:03.635771    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-vfmdv_1e3ca1ec-6856-4afb-850e-85e3fa968b3f/console-operator/1.log"
Sep 18 16:31:52.121865 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:31:52.121827    2566 generic.go:356] "Generic (PLEG): container finished" podID="edb62ba8-aecc-40fa-93d7-c865713ab9e8" containerID="f05344709842209f4aa908d91ca65e480dad8eceb5991a8a68d6826234867c69" exitCode=0
Sep 18 16:31:52.122394 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:31:52.121902    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5lvl8" event={"ID":"edb62ba8-aecc-40fa-93d7-c865713ab9e8","Type":"ContainerDied","Data":"f05344709842209f4aa908d91ca65e480dad8eceb5991a8a68d6826234867c69"}
Sep 18 16:31:53.253593 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:31:53.253564    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5lvl8"
Sep 18 16:31:53.319989 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:31:53.319949    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rf2b9\" (UniqueName: \"kubernetes.io/projected/edb62ba8-aecc-40fa-93d7-c865713ab9e8-kube-api-access-rf2b9\") pod \"edb62ba8-aecc-40fa-93d7-c865713ab9e8\" (UID: \"edb62ba8-aecc-40fa-93d7-c865713ab9e8\") "
Sep 18 16:31:53.319989 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:31:53.319989    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edb62ba8-aecc-40fa-93d7-c865713ab9e8-model-cache\") pod \"edb62ba8-aecc-40fa-93d7-c865713ab9e8\" (UID: \"edb62ba8-aecc-40fa-93d7-c865713ab9e8\") "
Sep 18 16:31:53.322180 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:31:53.322149    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb62ba8-aecc-40fa-93d7-c865713ab9e8-kube-api-access-rf2b9" (OuterVolumeSpecName: "kube-api-access-rf2b9") pod "edb62ba8-aecc-40fa-93d7-c865713ab9e8" (UID: "edb62ba8-aecc-40fa-93d7-c865713ab9e8"). InnerVolumeSpecName "kube-api-access-rf2b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:31:53.338509 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:31:53.338468    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb62ba8-aecc-40fa-93d7-c865713ab9e8-model-cache" (OuterVolumeSpecName: "model-cache") pod "edb62ba8-aecc-40fa-93d7-c865713ab9e8" (UID: "edb62ba8-aecc-40fa-93d7-c865713ab9e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:31:53.421365 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:31:53.421279    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf2b9\" (UniqueName: \"kubernetes.io/projected/edb62ba8-aecc-40fa-93d7-c865713ab9e8-kube-api-access-rf2b9\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:31:53.421365 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:31:53.421313    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edb62ba8-aecc-40fa-93d7-c865713ab9e8-model-cache\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:31:54.132174 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:31:54.132134    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5lvl8" event={"ID":"edb62ba8-aecc-40fa-93d7-c865713ab9e8","Type":"ContainerDied","Data":"3b43c51a7a952b0d1349ee5df0f0f3132a9152d9c785582ffc74a52e11d972a2"}
Sep 18 16:31:54.132174 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:31:54.132173    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5lvl8"
Sep 18 16:31:54.132384 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:31:54.132182    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b43c51a7a952b0d1349ee5df0f0f3132a9152d9c785582ffc74a52e11d972a2"
Sep 18 16:32:06.747832 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.747798    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"]
Sep 18 16:32:06.748289 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.748229    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edb62ba8-aecc-40fa-93d7-c865713ab9e8" containerName="s3-init"
Sep 18 16:32:06.748289 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.748244    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="edb62ba8-aecc-40fa-93d7-c865713ab9e8" containerName="s3-init"
Sep 18 16:32:06.748289 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.748274    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edb62ba8-aecc-40fa-93d7-c865713ab9e8" containerName="download-hf-model"
Sep 18 16:32:06.748289 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.748283    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="edb62ba8-aecc-40fa-93d7-c865713ab9e8" containerName="download-hf-model"
Sep 18 16:32:06.748421 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.748343    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="edb62ba8-aecc-40fa-93d7-c865713ab9e8" containerName="s3-init"
Sep 18 16:32:06.751997 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.751980    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"
Sep 18 16:32:06.759354 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.759330    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 18 16:32:06.760211 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.760192    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6lh7v\""
Sep 18 16:32:06.760332 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.760240    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 16:32:06.760332 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.760324    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 16:32:06.760461 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.760444    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 16:32:06.760551 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.760530    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 18 16:32:06.761366 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.761349    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 16:32:06.768605 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.768583    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"]
Sep 18 16:32:06.774477 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.774449    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"]
Sep 18 16:32:06.778382 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.778365    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"
Sep 18 16:32:06.783700 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.783671    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 18 16:32:06.794622 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.794594    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"]
Sep 18 16:32:06.843604 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.843571    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a799c2d6-f36f-41a9-9104-db5ed9e102c2-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf\" (UID: \"a799c2d6-f36f-41a9-9104-db5ed9e102c2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"
Sep 18 16:32:06.843783 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.843630    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a799c2d6-f36f-41a9-9104-db5ed9e102c2-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf\" (UID: \"a799c2d6-f36f-41a9-9104-db5ed9e102c2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"
Sep 18 16:32:06.843783 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.843659    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a799c2d6-f36f-41a9-9104-db5ed9e102c2-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf\" (UID: \"a799c2d6-f36f-41a9-9104-db5ed9e102c2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"
Sep 18 16:32:06.843783 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.843701    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhlnn\" (UniqueName: \"kubernetes.io/projected/a799c2d6-f36f-41a9-9104-db5ed9e102c2-kube-api-access-fhlnn\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf\" (UID: \"a799c2d6-f36f-41a9-9104-db5ed9e102c2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"
Sep 18 16:32:06.843939 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.843777    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a799c2d6-f36f-41a9-9104-db5ed9e102c2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf\" (UID: \"a799c2d6-f36f-41a9-9104-db5ed9e102c2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"
Sep 18 16:32:06.944723 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.944678    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a799c2d6-f36f-41a9-9104-db5ed9e102c2-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf\" (UID: \"a799c2d6-f36f-41a9-9104-db5ed9e102c2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"
Sep 18 16:32:06.944922 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.944739    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a799c2d6-f36f-41a9-9104-db5ed9e102c2-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf\" (UID: \"a799c2d6-f36f-41a9-9104-db5ed9e102c2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"
Sep 18 16:32:06.944922 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.944767    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a799c2d6-f36f-41a9-9104-db5ed9e102c2-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf\" (UID: \"a799c2d6-f36f-41a9-9104-db5ed9e102c2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"
Sep 18 16:32:06.944922 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.944807    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89932c02-ee32-4fd0-8e70-631066926d34-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-sxhhv\" (UID: \"89932c02-ee32-4fd0-8e70-631066926d34\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"
Sep 18 16:32:06.944922 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.944836    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhlnn\" (UniqueName: \"kubernetes.io/projected/a799c2d6-f36f-41a9-9104-db5ed9e102c2-kube-api-access-fhlnn\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf\" (UID: \"a799c2d6-f36f-41a9-9104-db5ed9e102c2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"
Sep 18 16:32:06.944922 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:32:06.944841    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 18 16:32:06.944922 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.944866    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdnsb\" (UniqueName: \"kubernetes.io/projected/89932c02-ee32-4fd0-8e70-631066926d34-kube-api-access-bdnsb\") pod \"isvc-canary-rollback-predictor-7f7b644c9-sxhhv\" (UID: \"89932c02-ee32-4fd0-8e70-631066926d34\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"
Sep 18 16:32:06.945266 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:32:06.944929    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a799c2d6-f36f-41a9-9104-db5ed9e102c2-proxy-tls podName:a799c2d6-f36f-41a9-9104-db5ed9e102c2 nodeName:}" failed. No retries permitted until 2026-09-18 16:32:07.444910547 +0000 UTC m=+965.900363602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a799c2d6-f36f-41a9-9104-db5ed9e102c2-proxy-tls") pod "isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf" (UID: "a799c2d6-f36f-41a9-9104-db5ed9e102c2") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 18 16:32:06.945266 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.945004    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89932c02-ee32-4fd0-8e70-631066926d34-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-sxhhv\" (UID: \"89932c02-ee32-4fd0-8e70-631066926d34\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"
Sep 18 16:32:06.945266 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.945037    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a799c2d6-f36f-41a9-9104-db5ed9e102c2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf\" (UID: \"a799c2d6-f36f-41a9-9104-db5ed9e102c2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"
Sep 18 16:32:06.945266 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.945126    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89932c02-ee32-4fd0-8e70-631066926d34-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-sxhhv\" (UID: \"89932c02-ee32-4fd0-8e70-631066926d34\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"
Sep 18 16:32:06.945266 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.945164    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a799c2d6-f36f-41a9-9104-db5ed9e102c2-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf\" (UID: \"a799c2d6-f36f-41a9-9104-db5ed9e102c2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"
Sep 18 16:32:06.945266 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.945167    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89932c02-ee32-4fd0-8e70-631066926d34-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-sxhhv\" (UID: \"89932c02-ee32-4fd0-8e70-631066926d34\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"
Sep 18 16:32:06.945591 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.945500    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a799c2d6-f36f-41a9-9104-db5ed9e102c2-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf\" (UID: \"a799c2d6-f36f-41a9-9104-db5ed9e102c2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"
Sep 18 16:32:06.945701 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.945680    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a799c2d6-f36f-41a9-9104-db5ed9e102c2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf\" (UID: \"a799c2d6-f36f-41a9-9104-db5ed9e102c2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"
Sep 18 16:32:06.965335 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:06.965297    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhlnn\" (UniqueName: \"kubernetes.io/projected/a799c2d6-f36f-41a9-9104-db5ed9e102c2-kube-api-access-fhlnn\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf\" (UID: \"a799c2d6-f36f-41a9-9104-db5ed9e102c2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"
Sep 18 16:32:07.046423 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:07.046320    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89932c02-ee32-4fd0-8e70-631066926d34-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-sxhhv\" (UID: \"89932c02-ee32-4fd0-8e70-631066926d34\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"
Sep 18 16:32:07.046423 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:07.046378    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdnsb\" (UniqueName: \"kubernetes.io/projected/89932c02-ee32-4fd0-8e70-631066926d34-kube-api-access-bdnsb\") pod \"isvc-canary-rollback-predictor-7f7b644c9-sxhhv\" (UID: \"89932c02-ee32-4fd0-8e70-631066926d34\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"
Sep 18 16:32:07.046423 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:07.046412    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89932c02-ee32-4fd0-8e70-631066926d34-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-sxhhv\" (UID: \"89932c02-ee32-4fd0-8e70-631066926d34\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"
Sep 18 16:32:07.046706 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:07.046462    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89932c02-ee32-4fd0-8e70-631066926d34-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-sxhhv\" (UID: \"89932c02-ee32-4fd0-8e70-631066926d34\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"
Sep 18 16:32:07.046706 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:07.046499    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89932c02-ee32-4fd0-8e70-631066926d34-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-sxhhv\" (UID: \"89932c02-ee32-4fd0-8e70-631066926d34\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"
Sep 18 16:32:07.046841 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:07.046822    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89932c02-ee32-4fd0-8e70-631066926d34-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-sxhhv\" (UID: \"89932c02-ee32-4fd0-8e70-631066926d34\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"
Sep 18 16:32:07.047140 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:07.047115    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89932c02-ee32-4fd0-8e70-631066926d34-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-sxhhv\" (UID: \"89932c02-ee32-4fd0-8e70-631066926d34\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"
Sep 18 16:32:07.047253 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:07.047205    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89932c02-ee32-4fd0-8e70-631066926d34-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-sxhhv\" (UID: \"89932c02-ee32-4fd0-8e70-631066926d34\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"
Sep 18 16:32:07.048867 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:07.048851    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89932c02-ee32-4fd0-8e70-631066926d34-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-sxhhv\" (UID: \"89932c02-ee32-4fd0-8e70-631066926d34\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"
Sep 18 16:32:07.056583 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:07.056562    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdnsb\" (UniqueName: \"kubernetes.io/projected/89932c02-ee32-4fd0-8e70-631066926d34-kube-api-access-bdnsb\") pod \"isvc-canary-rollback-predictor-7f7b644c9-sxhhv\" (UID: \"89932c02-ee32-4fd0-8e70-631066926d34\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"
Sep 18 16:32:07.089550 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:07.089519    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"
Sep 18 16:32:07.394320 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:07.394283    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"]
Sep 18 16:32:07.398813 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:32:07.398791    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89932c02_ee32_4fd0_8e70_631066926d34.slice/crio-aeed7cea3bfcbc9029b982c15ebb0c1df7b657957e914f8111d4a67cc66de1b6 WatchSource:0}: Error finding container aeed7cea3bfcbc9029b982c15ebb0c1df7b657957e914f8111d4a67cc66de1b6: Status 404 returned error can't find the container with id aeed7cea3bfcbc9029b982c15ebb0c1df7b657957e914f8111d4a67cc66de1b6
Sep 18 16:32:07.451033 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:07.451003    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a799c2d6-f36f-41a9-9104-db5ed9e102c2-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf\" (UID: \"a799c2d6-f36f-41a9-9104-db5ed9e102c2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"
Sep 18 16:32:07.453368 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:07.453328    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a799c2d6-f36f-41a9-9104-db5ed9e102c2-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf\" (UID: \"a799c2d6-f36f-41a9-9104-db5ed9e102c2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"
Sep 18 16:32:07.662849 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:07.662773    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"
Sep 18 16:32:07.938754 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:07.938655    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"]
Sep 18 16:32:07.941251 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:32:07.941225    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda799c2d6_f36f_41a9_9104_db5ed9e102c2.slice/crio-43b6d291dcebc2bf4a5c6b506e1935e660dd4161fb24daa083c1e7646bd2bee6 WatchSource:0}: Error finding container 43b6d291dcebc2bf4a5c6b506e1935e660dd4161fb24daa083c1e7646bd2bee6: Status 404 returned error can't find the container with id 43b6d291dcebc2bf4a5c6b506e1935e660dd4161fb24daa083c1e7646bd2bee6
Sep 18 16:32:08.187657 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:08.187619    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf" event={"ID":"a799c2d6-f36f-41a9-9104-db5ed9e102c2","Type":"ContainerStarted","Data":"24922351a96f977dfc1e2a54aaeb8fcb156c1897b7f6b253f15bd61d40d8f180"}
Sep 18 16:32:08.187870 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:08.187697    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf" event={"ID":"a799c2d6-f36f-41a9-9104-db5ed9e102c2","Type":"ContainerStarted","Data":"43b6d291dcebc2bf4a5c6b506e1935e660dd4161fb24daa083c1e7646bd2bee6"}
Sep 18 16:32:08.189286 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:08.189215    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv" event={"ID":"89932c02-ee32-4fd0-8e70-631066926d34","Type":"ContainerStarted","Data":"f2d1cb29eef0aa7119e12046541d9bccb989801ef84862ca598e73f948b63afa"}
Sep 18 16:32:08.189286 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:08.189274    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv" event={"ID":"89932c02-ee32-4fd0-8e70-631066926d34","Type":"ContainerStarted","Data":"aeed7cea3bfcbc9029b982c15ebb0c1df7b657957e914f8111d4a67cc66de1b6"}
Sep 18 16:32:11.203160 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:11.203126    2566 generic.go:356] "Generic (PLEG): container finished" podID="89932c02-ee32-4fd0-8e70-631066926d34" containerID="f2d1cb29eef0aa7119e12046541d9bccb989801ef84862ca598e73f948b63afa" exitCode=0
Sep 18 16:32:11.203505 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:11.203201    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv" event={"ID":"89932c02-ee32-4fd0-8e70-631066926d34","Type":"ContainerDied","Data":"f2d1cb29eef0aa7119e12046541d9bccb989801ef84862ca598e73f948b63afa"}
Sep 18 16:32:12.212404 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:12.212369    2566 generic.go:356] "Generic (PLEG): container finished" podID="a799c2d6-f36f-41a9-9104-db5ed9e102c2" containerID="24922351a96f977dfc1e2a54aaeb8fcb156c1897b7f6b253f15bd61d40d8f180" exitCode=0
Sep 18 16:32:12.212953 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:12.212444    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf" event={"ID":"a799c2d6-f36f-41a9-9104-db5ed9e102c2","Type":"ContainerDied","Data":"24922351a96f977dfc1e2a54aaeb8fcb156c1897b7f6b253f15bd61d40d8f180"}
Sep 18 16:32:29.321539 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:29.321495    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv" event={"ID":"89932c02-ee32-4fd0-8e70-631066926d34","Type":"ContainerStarted","Data":"ab66d9723213f889673e27ba36bed9015945e78762a49f8c6ffbcdd69ec80ceb"}
Sep 18 16:32:29.323800 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:29.323771    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf" event={"ID":"a799c2d6-f36f-41a9-9104-db5ed9e102c2","Type":"ContainerStarted","Data":"2fc0a9559d274cc462f146a162e21a474a2128cfeffb22fbf51065f3c101f27c"}
Sep 18 16:32:32.345263 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:32.345221    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf" event={"ID":"a799c2d6-f36f-41a9-9104-db5ed9e102c2","Type":"ContainerStarted","Data":"26488b8c9a1706888240eed5ddc741597a83e95e5df29668796e32ad3882d521"}
Sep 18 16:32:32.345730 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:32.345347    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"
Sep 18 16:32:32.345730 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:32.345407    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"
Sep 18 16:32:32.346902 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:32.346876    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf" podUID="a799c2d6-f36f-41a9-9104-db5ed9e102c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 18 16:32:32.347400 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:32.347374    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv" event={"ID":"89932c02-ee32-4fd0-8e70-631066926d34","Type":"ContainerStarted","Data":"2807964553ba825f981c29bc2435b7515a980438ee253fad828d2d9c511b3485"}
Sep 18 16:32:32.347520 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:32.347505    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"
Sep 18 16:32:32.347587 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:32.347541    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"
Sep 18 16:32:32.348501 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:32.348477    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv" podUID="89932c02-ee32-4fd0-8e70-631066926d34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 18 16:32:32.365217 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:32.365175    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf" podStartSLOduration=3.415939442 podStartE2EDuration="26.365162511s" totalImagesPullingTime="19.149264807s" totalInitContainerRuntime="3.799958262s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:32:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:32:32.363792099 +0000 UTC m=+990.819245177" watchObservedRunningTime="2026-09-18 16:32:32.365162511 +0000 UTC m=+990.820615625"
Sep 18 16:32:32.383816 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:32.383776    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv" podStartSLOduration=2.57880713 podStartE2EDuration="26.383764019s" totalImagesPullingTime="20.097156427s" totalInitContainerRuntime="3.707800462s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:32:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:32:32.382907209 +0000 UTC m=+990.838360288" watchObservedRunningTime="2026-09-18 16:32:32.383764019 +0000 UTC m=+990.839217147"
Sep 18 16:32:33.351350 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:33.351309    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf" podUID="a799c2d6-f36f-41a9-9104-db5ed9e102c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 18 16:32:33.351725 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:33.351415    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv" podUID="89932c02-ee32-4fd0-8e70-631066926d34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 18 16:32:38.356151 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:38.356121    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"
Sep 18 16:32:38.356597 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:38.356215    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"
Sep 18 16:32:38.356675 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:38.356646    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv" podUID="89932c02-ee32-4fd0-8e70-631066926d34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 18 16:32:38.356779 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:38.356753    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf" podUID="a799c2d6-f36f-41a9-9104-db5ed9e102c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 18 16:32:48.356707 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:48.356665    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv" podUID="89932c02-ee32-4fd0-8e70-631066926d34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 18 16:32:48.357173 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:48.356665    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf" podUID="a799c2d6-f36f-41a9-9104-db5ed9e102c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 18 16:32:58.356610 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:58.356568    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv" podUID="89932c02-ee32-4fd0-8e70-631066926d34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 18 16:32:58.357093 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:32:58.356568    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf" podUID="a799c2d6-f36f-41a9-9104-db5ed9e102c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 18 16:33:08.357244 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:33:08.357202    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv" podUID="89932c02-ee32-4fd0-8e70-631066926d34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 18 16:33:08.357721 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:33:08.357210    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf" podUID="a799c2d6-f36f-41a9-9104-db5ed9e102c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 18 16:33:18.357305 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:33:18.357266    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf" podUID="a799c2d6-f36f-41a9-9104-db5ed9e102c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 18 16:33:18.357305 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:33:18.357290    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv" podUID="89932c02-ee32-4fd0-8e70-631066926d34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 18 16:33:28.356713 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:33:28.356667    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf" podUID="a799c2d6-f36f-41a9-9104-db5ed9e102c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 18 16:33:28.357102 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:33:28.356667    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv" podUID="89932c02-ee32-4fd0-8e70-631066926d34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 18 16:33:38.357237 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:33:38.357200    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv" podUID="89932c02-ee32-4fd0-8e70-631066926d34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 18 16:33:38.357607 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:33:38.357202    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf" podUID="a799c2d6-f36f-41a9-9104-db5ed9e102c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 18 16:33:47.686797 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:33:47.686718    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"
Sep 18 16:33:48.357681 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:33:48.357645    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"
Sep 18 16:33:56.681240 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:33:56.681204    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"]
Sep 18 16:33:56.681779 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:33:56.681521    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf" podUID="a799c2d6-f36f-41a9-9104-db5ed9e102c2" containerName="kserve-container" containerID="cri-o://2fc0a9559d274cc462f146a162e21a474a2128cfeffb22fbf51065f3c101f27c" gracePeriod=30
Sep 18 16:33:56.681779 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:33:56.681591    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf" podUID="a799c2d6-f36f-41a9-9104-db5ed9e102c2" containerName="kube-rbac-proxy" containerID="cri-o://26488b8c9a1706888240eed5ddc741597a83e95e5df29668796e32ad3882d521" gracePeriod=30
Sep 18 16:33:57.694328 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:33:57.694298    2566 generic.go:356] "Generic (PLEG): container finished" podID="a799c2d6-f36f-41a9-9104-db5ed9e102c2" containerID="26488b8c9a1706888240eed5ddc741597a83e95e5df29668796e32ad3882d521" exitCode=2
Sep 18 16:33:57.694698 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:33:57.694378    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf" event={"ID":"a799c2d6-f36f-41a9-9104-db5ed9e102c2","Type":"ContainerDied","Data":"26488b8c9a1706888240eed5ddc741597a83e95e5df29668796e32ad3882d521"}
Sep 18 16:33:58.352415 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:33:58.352371    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf" podUID="a799c2d6-f36f-41a9-9104-db5ed9e102c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 18 16:33:58.356715 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:33:58.356684    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf" podUID="a799c2d6-f36f-41a9-9104-db5ed9e102c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 18 16:34:01.618703 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.618672    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"
Sep 18 16:34:01.669447 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.669417    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a799c2d6-f36f-41a9-9104-db5ed9e102c2-cabundle-cert\") pod \"a799c2d6-f36f-41a9-9104-db5ed9e102c2\" (UID: \"a799c2d6-f36f-41a9-9104-db5ed9e102c2\") "
Sep 18 16:34:01.669633 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.669457    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhlnn\" (UniqueName: \"kubernetes.io/projected/a799c2d6-f36f-41a9-9104-db5ed9e102c2-kube-api-access-fhlnn\") pod \"a799c2d6-f36f-41a9-9104-db5ed9e102c2\" (UID: \"a799c2d6-f36f-41a9-9104-db5ed9e102c2\") "
Sep 18 16:34:01.669633 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.669483    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a799c2d6-f36f-41a9-9104-db5ed9e102c2-kserve-provision-location\") pod \"a799c2d6-f36f-41a9-9104-db5ed9e102c2\" (UID: \"a799c2d6-f36f-41a9-9104-db5ed9e102c2\") "
Sep 18 16:34:01.669633 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.669505    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a799c2d6-f36f-41a9-9104-db5ed9e102c2-kube-rbac-proxy-sar-config\") pod \"a799c2d6-f36f-41a9-9104-db5ed9e102c2\" (UID: \"a799c2d6-f36f-41a9-9104-db5ed9e102c2\") "
Sep 18 16:34:01.669633 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.669551    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a799c2d6-f36f-41a9-9104-db5ed9e102c2-proxy-tls\") pod \"a799c2d6-f36f-41a9-9104-db5ed9e102c2\" (UID: \"a799c2d6-f36f-41a9-9104-db5ed9e102c2\") "
Sep 18 16:34:01.669851 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.669796    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a799c2d6-f36f-41a9-9104-db5ed9e102c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a799c2d6-f36f-41a9-9104-db5ed9e102c2" (UID: "a799c2d6-f36f-41a9-9104-db5ed9e102c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:01.669895 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.669839    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a799c2d6-f36f-41a9-9104-db5ed9e102c2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a799c2d6-f36f-41a9-9104-db5ed9e102c2" (UID: "a799c2d6-f36f-41a9-9104-db5ed9e102c2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:34:01.669895 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.669864    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a799c2d6-f36f-41a9-9104-db5ed9e102c2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a799c2d6-f36f-41a9-9104-db5ed9e102c2" (UID: "a799c2d6-f36f-41a9-9104-db5ed9e102c2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:34:01.671701 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.671677    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a799c2d6-f36f-41a9-9104-db5ed9e102c2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a799c2d6-f36f-41a9-9104-db5ed9e102c2" (UID: "a799c2d6-f36f-41a9-9104-db5ed9e102c2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:34:01.671791 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.671726    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a799c2d6-f36f-41a9-9104-db5ed9e102c2-kube-api-access-fhlnn" (OuterVolumeSpecName: "kube-api-access-fhlnn") pod "a799c2d6-f36f-41a9-9104-db5ed9e102c2" (UID: "a799c2d6-f36f-41a9-9104-db5ed9e102c2"). InnerVolumeSpecName "kube-api-access-fhlnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:34:01.712504 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.712464    2566 generic.go:356] "Generic (PLEG): container finished" podID="a799c2d6-f36f-41a9-9104-db5ed9e102c2" containerID="2fc0a9559d274cc462f146a162e21a474a2128cfeffb22fbf51065f3c101f27c" exitCode=0
Sep 18 16:34:01.712641 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.712527    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"
Sep 18 16:34:01.712641 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.712539    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf" event={"ID":"a799c2d6-f36f-41a9-9104-db5ed9e102c2","Type":"ContainerDied","Data":"2fc0a9559d274cc462f146a162e21a474a2128cfeffb22fbf51065f3c101f27c"}
Sep 18 16:34:01.712641 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.712607    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf" event={"ID":"a799c2d6-f36f-41a9-9104-db5ed9e102c2","Type":"ContainerDied","Data":"43b6d291dcebc2bf4a5c6b506e1935e660dd4161fb24daa083c1e7646bd2bee6"}
Sep 18 16:34:01.712641 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.712634    2566 scope.go:117] "RemoveContainer" containerID="26488b8c9a1706888240eed5ddc741597a83e95e5df29668796e32ad3882d521"
Sep 18 16:34:01.720488 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.720469    2566 scope.go:117] "RemoveContainer" containerID="2fc0a9559d274cc462f146a162e21a474a2128cfeffb22fbf51065f3c101f27c"
Sep 18 16:34:01.727554 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.727530    2566 scope.go:117] "RemoveContainer" containerID="24922351a96f977dfc1e2a54aaeb8fcb156c1897b7f6b253f15bd61d40d8f180"
Sep 18 16:34:01.731172 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.731147    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"]
Sep 18 16:34:01.735653 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.735633    2566 scope.go:117] "RemoveContainer" containerID="26488b8c9a1706888240eed5ddc741597a83e95e5df29668796e32ad3882d521"
Sep 18 16:34:01.735967 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.735950    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-nzrtf"]
Sep 18 16:34:01.736009 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:34:01.735951    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26488b8c9a1706888240eed5ddc741597a83e95e5df29668796e32ad3882d521\": container with ID starting with 26488b8c9a1706888240eed5ddc741597a83e95e5df29668796e32ad3882d521 not found: ID does not exist" containerID="26488b8c9a1706888240eed5ddc741597a83e95e5df29668796e32ad3882d521"
Sep 18 16:34:01.736045 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.736012    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26488b8c9a1706888240eed5ddc741597a83e95e5df29668796e32ad3882d521"} err="failed to get container status \"26488b8c9a1706888240eed5ddc741597a83e95e5df29668796e32ad3882d521\": rpc error: code = NotFound desc = could not find container \"26488b8c9a1706888240eed5ddc741597a83e95e5df29668796e32ad3882d521\": container with ID starting with 26488b8c9a1706888240eed5ddc741597a83e95e5df29668796e32ad3882d521 not found: ID does not exist"
Sep 18 16:34:01.736045 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.736032    2566 scope.go:117] "RemoveContainer" containerID="2fc0a9559d274cc462f146a162e21a474a2128cfeffb22fbf51065f3c101f27c"
Sep 18 16:34:01.736311 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:34:01.736293    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc0a9559d274cc462f146a162e21a474a2128cfeffb22fbf51065f3c101f27c\": container with ID starting with 2fc0a9559d274cc462f146a162e21a474a2128cfeffb22fbf51065f3c101f27c not found: ID does not exist" containerID="2fc0a9559d274cc462f146a162e21a474a2128cfeffb22fbf51065f3c101f27c"
Sep 18 16:34:01.736368 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.736317    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc0a9559d274cc462f146a162e21a474a2128cfeffb22fbf51065f3c101f27c"} err="failed to get container status \"2fc0a9559d274cc462f146a162e21a474a2128cfeffb22fbf51065f3c101f27c\": rpc error: code = NotFound desc = could not find container \"2fc0a9559d274cc462f146a162e21a474a2128cfeffb22fbf51065f3c101f27c\": container with ID starting with 2fc0a9559d274cc462f146a162e21a474a2128cfeffb22fbf51065f3c101f27c not found: ID does not exist"
Sep 18 16:34:01.736368 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.736334    2566 scope.go:117] "RemoveContainer" containerID="24922351a96f977dfc1e2a54aaeb8fcb156c1897b7f6b253f15bd61d40d8f180"
Sep 18 16:34:01.736578 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:34:01.736565    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24922351a96f977dfc1e2a54aaeb8fcb156c1897b7f6b253f15bd61d40d8f180\": container with ID starting with 24922351a96f977dfc1e2a54aaeb8fcb156c1897b7f6b253f15bd61d40d8f180 not found: ID does not exist" containerID="24922351a96f977dfc1e2a54aaeb8fcb156c1897b7f6b253f15bd61d40d8f180"
Sep 18 16:34:01.736616 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.736580    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24922351a96f977dfc1e2a54aaeb8fcb156c1897b7f6b253f15bd61d40d8f180"} err="failed to get container status \"24922351a96f977dfc1e2a54aaeb8fcb156c1897b7f6b253f15bd61d40d8f180\": rpc error: code = NotFound desc = could not find container \"24922351a96f977dfc1e2a54aaeb8fcb156c1897b7f6b253f15bd61d40d8f180\": container with ID starting with 24922351a96f977dfc1e2a54aaeb8fcb156c1897b7f6b253f15bd61d40d8f180 not found: ID does not exist"
Sep 18 16:34:01.771209 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.771177    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a799c2d6-f36f-41a9-9104-db5ed9e102c2-cabundle-cert\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:34:01.771209 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.771206    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhlnn\" (UniqueName: \"kubernetes.io/projected/a799c2d6-f36f-41a9-9104-db5ed9e102c2-kube-api-access-fhlnn\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:34:01.771352 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.771216    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a799c2d6-f36f-41a9-9104-db5ed9e102c2-kserve-provision-location\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:34:01.771352 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.771226    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a799c2d6-f36f-41a9-9104-db5ed9e102c2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:34:01.771352 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:01.771234    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a799c2d6-f36f-41a9-9104-db5ed9e102c2-proxy-tls\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.196009 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.195976    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"]
Sep 18 16:34:02.196336 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.196323    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a799c2d6-f36f-41a9-9104-db5ed9e102c2" containerName="kube-rbac-proxy"
Sep 18 16:34:02.196383 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.196339    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a799c2d6-f36f-41a9-9104-db5ed9e102c2" containerName="kube-rbac-proxy"
Sep 18 16:34:02.196383 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.196350    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a799c2d6-f36f-41a9-9104-db5ed9e102c2" containerName="storage-initializer"
Sep 18 16:34:02.196383 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.196356    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a799c2d6-f36f-41a9-9104-db5ed9e102c2" containerName="storage-initializer"
Sep 18 16:34:02.196383 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.196372    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a799c2d6-f36f-41a9-9104-db5ed9e102c2" containerName="kserve-container"
Sep 18 16:34:02.196383 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.196378    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a799c2d6-f36f-41a9-9104-db5ed9e102c2" containerName="kserve-container"
Sep 18 16:34:02.196529 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.196430    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a799c2d6-f36f-41a9-9104-db5ed9e102c2" containerName="kserve-container"
Sep 18 16:34:02.196529 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.196439    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a799c2d6-f36f-41a9-9104-db5ed9e102c2" containerName="kube-rbac-proxy"
Sep 18 16:34:02.200736 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.200719    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"
Sep 18 16:34:02.203090 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.203047    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-0a5a4-predictor-serving-cert\""
Sep 18 16:34:02.203216 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.203187    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-0a5a4-kube-rbac-proxy-sar-config\""
Sep 18 16:34:02.211834 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.211806    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"]
Sep 18 16:34:02.275359 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.275325    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b73b611-6608-4cb4-9d17-986e245e01ec-proxy-tls\") pod \"isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl\" (UID: \"6b73b611-6608-4cb4-9d17-986e245e01ec\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"
Sep 18 16:34:02.275561 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.275363    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b73b611-6608-4cb4-9d17-986e245e01ec-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl\" (UID: \"6b73b611-6608-4cb4-9d17-986e245e01ec\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"
Sep 18 16:34:02.275561 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.275398    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b73b611-6608-4cb4-9d17-986e245e01ec-cabundle-cert\") pod \"isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl\" (UID: \"6b73b611-6608-4cb4-9d17-986e245e01ec\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"
Sep 18 16:34:02.275561 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.275516    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvx5j\" (UniqueName: \"kubernetes.io/projected/6b73b611-6608-4cb4-9d17-986e245e01ec-kube-api-access-wvx5j\") pod \"isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl\" (UID: \"6b73b611-6608-4cb4-9d17-986e245e01ec\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"
Sep 18 16:34:02.275734 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.275575    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b73b611-6608-4cb4-9d17-986e245e01ec-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl\" (UID: \"6b73b611-6608-4cb4-9d17-986e245e01ec\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"
Sep 18 16:34:02.376601 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.376564    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b73b611-6608-4cb4-9d17-986e245e01ec-cabundle-cert\") pod \"isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl\" (UID: \"6b73b611-6608-4cb4-9d17-986e245e01ec\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"
Sep 18 16:34:02.376821 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.376629    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvx5j\" (UniqueName: \"kubernetes.io/projected/6b73b611-6608-4cb4-9d17-986e245e01ec-kube-api-access-wvx5j\") pod \"isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl\" (UID: \"6b73b611-6608-4cb4-9d17-986e245e01ec\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"
Sep 18 16:34:02.376821 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.376679    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b73b611-6608-4cb4-9d17-986e245e01ec-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl\" (UID: \"6b73b611-6608-4cb4-9d17-986e245e01ec\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"
Sep 18 16:34:02.376821 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.376754    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b73b611-6608-4cb4-9d17-986e245e01ec-proxy-tls\") pod \"isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl\" (UID: \"6b73b611-6608-4cb4-9d17-986e245e01ec\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"
Sep 18 16:34:02.376821 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.376782    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b73b611-6608-4cb4-9d17-986e245e01ec-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl\" (UID: \"6b73b611-6608-4cb4-9d17-986e245e01ec\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"
Sep 18 16:34:02.377043 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:34:02.376987    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-serving-cert: secret "isvc-sklearn-graph-raw-0a5a4-predictor-serving-cert" not found
Sep 18 16:34:02.377117 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:34:02.377061    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b73b611-6608-4cb4-9d17-986e245e01ec-proxy-tls podName:6b73b611-6608-4cb4-9d17-986e245e01ec nodeName:}" failed. No retries permitted until 2026-09-18 16:34:02.877042369 +0000 UTC m=+1081.332495427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6b73b611-6608-4cb4-9d17-986e245e01ec-proxy-tls") pod "isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl" (UID: "6b73b611-6608-4cb4-9d17-986e245e01ec") : secret "isvc-sklearn-graph-raw-0a5a4-predictor-serving-cert" not found
Sep 18 16:34:02.377190 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.377157    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b73b611-6608-4cb4-9d17-986e245e01ec-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl\" (UID: \"6b73b611-6608-4cb4-9d17-986e245e01ec\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"
Sep 18 16:34:02.377407 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.377382    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b73b611-6608-4cb4-9d17-986e245e01ec-cabundle-cert\") pod \"isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl\" (UID: \"6b73b611-6608-4cb4-9d17-986e245e01ec\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"
Sep 18 16:34:02.377566 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.377544    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b73b611-6608-4cb4-9d17-986e245e01ec-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl\" (UID: \"6b73b611-6608-4cb4-9d17-986e245e01ec\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"
Sep 18 16:34:02.392457 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.392417    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"]
Sep 18 16:34:02.394827 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.394798    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvx5j\" (UniqueName: \"kubernetes.io/projected/6b73b611-6608-4cb4-9d17-986e245e01ec-kube-api-access-wvx5j\") pod \"isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl\" (UID: \"6b73b611-6608-4cb4-9d17-986e245e01ec\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"
Sep 18 16:34:02.396595 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.396579    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"
Sep 18 16:34:02.401821 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.401797    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-0a5a4-predictor-serving-cert\""
Sep 18 16:34:02.401956 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.401820    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-0a5a4-kube-rbac-proxy-sar-config\""
Sep 18 16:34:02.419363 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.419331    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"]
Sep 18 16:34:02.477435 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.477345    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz78d\" (UniqueName: \"kubernetes.io/projected/65f04b2a-0186-41a3-87d8-44be882ad432-kube-api-access-zz78d\") pod \"isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn\" (UID: \"65f04b2a-0186-41a3-87d8-44be882ad432\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"
Sep 18 16:34:02.477435 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.477427    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65f04b2a-0186-41a3-87d8-44be882ad432-proxy-tls\") pod \"isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn\" (UID: \"65f04b2a-0186-41a3-87d8-44be882ad432\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"
Sep 18 16:34:02.477657 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.477454    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65f04b2a-0186-41a3-87d8-44be882ad432-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn\" (UID: \"65f04b2a-0186-41a3-87d8-44be882ad432\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"
Sep 18 16:34:02.477657 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.477489    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65f04b2a-0186-41a3-87d8-44be882ad432-cabundle-cert\") pod \"isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn\" (UID: \"65f04b2a-0186-41a3-87d8-44be882ad432\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"
Sep 18 16:34:02.477657 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.477514    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65f04b2a-0186-41a3-87d8-44be882ad432-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn\" (UID: \"65f04b2a-0186-41a3-87d8-44be882ad432\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"
Sep 18 16:34:02.577978 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.577949    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65f04b2a-0186-41a3-87d8-44be882ad432-proxy-tls\") pod \"isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn\" (UID: \"65f04b2a-0186-41a3-87d8-44be882ad432\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"
Sep 18 16:34:02.577978 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.577981    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65f04b2a-0186-41a3-87d8-44be882ad432-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn\" (UID: \"65f04b2a-0186-41a3-87d8-44be882ad432\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"
Sep 18 16:34:02.578254 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.578019    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65f04b2a-0186-41a3-87d8-44be882ad432-cabundle-cert\") pod \"isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn\" (UID: \"65f04b2a-0186-41a3-87d8-44be882ad432\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"
Sep 18 16:34:02.578254 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.578046    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65f04b2a-0186-41a3-87d8-44be882ad432-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn\" (UID: \"65f04b2a-0186-41a3-87d8-44be882ad432\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"
Sep 18 16:34:02.578254 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:34:02.578137    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-serving-cert: secret "isvc-xgboost-graph-raw-0a5a4-predictor-serving-cert" not found
Sep 18 16:34:02.578254 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.578146    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz78d\" (UniqueName: \"kubernetes.io/projected/65f04b2a-0186-41a3-87d8-44be882ad432-kube-api-access-zz78d\") pod \"isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn\" (UID: \"65f04b2a-0186-41a3-87d8-44be882ad432\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"
Sep 18 16:34:02.578254 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:34:02.578205    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65f04b2a-0186-41a3-87d8-44be882ad432-proxy-tls podName:65f04b2a-0186-41a3-87d8-44be882ad432 nodeName:}" failed. No retries permitted until 2026-09-18 16:34:03.078183718 +0000 UTC m=+1081.533636793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/65f04b2a-0186-41a3-87d8-44be882ad432-proxy-tls") pod "isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn" (UID: "65f04b2a-0186-41a3-87d8-44be882ad432") : secret "isvc-xgboost-graph-raw-0a5a4-predictor-serving-cert" not found
Sep 18 16:34:02.578555 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.578529    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65f04b2a-0186-41a3-87d8-44be882ad432-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn\" (UID: \"65f04b2a-0186-41a3-87d8-44be882ad432\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"
Sep 18 16:34:02.578780 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.578759    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65f04b2a-0186-41a3-87d8-44be882ad432-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn\" (UID: \"65f04b2a-0186-41a3-87d8-44be882ad432\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"
Sep 18 16:34:02.578780 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.578769    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65f04b2a-0186-41a3-87d8-44be882ad432-cabundle-cert\") pod \"isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn\" (UID: \"65f04b2a-0186-41a3-87d8-44be882ad432\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"
Sep 18 16:34:02.588770 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.588742    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz78d\" (UniqueName: \"kubernetes.io/projected/65f04b2a-0186-41a3-87d8-44be882ad432-kube-api-access-zz78d\") pod \"isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn\" (UID: \"65f04b2a-0186-41a3-87d8-44be882ad432\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"
Sep 18 16:34:02.881915 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.881881    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b73b611-6608-4cb4-9d17-986e245e01ec-proxy-tls\") pod \"isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl\" (UID: \"6b73b611-6608-4cb4-9d17-986e245e01ec\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"
Sep 18 16:34:02.884266 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:02.884247    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b73b611-6608-4cb4-9d17-986e245e01ec-proxy-tls\") pod \"isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl\" (UID: \"6b73b611-6608-4cb4-9d17-986e245e01ec\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"
Sep 18 16:34:03.083792 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:03.083752    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65f04b2a-0186-41a3-87d8-44be882ad432-proxy-tls\") pod \"isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn\" (UID: \"65f04b2a-0186-41a3-87d8-44be882ad432\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"
Sep 18 16:34:03.086140 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:03.086118    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65f04b2a-0186-41a3-87d8-44be882ad432-proxy-tls\") pod \"isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn\" (UID: \"65f04b2a-0186-41a3-87d8-44be882ad432\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"
Sep 18 16:34:03.110718 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:03.110684    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"
Sep 18 16:34:03.309925 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:03.309878    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"
Sep 18 16:34:03.387018 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:03.386989    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"]
Sep 18 16:34:03.391988 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:34:03.389868    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b73b611_6608_4cb4_9d17_986e245e01ec.slice/crio-2703a062d275742d54eb2166a7414c883417cc3866166737834d87c55400a5d2 WatchSource:0}: Error finding container 2703a062d275742d54eb2166a7414c883417cc3866166737834d87c55400a5d2: Status 404 returned error can't find the container with id 2703a062d275742d54eb2166a7414c883417cc3866166737834d87c55400a5d2
Sep 18 16:34:03.584330 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:03.584296    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"]
Sep 18 16:34:03.587798 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:34:03.587764    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f04b2a_0186_41a3_87d8_44be882ad432.slice/crio-86bed98850938e10e0003f83a9b784b1b223a60aed2e885d1b1db240725a04ad WatchSource:0}: Error finding container 86bed98850938e10e0003f83a9b784b1b223a60aed2e885d1b1db240725a04ad: Status 404 returned error can't find the container with id 86bed98850938e10e0003f83a9b784b1b223a60aed2e885d1b1db240725a04ad
Sep 18 16:34:03.688347 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:03.688277    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a799c2d6-f36f-41a9-9104-db5ed9e102c2" path="/var/lib/kubelet/pods/a799c2d6-f36f-41a9-9104-db5ed9e102c2/volumes"
Sep 18 16:34:03.724703 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:03.724670    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl" event={"ID":"6b73b611-6608-4cb4-9d17-986e245e01ec","Type":"ContainerStarted","Data":"8352dff08ffadd238bf469135c759aee4777ebd4319e2424922c0d9f84e54a63"}
Sep 18 16:34:03.724860 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:03.724740    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl" event={"ID":"6b73b611-6608-4cb4-9d17-986e245e01ec","Type":"ContainerStarted","Data":"2703a062d275742d54eb2166a7414c883417cc3866166737834d87c55400a5d2"}
Sep 18 16:34:03.726644 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:03.726615    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn" event={"ID":"65f04b2a-0186-41a3-87d8-44be882ad432","Type":"ContainerStarted","Data":"0a887c94f4a9506ecee52fdcd4a34fe329688d424cbb9240e92cb504e8fc0388"}
Sep 18 16:34:03.726743 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:03.726687    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn" event={"ID":"65f04b2a-0186-41a3-87d8-44be882ad432","Type":"ContainerStarted","Data":"86bed98850938e10e0003f83a9b784b1b223a60aed2e885d1b1db240725a04ad"}
Sep 18 16:34:07.744383 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:07.744299    2566 generic.go:356] "Generic (PLEG): container finished" podID="6b73b611-6608-4cb4-9d17-986e245e01ec" containerID="8352dff08ffadd238bf469135c759aee4777ebd4319e2424922c0d9f84e54a63" exitCode=0
Sep 18 16:34:07.744783 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:07.744375    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl" event={"ID":"6b73b611-6608-4cb4-9d17-986e245e01ec","Type":"ContainerDied","Data":"8352dff08ffadd238bf469135c759aee4777ebd4319e2424922c0d9f84e54a63"}
Sep 18 16:34:07.745979 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:07.745952    2566 generic.go:356] "Generic (PLEG): container finished" podID="65f04b2a-0186-41a3-87d8-44be882ad432" containerID="0a887c94f4a9506ecee52fdcd4a34fe329688d424cbb9240e92cb504e8fc0388" exitCode=0
Sep 18 16:34:07.746105 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:07.746021    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn" event={"ID":"65f04b2a-0186-41a3-87d8-44be882ad432","Type":"ContainerDied","Data":"0a887c94f4a9506ecee52fdcd4a34fe329688d424cbb9240e92cb504e8fc0388"}
Sep 18 16:34:07.747226 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:07.747210    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:34:08.754155 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:08.754117    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl" event={"ID":"6b73b611-6608-4cb4-9d17-986e245e01ec","Type":"ContainerStarted","Data":"0798a545d7b8e0d7829b912af75b18d4adbba8a7c0ac3269d5aa0d45727fc6f1"}
Sep 18 16:34:08.754624 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:08.754218    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl" event={"ID":"6b73b611-6608-4cb4-9d17-986e245e01ec","Type":"ContainerStarted","Data":"65746e6b65d6979b0c02ccc00e09c241e5dcc5e8cd848b0ef1a4df0488795d59"}
Sep 18 16:34:08.754624 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:08.754437    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"
Sep 18 16:34:08.774555 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:08.774502    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl" podStartSLOduration=2.896575822 podStartE2EDuration="6.77448187s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.877906048s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:34:08.772679605 +0000 UTC m=+1087.228132956" watchObservedRunningTime="2026-09-18 16:34:08.77448187 +0000 UTC m=+1087.229934948"
Sep 18 16:34:09.760922 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:09.760843    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"
Sep 18 16:34:09.762635 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:09.762600    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl" podUID="6b73b611-6608-4cb4-9d17-986e245e01ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 18 16:34:10.765250 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:10.765207    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl" podUID="6b73b611-6608-4cb4-9d17-986e245e01ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 18 16:34:15.769736 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:15.769708    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"
Sep 18 16:34:15.770229 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:15.770202    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl" podUID="6b73b611-6608-4cb4-9d17-986e245e01ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 18 16:34:25.770733 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:25.770686    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl" podUID="6b73b611-6608-4cb4-9d17-986e245e01ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 18 16:34:28.849285 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:28.849253    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn" event={"ID":"65f04b2a-0186-41a3-87d8-44be882ad432","Type":"ContainerStarted","Data":"4aa22cae3b78ef639ab53dd9b01ae6c8bbb7d41d52fe975b717b71ac7ecb3f4a"}
Sep 18 16:34:28.849743 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:28.849316    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn" event={"ID":"65f04b2a-0186-41a3-87d8-44be882ad432","Type":"ContainerStarted","Data":"88f53521d65bce045c67e34a1e5267f091434890025b8bb88f97849562cfa631"}
Sep 18 16:34:28.849743 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:28.849510    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"
Sep 18 16:34:28.873002 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:28.872951    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn" podStartSLOduration=2.936544532 podStartE2EDuration="26.872936127s" totalImagesPullingTime="20.115453539s" totalInitContainerRuntime="3.820938056s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:34:28.870831995 +0000 UTC m=+1107.326285072" watchObservedRunningTime="2026-09-18 16:34:28.872936127 +0000 UTC m=+1107.328389203"
Sep 18 16:34:29.853255 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:29.853224    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"
Sep 18 16:34:29.854635 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:29.854608    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn" podUID="65f04b2a-0186-41a3-87d8-44be882ad432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 16:34:30.856586 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:30.856546    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn" podUID="65f04b2a-0186-41a3-87d8-44be882ad432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 16:34:35.770985 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:35.770941    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl" podUID="6b73b611-6608-4cb4-9d17-986e245e01ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 18 16:34:35.861198 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:35.861168    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"
Sep 18 16:34:35.861799 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:35.861774    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn" podUID="65f04b2a-0186-41a3-87d8-44be882ad432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 16:34:45.770801 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:45.770760    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl" podUID="6b73b611-6608-4cb4-9d17-986e245e01ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 18 16:34:45.861919 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:45.861881    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn" podUID="65f04b2a-0186-41a3-87d8-44be882ad432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 16:34:55.770208 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:55.770162    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl" podUID="6b73b611-6608-4cb4-9d17-986e245e01ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 18 16:34:55.862314 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:34:55.862273    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn" podUID="65f04b2a-0186-41a3-87d8-44be882ad432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 16:35:05.770722 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:05.770677    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl" podUID="6b73b611-6608-4cb4-9d17-986e245e01ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 18 16:35:05.862398 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:05.862360    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn" podUID="65f04b2a-0186-41a3-87d8-44be882ad432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 16:35:15.770441 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:15.770362    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl" podUID="6b73b611-6608-4cb4-9d17-986e245e01ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 18 16:35:15.862360 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:15.862322    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn" podUID="65f04b2a-0186-41a3-87d8-44be882ad432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 16:35:25.770470 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:25.770421    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl" podUID="6b73b611-6608-4cb4-9d17-986e245e01ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 18 16:35:25.862001 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:25.861966    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn" podUID="65f04b2a-0186-41a3-87d8-44be882ad432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 16:35:26.752225 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:26.752190    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"]
Sep 18 16:35:26.752553 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:26.752505    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv" podUID="89932c02-ee32-4fd0-8e70-631066926d34" containerName="kserve-container" containerID="cri-o://ab66d9723213f889673e27ba36bed9015945e78762a49f8c6ffbcdd69ec80ceb" gracePeriod=30
Sep 18 16:35:26.752601 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:26.752576    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv" podUID="89932c02-ee32-4fd0-8e70-631066926d34" containerName="kube-rbac-proxy" containerID="cri-o://2807964553ba825f981c29bc2435b7515a980438ee253fad828d2d9c511b3485" gracePeriod=30
Sep 18 16:35:27.097204 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:27.097172    2566 generic.go:356] "Generic (PLEG): container finished" podID="89932c02-ee32-4fd0-8e70-631066926d34" containerID="2807964553ba825f981c29bc2435b7515a980438ee253fad828d2d9c511b3485" exitCode=2
Sep 18 16:35:27.097588 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:27.097254    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv" event={"ID":"89932c02-ee32-4fd0-8e70-631066926d34","Type":"ContainerDied","Data":"2807964553ba825f981c29bc2435b7515a980438ee253fad828d2d9c511b3485"}
Sep 18 16:35:27.686171 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:27.686130    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv" podUID="89932c02-ee32-4fd0-8e70-631066926d34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 18 16:35:28.352269 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:28.352230    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv" podUID="89932c02-ee32-4fd0-8e70-631066926d34" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Sep 18 16:35:31.491400 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:31.491378    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"
Sep 18 16:35:31.658661 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:31.658586    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdnsb\" (UniqueName: \"kubernetes.io/projected/89932c02-ee32-4fd0-8e70-631066926d34-kube-api-access-bdnsb\") pod \"89932c02-ee32-4fd0-8e70-631066926d34\" (UID: \"89932c02-ee32-4fd0-8e70-631066926d34\") "
Sep 18 16:35:31.658661 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:31.658633    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89932c02-ee32-4fd0-8e70-631066926d34-kube-rbac-proxy-sar-config\") pod \"89932c02-ee32-4fd0-8e70-631066926d34\" (UID: \"89932c02-ee32-4fd0-8e70-631066926d34\") "
Sep 18 16:35:31.658836 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:31.658736    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89932c02-ee32-4fd0-8e70-631066926d34-cabundle-cert\") pod \"89932c02-ee32-4fd0-8e70-631066926d34\" (UID: \"89932c02-ee32-4fd0-8e70-631066926d34\") "
Sep 18 16:35:31.658878 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:31.658828    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89932c02-ee32-4fd0-8e70-631066926d34-kserve-provision-location\") pod \"89932c02-ee32-4fd0-8e70-631066926d34\" (UID: \"89932c02-ee32-4fd0-8e70-631066926d34\") "
Sep 18 16:35:31.658929 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:31.658878    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89932c02-ee32-4fd0-8e70-631066926d34-proxy-tls\") pod \"89932c02-ee32-4fd0-8e70-631066926d34\" (UID: \"89932c02-ee32-4fd0-8e70-631066926d34\") "
Sep 18 16:35:31.659117 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:31.659069    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89932c02-ee32-4fd0-8e70-631066926d34-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "89932c02-ee32-4fd0-8e70-631066926d34" (UID: "89932c02-ee32-4fd0-8e70-631066926d34"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:35:31.659248 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:31.659136    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89932c02-ee32-4fd0-8e70-631066926d34-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "89932c02-ee32-4fd0-8e70-631066926d34" (UID: "89932c02-ee32-4fd0-8e70-631066926d34"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:35:31.659248 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:31.659174    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89932c02-ee32-4fd0-8e70-631066926d34-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "89932c02-ee32-4fd0-8e70-631066926d34" (UID: "89932c02-ee32-4fd0-8e70-631066926d34"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:35:31.660804 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:31.660783    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89932c02-ee32-4fd0-8e70-631066926d34-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "89932c02-ee32-4fd0-8e70-631066926d34" (UID: "89932c02-ee32-4fd0-8e70-631066926d34"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:35:31.660922 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:31.660902    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89932c02-ee32-4fd0-8e70-631066926d34-kube-api-access-bdnsb" (OuterVolumeSpecName: "kube-api-access-bdnsb") pod "89932c02-ee32-4fd0-8e70-631066926d34" (UID: "89932c02-ee32-4fd0-8e70-631066926d34"). InnerVolumeSpecName "kube-api-access-bdnsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:35:31.759593 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:31.759566    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdnsb\" (UniqueName: \"kubernetes.io/projected/89932c02-ee32-4fd0-8e70-631066926d34-kube-api-access-bdnsb\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:35:31.759593 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:31.759589    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89932c02-ee32-4fd0-8e70-631066926d34-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:35:31.759593 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:31.759600    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89932c02-ee32-4fd0-8e70-631066926d34-cabundle-cert\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:35:31.759779 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:31.759610    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89932c02-ee32-4fd0-8e70-631066926d34-kserve-provision-location\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:35:31.759779 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:31.759620    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89932c02-ee32-4fd0-8e70-631066926d34-proxy-tls\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:35:32.016320 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.016248    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"]
Sep 18 16:35:32.016566 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.016554    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89932c02-ee32-4fd0-8e70-631066926d34" containerName="kserve-container"
Sep 18 16:35:32.016608 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.016568    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="89932c02-ee32-4fd0-8e70-631066926d34" containerName="kserve-container"
Sep 18 16:35:32.016608 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.016578    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89932c02-ee32-4fd0-8e70-631066926d34" containerName="kube-rbac-proxy"
Sep 18 16:35:32.016608 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.016584    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="89932c02-ee32-4fd0-8e70-631066926d34" containerName="kube-rbac-proxy"
Sep 18 16:35:32.016608 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.016607    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89932c02-ee32-4fd0-8e70-631066926d34" containerName="storage-initializer"
Sep 18 16:35:32.016728 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.016613    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="89932c02-ee32-4fd0-8e70-631066926d34" containerName="storage-initializer"
Sep 18 16:35:32.016728 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.016661    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="89932c02-ee32-4fd0-8e70-631066926d34" containerName="kserve-container"
Sep 18 16:35:32.016728 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.016668    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="89932c02-ee32-4fd0-8e70-631066926d34" containerName="kube-rbac-proxy"
Sep 18 16:35:32.019163 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.019142    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"
Sep 18 16:35:32.021741 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.021714    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 18 16:35:32.021864 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.021743    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 18 16:35:32.032222 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.032199    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"]
Sep 18 16:35:32.119405 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.119377    2566 generic.go:356] "Generic (PLEG): container finished" podID="89932c02-ee32-4fd0-8e70-631066926d34" containerID="ab66d9723213f889673e27ba36bed9015945e78762a49f8c6ffbcdd69ec80ceb" exitCode=0
Sep 18 16:35:32.119542 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.119433    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"
Sep 18 16:35:32.119542 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.119446    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv" event={"ID":"89932c02-ee32-4fd0-8e70-631066926d34","Type":"ContainerDied","Data":"ab66d9723213f889673e27ba36bed9015945e78762a49f8c6ffbcdd69ec80ceb"}
Sep 18 16:35:32.119542 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.119489    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv" event={"ID":"89932c02-ee32-4fd0-8e70-631066926d34","Type":"ContainerDied","Data":"aeed7cea3bfcbc9029b982c15ebb0c1df7b657957e914f8111d4a67cc66de1b6"}
Sep 18 16:35:32.119542 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.119515    2566 scope.go:117] "RemoveContainer" containerID="2807964553ba825f981c29bc2435b7515a980438ee253fad828d2d9c511b3485"
Sep 18 16:35:32.128059 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.128041    2566 scope.go:117] "RemoveContainer" containerID="ab66d9723213f889673e27ba36bed9015945e78762a49f8c6ffbcdd69ec80ceb"
Sep 18 16:35:32.135115 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.135096    2566 scope.go:117] "RemoveContainer" containerID="f2d1cb29eef0aa7119e12046541d9bccb989801ef84862ca598e73f948b63afa"
Sep 18 16:35:32.137147 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.137125    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"]
Sep 18 16:35:32.142231 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.142211    2566 scope.go:117] "RemoveContainer" containerID="2807964553ba825f981c29bc2435b7515a980438ee253fad828d2d9c511b3485"
Sep 18 16:35:32.142565 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:35:32.142543    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2807964553ba825f981c29bc2435b7515a980438ee253fad828d2d9c511b3485\": container with ID starting with 2807964553ba825f981c29bc2435b7515a980438ee253fad828d2d9c511b3485 not found: ID does not exist" containerID="2807964553ba825f981c29bc2435b7515a980438ee253fad828d2d9c511b3485"
Sep 18 16:35:32.142653 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.142574    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2807964553ba825f981c29bc2435b7515a980438ee253fad828d2d9c511b3485"} err="failed to get container status \"2807964553ba825f981c29bc2435b7515a980438ee253fad828d2d9c511b3485\": rpc error: code = NotFound desc = could not find container \"2807964553ba825f981c29bc2435b7515a980438ee253fad828d2d9c511b3485\": container with ID starting with 2807964553ba825f981c29bc2435b7515a980438ee253fad828d2d9c511b3485 not found: ID does not exist"
Sep 18 16:35:32.142653 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.142590    2566 scope.go:117] "RemoveContainer" containerID="ab66d9723213f889673e27ba36bed9015945e78762a49f8c6ffbcdd69ec80ceb"
Sep 18 16:35:32.142848 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:35:32.142829    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab66d9723213f889673e27ba36bed9015945e78762a49f8c6ffbcdd69ec80ceb\": container with ID starting with ab66d9723213f889673e27ba36bed9015945e78762a49f8c6ffbcdd69ec80ceb not found: ID does not exist" containerID="ab66d9723213f889673e27ba36bed9015945e78762a49f8c6ffbcdd69ec80ceb"
Sep 18 16:35:32.142907 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.142858    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab66d9723213f889673e27ba36bed9015945e78762a49f8c6ffbcdd69ec80ceb"} err="failed to get container status \"ab66d9723213f889673e27ba36bed9015945e78762a49f8c6ffbcdd69ec80ceb\": rpc error: code = NotFound desc = could not find container \"ab66d9723213f889673e27ba36bed9015945e78762a49f8c6ffbcdd69ec80ceb\": container with ID starting with ab66d9723213f889673e27ba36bed9015945e78762a49f8c6ffbcdd69ec80ceb not found: ID does not exist"
Sep 18 16:35:32.142907 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.142873    2566 scope.go:117] "RemoveContainer" containerID="f2d1cb29eef0aa7119e12046541d9bccb989801ef84862ca598e73f948b63afa"
Sep 18 16:35:32.143143 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:35:32.143128    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2d1cb29eef0aa7119e12046541d9bccb989801ef84862ca598e73f948b63afa\": container with ID starting with f2d1cb29eef0aa7119e12046541d9bccb989801ef84862ca598e73f948b63afa not found: ID does not exist" containerID="f2d1cb29eef0aa7119e12046541d9bccb989801ef84862ca598e73f948b63afa"
Sep 18 16:35:32.143212 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.143145    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2d1cb29eef0aa7119e12046541d9bccb989801ef84862ca598e73f948b63afa"} err="failed to get container status \"f2d1cb29eef0aa7119e12046541d9bccb989801ef84862ca598e73f948b63afa\": rpc error: code = NotFound desc = could not find container \"f2d1cb29eef0aa7119e12046541d9bccb989801ef84862ca598e73f948b63afa\": container with ID starting with f2d1cb29eef0aa7119e12046541d9bccb989801ef84862ca598e73f948b63afa not found: ID does not exist"
Sep 18 16:35:32.143871 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.143849    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-sxhhv"]
Sep 18 16:35:32.163532 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.163510    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03ed76b6-52be-4ecd-bbb3-755f5627619f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-q8mwv\" (UID: \"03ed76b6-52be-4ecd-bbb3-755f5627619f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"
Sep 18 16:35:32.163607 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.163556    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03ed76b6-52be-4ecd-bbb3-755f5627619f-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-q8mwv\" (UID: \"03ed76b6-52be-4ecd-bbb3-755f5627619f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"
Sep 18 16:35:32.163664 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.163650    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7zw7\" (UniqueName: \"kubernetes.io/projected/03ed76b6-52be-4ecd-bbb3-755f5627619f-kube-api-access-x7zw7\") pod \"isvc-canary-stop-predictor-5ffdccb94c-q8mwv\" (UID: \"03ed76b6-52be-4ecd-bbb3-755f5627619f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"
Sep 18 16:35:32.163700 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.163676    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03ed76b6-52be-4ecd-bbb3-755f5627619f-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-q8mwv\" (UID: \"03ed76b6-52be-4ecd-bbb3-755f5627619f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"
Sep 18 16:35:32.163700 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.163694    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03ed76b6-52be-4ecd-bbb3-755f5627619f-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-q8mwv\" (UID: \"03ed76b6-52be-4ecd-bbb3-755f5627619f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"
Sep 18 16:35:32.264854 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.264819    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7zw7\" (UniqueName: \"kubernetes.io/projected/03ed76b6-52be-4ecd-bbb3-755f5627619f-kube-api-access-x7zw7\") pod \"isvc-canary-stop-predictor-5ffdccb94c-q8mwv\" (UID: \"03ed76b6-52be-4ecd-bbb3-755f5627619f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"
Sep 18 16:35:32.264854 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.264861    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03ed76b6-52be-4ecd-bbb3-755f5627619f-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-q8mwv\" (UID: \"03ed76b6-52be-4ecd-bbb3-755f5627619f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"
Sep 18 16:35:32.264998 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.264884    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03ed76b6-52be-4ecd-bbb3-755f5627619f-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-q8mwv\" (UID: \"03ed76b6-52be-4ecd-bbb3-755f5627619f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"
Sep 18 16:35:32.264998 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.264911    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03ed76b6-52be-4ecd-bbb3-755f5627619f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-q8mwv\" (UID: \"03ed76b6-52be-4ecd-bbb3-755f5627619f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"
Sep 18 16:35:32.264998 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.264970    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03ed76b6-52be-4ecd-bbb3-755f5627619f-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-q8mwv\" (UID: \"03ed76b6-52be-4ecd-bbb3-755f5627619f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"
Sep 18 16:35:32.265297 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.265275    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03ed76b6-52be-4ecd-bbb3-755f5627619f-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-q8mwv\" (UID: \"03ed76b6-52be-4ecd-bbb3-755f5627619f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"
Sep 18 16:35:32.265611 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.265594    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03ed76b6-52be-4ecd-bbb3-755f5627619f-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-q8mwv\" (UID: \"03ed76b6-52be-4ecd-bbb3-755f5627619f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"
Sep 18 16:35:32.265652 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.265591    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03ed76b6-52be-4ecd-bbb3-755f5627619f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-q8mwv\" (UID: \"03ed76b6-52be-4ecd-bbb3-755f5627619f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"
Sep 18 16:35:32.267267 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.267227    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03ed76b6-52be-4ecd-bbb3-755f5627619f-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-q8mwv\" (UID: \"03ed76b6-52be-4ecd-bbb3-755f5627619f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"
Sep 18 16:35:32.273049 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.273027    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7zw7\" (UniqueName: \"kubernetes.io/projected/03ed76b6-52be-4ecd-bbb3-755f5627619f-kube-api-access-x7zw7\") pod \"isvc-canary-stop-predictor-5ffdccb94c-q8mwv\" (UID: \"03ed76b6-52be-4ecd-bbb3-755f5627619f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"
Sep 18 16:35:32.331775 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.331751    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"
Sep 18 16:35:32.605133 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:32.605110    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"]
Sep 18 16:35:32.607196 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:35:32.607169    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ed76b6_52be_4ecd_bbb3_755f5627619f.slice/crio-eff1cabfe1ad8cdad43412d34910b8b7b5b3edbcb4e15fc9725652454adf99e9 WatchSource:0}: Error finding container eff1cabfe1ad8cdad43412d34910b8b7b5b3edbcb4e15fc9725652454adf99e9: Status 404 returned error can't find the container with id eff1cabfe1ad8cdad43412d34910b8b7b5b3edbcb4e15fc9725652454adf99e9
Sep 18 16:35:33.128749 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:33.128706    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv" event={"ID":"03ed76b6-52be-4ecd-bbb3-755f5627619f","Type":"ContainerStarted","Data":"1d0ff445b4967aea7e3ab06e63d30e67bd36b883330db7a51954fede9c052dc7"}
Sep 18 16:35:33.128916 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:33.128822    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv" event={"ID":"03ed76b6-52be-4ecd-bbb3-755f5627619f","Type":"ContainerStarted","Data":"eff1cabfe1ad8cdad43412d34910b8b7b5b3edbcb4e15fc9725652454adf99e9"}
Sep 18 16:35:33.686190 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:33.686160    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89932c02-ee32-4fd0-8e70-631066926d34" path="/var/lib/kubelet/pods/89932c02-ee32-4fd0-8e70-631066926d34/volumes"
Sep 18 16:35:33.686613 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:33.686596    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"
Sep 18 16:35:35.863181 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:35.863151    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"
Sep 18 16:35:37.145440 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:37.145403    2566 generic.go:356] "Generic (PLEG): container finished" podID="03ed76b6-52be-4ecd-bbb3-755f5627619f" containerID="1d0ff445b4967aea7e3ab06e63d30e67bd36b883330db7a51954fede9c052dc7" exitCode=0
Sep 18 16:35:37.145774 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:37.145456    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv" event={"ID":"03ed76b6-52be-4ecd-bbb3-755f5627619f","Type":"ContainerDied","Data":"1d0ff445b4967aea7e3ab06e63d30e67bd36b883330db7a51954fede9c052dc7"}
Sep 18 16:35:38.151522 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:38.151489    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv" event={"ID":"03ed76b6-52be-4ecd-bbb3-755f5627619f","Type":"ContainerStarted","Data":"5d115a680eaf143f72e770c534099ab53f934898a126e8f313c80e5f740af697"}
Sep 18 16:35:38.151905 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:38.151548    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv" event={"ID":"03ed76b6-52be-4ecd-bbb3-755f5627619f","Type":"ContainerStarted","Data":"f1faf44b3771e5e5ef27375b51a2f564a5c62212483882e373d995c9ec24096e"}
Sep 18 16:35:38.151905 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:38.151778    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"
Sep 18 16:35:38.170665 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:38.170621    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv" podStartSLOduration=3.361320216 podStartE2EDuration="7.170607752s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.809287536s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:35:38.169194331 +0000 UTC m=+1176.624647408" watchObservedRunningTime="2026-09-18 16:35:38.170607752 +0000 UTC m=+1176.626060828"
Sep 18 16:35:39.156137 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:39.156103    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"
Sep 18 16:35:39.157463 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:39.157434    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv" podUID="03ed76b6-52be-4ecd-bbb3-755f5627619f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 18 16:35:40.159656 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:40.159614    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv" podUID="03ed76b6-52be-4ecd-bbb3-755f5627619f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 18 16:35:45.164525 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:45.164496    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"
Sep 18 16:35:45.164982 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:45.164955    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv" podUID="03ed76b6-52be-4ecd-bbb3-755f5627619f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 18 16:35:55.165960 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:35:55.165916    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv" podUID="03ed76b6-52be-4ecd-bbb3-755f5627619f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 18 16:36:02.373405 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:02.373362    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"]
Sep 18 16:36:02.373950 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:02.373644    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl" podUID="6b73b611-6608-4cb4-9d17-986e245e01ec" containerName="kserve-container" containerID="cri-o://65746e6b65d6979b0c02ccc00e09c241e5dcc5e8cd848b0ef1a4df0488795d59" gracePeriod=30
Sep 18 16:36:02.373950 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:02.373660    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl" podUID="6b73b611-6608-4cb4-9d17-986e245e01ec" containerName="kube-rbac-proxy" containerID="cri-o://0798a545d7b8e0d7829b912af75b18d4adbba8a7c0ac3269d5aa0d45727fc6f1" gracePeriod=30
Sep 18 16:36:02.413399 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:02.413365    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"]
Sep 18 16:36:02.413664 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:02.413613    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn" podUID="65f04b2a-0186-41a3-87d8-44be882ad432" containerName="kserve-container" containerID="cri-o://88f53521d65bce045c67e34a1e5267f091434890025b8bb88f97849562cfa631" gracePeriod=30
Sep 18 16:36:02.413732 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:02.413696    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn" podUID="65f04b2a-0186-41a3-87d8-44be882ad432" containerName="kube-rbac-proxy" containerID="cri-o://4aa22cae3b78ef639ab53dd9b01ae6c8bbb7d41d52fe975b717b71ac7ecb3f4a" gracePeriod=30
Sep 18 16:36:03.258173 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:03.258139    2566 generic.go:356] "Generic (PLEG): container finished" podID="6b73b611-6608-4cb4-9d17-986e245e01ec" containerID="0798a545d7b8e0d7829b912af75b18d4adbba8a7c0ac3269d5aa0d45727fc6f1" exitCode=2
Sep 18 16:36:03.258341 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:03.258214    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl" event={"ID":"6b73b611-6608-4cb4-9d17-986e245e01ec","Type":"ContainerDied","Data":"0798a545d7b8e0d7829b912af75b18d4adbba8a7c0ac3269d5aa0d45727fc6f1"}
Sep 18 16:36:03.260318 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:03.260296    2566 generic.go:356] "Generic (PLEG): container finished" podID="65f04b2a-0186-41a3-87d8-44be882ad432" containerID="4aa22cae3b78ef639ab53dd9b01ae6c8bbb7d41d52fe975b717b71ac7ecb3f4a" exitCode=2
Sep 18 16:36:03.260432 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:03.260324    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn" event={"ID":"65f04b2a-0186-41a3-87d8-44be882ad432","Type":"ContainerDied","Data":"4aa22cae3b78ef639ab53dd9b01ae6c8bbb7d41d52fe975b717b71ac7ecb3f4a"}
Sep 18 16:36:03.642934 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:03.642903    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-vfmdv_1e3ca1ec-6856-4afb-850e-85e3fa968b3f/console-operator/1.log"
Sep 18 16:36:03.661548 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:03.661523    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-vfmdv_1e3ca1ec-6856-4afb-850e-85e3fa968b3f/console-operator/1.log"
Sep 18 16:36:03.683663 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:03.683624    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl" podUID="6b73b611-6608-4cb4-9d17-986e245e01ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 18 16:36:05.165478 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:05.165442    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv" podUID="03ed76b6-52be-4ecd-bbb3-755f5627619f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 18 16:36:05.766154 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:05.766114    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl" podUID="6b73b611-6608-4cb4-9d17-986e245e01ec" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Sep 18 16:36:05.857120 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:05.857055    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn" podUID="65f04b2a-0186-41a3-87d8-44be882ad432" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 18 16:36:05.862426 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:05.862399    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn" podUID="65f04b2a-0186-41a3-87d8-44be882ad432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 18 16:36:06.760463 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:06.760439    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"
Sep 18 16:36:06.810123 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:06.810015    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65f04b2a-0186-41a3-87d8-44be882ad432-cabundle-cert\") pod \"65f04b2a-0186-41a3-87d8-44be882ad432\" (UID: \"65f04b2a-0186-41a3-87d8-44be882ad432\") "
Sep 18 16:36:06.810123 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:06.810094    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65f04b2a-0186-41a3-87d8-44be882ad432-kserve-provision-location\") pod \"65f04b2a-0186-41a3-87d8-44be882ad432\" (UID: \"65f04b2a-0186-41a3-87d8-44be882ad432\") "
Sep 18 16:36:06.810300 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:06.810131    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zz78d\" (UniqueName: \"kubernetes.io/projected/65f04b2a-0186-41a3-87d8-44be882ad432-kube-api-access-zz78d\") pod \"65f04b2a-0186-41a3-87d8-44be882ad432\" (UID: \"65f04b2a-0186-41a3-87d8-44be882ad432\") "
Sep 18 16:36:06.810300 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:06.810162    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65f04b2a-0186-41a3-87d8-44be882ad432-kube-rbac-proxy-sar-config\") pod \"65f04b2a-0186-41a3-87d8-44be882ad432\" (UID: \"65f04b2a-0186-41a3-87d8-44be882ad432\") "
Sep 18 16:36:06.810300 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:06.810239    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65f04b2a-0186-41a3-87d8-44be882ad432-proxy-tls\") pod \"65f04b2a-0186-41a3-87d8-44be882ad432\" (UID: \"65f04b2a-0186-41a3-87d8-44be882ad432\") "
Sep 18 16:36:06.810555 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:06.810523    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f04b2a-0186-41a3-87d8-44be882ad432-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "65f04b2a-0186-41a3-87d8-44be882ad432" (UID: "65f04b2a-0186-41a3-87d8-44be882ad432"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:36:06.810622 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:06.810432    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f04b2a-0186-41a3-87d8-44be882ad432-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "65f04b2a-0186-41a3-87d8-44be882ad432" (UID: "65f04b2a-0186-41a3-87d8-44be882ad432"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:36:06.810622 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:06.810589    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f04b2a-0186-41a3-87d8-44be882ad432-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "65f04b2a-0186-41a3-87d8-44be882ad432" (UID: "65f04b2a-0186-41a3-87d8-44be882ad432"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:36:06.812315 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:06.812284    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f04b2a-0186-41a3-87d8-44be882ad432-kube-api-access-zz78d" (OuterVolumeSpecName: "kube-api-access-zz78d") pod "65f04b2a-0186-41a3-87d8-44be882ad432" (UID: "65f04b2a-0186-41a3-87d8-44be882ad432"). InnerVolumeSpecName "kube-api-access-zz78d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:36:06.812420 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:06.812314    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65f04b2a-0186-41a3-87d8-44be882ad432-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "65f04b2a-0186-41a3-87d8-44be882ad432" (UID: "65f04b2a-0186-41a3-87d8-44be882ad432"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:36:06.911125 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:06.911091    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65f04b2a-0186-41a3-87d8-44be882ad432-cabundle-cert\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:36:06.911125 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:06.911119    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65f04b2a-0186-41a3-87d8-44be882ad432-kserve-provision-location\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:36:06.911125 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:06.911130    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz78d\" (UniqueName: \"kubernetes.io/projected/65f04b2a-0186-41a3-87d8-44be882ad432-kube-api-access-zz78d\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:36:06.911344 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:06.911138    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65f04b2a-0186-41a3-87d8-44be882ad432-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:36:06.911344 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:06.911148    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65f04b2a-0186-41a3-87d8-44be882ad432-proxy-tls\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:36:07.278126 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.278060    2566 generic.go:356] "Generic (PLEG): container finished" podID="65f04b2a-0186-41a3-87d8-44be882ad432" containerID="88f53521d65bce045c67e34a1e5267f091434890025b8bb88f97849562cfa631" exitCode=0
Sep 18 16:36:07.278126 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.278106    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn" event={"ID":"65f04b2a-0186-41a3-87d8-44be882ad432","Type":"ContainerDied","Data":"88f53521d65bce045c67e34a1e5267f091434890025b8bb88f97849562cfa631"}
Sep 18 16:36:07.278326 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.278137    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"
Sep 18 16:36:07.278326 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.278198    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn" event={"ID":"65f04b2a-0186-41a3-87d8-44be882ad432","Type":"ContainerDied","Data":"86bed98850938e10e0003f83a9b784b1b223a60aed2e885d1b1db240725a04ad"}
Sep 18 16:36:07.278326 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.278225    2566 scope.go:117] "RemoveContainer" containerID="4aa22cae3b78ef639ab53dd9b01ae6c8bbb7d41d52fe975b717b71ac7ecb3f4a"
Sep 18 16:36:07.286761 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.286744    2566 scope.go:117] "RemoveContainer" containerID="88f53521d65bce045c67e34a1e5267f091434890025b8bb88f97849562cfa631"
Sep 18 16:36:07.293872 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.293857    2566 scope.go:117] "RemoveContainer" containerID="0a887c94f4a9506ecee52fdcd4a34fe329688d424cbb9240e92cb504e8fc0388"
Sep 18 16:36:07.299399 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.299377    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"]
Sep 18 16:36:07.302183 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.302158    2566 scope.go:117] "RemoveContainer" containerID="4aa22cae3b78ef639ab53dd9b01ae6c8bbb7d41d52fe975b717b71ac7ecb3f4a"
Sep 18 16:36:07.302455 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:36:07.302436    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aa22cae3b78ef639ab53dd9b01ae6c8bbb7d41d52fe975b717b71ac7ecb3f4a\": container with ID starting with 4aa22cae3b78ef639ab53dd9b01ae6c8bbb7d41d52fe975b717b71ac7ecb3f4a not found: ID does not exist" containerID="4aa22cae3b78ef639ab53dd9b01ae6c8bbb7d41d52fe975b717b71ac7ecb3f4a"
Sep 18 16:36:07.302530 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.302462    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa22cae3b78ef639ab53dd9b01ae6c8bbb7d41d52fe975b717b71ac7ecb3f4a"} err="failed to get container status \"4aa22cae3b78ef639ab53dd9b01ae6c8bbb7d41d52fe975b717b71ac7ecb3f4a\": rpc error: code = NotFound desc = could not find container \"4aa22cae3b78ef639ab53dd9b01ae6c8bbb7d41d52fe975b717b71ac7ecb3f4a\": container with ID starting with 4aa22cae3b78ef639ab53dd9b01ae6c8bbb7d41d52fe975b717b71ac7ecb3f4a not found: ID does not exist"
Sep 18 16:36:07.302530 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.302488    2566 scope.go:117] "RemoveContainer" containerID="88f53521d65bce045c67e34a1e5267f091434890025b8bb88f97849562cfa631"
Sep 18 16:36:07.302729 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:36:07.302709    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88f53521d65bce045c67e34a1e5267f091434890025b8bb88f97849562cfa631\": container with ID starting with 88f53521d65bce045c67e34a1e5267f091434890025b8bb88f97849562cfa631 not found: ID does not exist" containerID="88f53521d65bce045c67e34a1e5267f091434890025b8bb88f97849562cfa631"
Sep 18 16:36:07.302792 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.302738    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f53521d65bce045c67e34a1e5267f091434890025b8bb88f97849562cfa631"} err="failed to get container status \"88f53521d65bce045c67e34a1e5267f091434890025b8bb88f97849562cfa631\": rpc error: code = NotFound desc = could not find container \"88f53521d65bce045c67e34a1e5267f091434890025b8bb88f97849562cfa631\": container with ID starting with 88f53521d65bce045c67e34a1e5267f091434890025b8bb88f97849562cfa631 not found: ID does not exist"
Sep 18 16:36:07.302792 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.302753    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-0a5a4-predictor-844d56bc6f-pbkcn"]
Sep 18 16:36:07.302792 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.302763    2566 scope.go:117] "RemoveContainer" containerID="0a887c94f4a9506ecee52fdcd4a34fe329688d424cbb9240e92cb504e8fc0388"
Sep 18 16:36:07.303017 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:36:07.303001    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a887c94f4a9506ecee52fdcd4a34fe329688d424cbb9240e92cb504e8fc0388\": container with ID starting with 0a887c94f4a9506ecee52fdcd4a34fe329688d424cbb9240e92cb504e8fc0388 not found: ID does not exist" containerID="0a887c94f4a9506ecee52fdcd4a34fe329688d424cbb9240e92cb504e8fc0388"
Sep 18 16:36:07.303060 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.303021    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a887c94f4a9506ecee52fdcd4a34fe329688d424cbb9240e92cb504e8fc0388"} err="failed to get container status \"0a887c94f4a9506ecee52fdcd4a34fe329688d424cbb9240e92cb504e8fc0388\": rpc error: code = NotFound desc = could not find container \"0a887c94f4a9506ecee52fdcd4a34fe329688d424cbb9240e92cb504e8fc0388\": container with ID starting with 0a887c94f4a9506ecee52fdcd4a34fe329688d424cbb9240e92cb504e8fc0388 not found: ID does not exist"
Sep 18 16:36:07.503547 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.503526    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"
Sep 18 16:36:07.515496 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.515474    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b73b611-6608-4cb4-9d17-986e245e01ec-cabundle-cert\") pod \"6b73b611-6608-4cb4-9d17-986e245e01ec\" (UID: \"6b73b611-6608-4cb4-9d17-986e245e01ec\") "
Sep 18 16:36:07.515590 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.515515    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b73b611-6608-4cb4-9d17-986e245e01ec-kube-rbac-proxy-sar-config\") pod \"6b73b611-6608-4cb4-9d17-986e245e01ec\" (UID: \"6b73b611-6608-4cb4-9d17-986e245e01ec\") "
Sep 18 16:36:07.515590 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.515555    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b73b611-6608-4cb4-9d17-986e245e01ec-kserve-provision-location\") pod \"6b73b611-6608-4cb4-9d17-986e245e01ec\" (UID: \"6b73b611-6608-4cb4-9d17-986e245e01ec\") "
Sep 18 16:36:07.515687 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.515597    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvx5j\" (UniqueName: \"kubernetes.io/projected/6b73b611-6608-4cb4-9d17-986e245e01ec-kube-api-access-wvx5j\") pod \"6b73b611-6608-4cb4-9d17-986e245e01ec\" (UID: \"6b73b611-6608-4cb4-9d17-986e245e01ec\") "
Sep 18 16:36:07.515687 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.515625    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b73b611-6608-4cb4-9d17-986e245e01ec-proxy-tls\") pod \"6b73b611-6608-4cb4-9d17-986e245e01ec\" (UID: \"6b73b611-6608-4cb4-9d17-986e245e01ec\") "
Sep 18 16:36:07.515892 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.515872    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b73b611-6608-4cb4-9d17-986e245e01ec-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6b73b611-6608-4cb4-9d17-986e245e01ec" (UID: "6b73b611-6608-4cb4-9d17-986e245e01ec"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:36:07.515939 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.515908    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b73b611-6608-4cb4-9d17-986e245e01ec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6b73b611-6608-4cb4-9d17-986e245e01ec" (UID: "6b73b611-6608-4cb4-9d17-986e245e01ec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:36:07.516004 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.515962    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b73b611-6608-4cb4-9d17-986e245e01ec-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6b73b611-6608-4cb4-9d17-986e245e01ec" (UID: "6b73b611-6608-4cb4-9d17-986e245e01ec"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:36:07.517598 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.517575    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b73b611-6608-4cb4-9d17-986e245e01ec-kube-api-access-wvx5j" (OuterVolumeSpecName: "kube-api-access-wvx5j") pod "6b73b611-6608-4cb4-9d17-986e245e01ec" (UID: "6b73b611-6608-4cb4-9d17-986e245e01ec"). InnerVolumeSpecName "kube-api-access-wvx5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:36:07.517699 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.517677    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b73b611-6608-4cb4-9d17-986e245e01ec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6b73b611-6608-4cb4-9d17-986e245e01ec" (UID: "6b73b611-6608-4cb4-9d17-986e245e01ec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:36:07.616578 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.616526    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b73b611-6608-4cb4-9d17-986e245e01ec-cabundle-cert\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:36:07.616578 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.616573    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b73b611-6608-4cb4-9d17-986e245e01ec-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:36:07.616578 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.616583    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b73b611-6608-4cb4-9d17-986e245e01ec-kserve-provision-location\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:36:07.616578 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.616591    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvx5j\" (UniqueName: \"kubernetes.io/projected/6b73b611-6608-4cb4-9d17-986e245e01ec-kube-api-access-wvx5j\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:36:07.616578 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.616600    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b73b611-6608-4cb4-9d17-986e245e01ec-proxy-tls\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:36:07.686815 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:07.686784    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65f04b2a-0186-41a3-87d8-44be882ad432" path="/var/lib/kubelet/pods/65f04b2a-0186-41a3-87d8-44be882ad432/volumes"
Sep 18 16:36:08.283970 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:08.283940    2566 generic.go:356] "Generic (PLEG): container finished" podID="6b73b611-6608-4cb4-9d17-986e245e01ec" containerID="65746e6b65d6979b0c02ccc00e09c241e5dcc5e8cd848b0ef1a4df0488795d59" exitCode=0
Sep 18 16:36:08.284394 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:08.283994    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"
Sep 18 16:36:08.284394 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:08.284018    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl" event={"ID":"6b73b611-6608-4cb4-9d17-986e245e01ec","Type":"ContainerDied","Data":"65746e6b65d6979b0c02ccc00e09c241e5dcc5e8cd848b0ef1a4df0488795d59"}
Sep 18 16:36:08.284394 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:08.284107    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl" event={"ID":"6b73b611-6608-4cb4-9d17-986e245e01ec","Type":"ContainerDied","Data":"2703a062d275742d54eb2166a7414c883417cc3866166737834d87c55400a5d2"}
Sep 18 16:36:08.284394 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:08.284145    2566 scope.go:117] "RemoveContainer" containerID="0798a545d7b8e0d7829b912af75b18d4adbba8a7c0ac3269d5aa0d45727fc6f1"
Sep 18 16:36:08.293918 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:08.293895    2566 scope.go:117] "RemoveContainer" containerID="65746e6b65d6979b0c02ccc00e09c241e5dcc5e8cd848b0ef1a4df0488795d59"
Sep 18 16:36:08.301363 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:08.301346    2566 scope.go:117] "RemoveContainer" containerID="8352dff08ffadd238bf469135c759aee4777ebd4319e2424922c0d9f84e54a63"
Sep 18 16:36:08.304222 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:08.304201    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"]
Sep 18 16:36:08.307465 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:08.307442    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-0a5a4-predictor-9d6d5f65d-92ffl"]
Sep 18 16:36:08.308681 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:08.308667    2566 scope.go:117] "RemoveContainer" containerID="0798a545d7b8e0d7829b912af75b18d4adbba8a7c0ac3269d5aa0d45727fc6f1"
Sep 18 16:36:08.308927 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:36:08.308909    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0798a545d7b8e0d7829b912af75b18d4adbba8a7c0ac3269d5aa0d45727fc6f1\": container with ID starting with 0798a545d7b8e0d7829b912af75b18d4adbba8a7c0ac3269d5aa0d45727fc6f1 not found: ID does not exist" containerID="0798a545d7b8e0d7829b912af75b18d4adbba8a7c0ac3269d5aa0d45727fc6f1"
Sep 18 16:36:08.308970 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:08.308932    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0798a545d7b8e0d7829b912af75b18d4adbba8a7c0ac3269d5aa0d45727fc6f1"} err="failed to get container status \"0798a545d7b8e0d7829b912af75b18d4adbba8a7c0ac3269d5aa0d45727fc6f1\": rpc error: code = NotFound desc = could not find container \"0798a545d7b8e0d7829b912af75b18d4adbba8a7c0ac3269d5aa0d45727fc6f1\": container with ID starting with 0798a545d7b8e0d7829b912af75b18d4adbba8a7c0ac3269d5aa0d45727fc6f1 not found: ID does not exist"
Sep 18 16:36:08.308970 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:08.308947    2566 scope.go:117] "RemoveContainer" containerID="65746e6b65d6979b0c02ccc00e09c241e5dcc5e8cd848b0ef1a4df0488795d59"
Sep 18 16:36:08.309193 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:36:08.309177    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65746e6b65d6979b0c02ccc00e09c241e5dcc5e8cd848b0ef1a4df0488795d59\": container with ID starting with 65746e6b65d6979b0c02ccc00e09c241e5dcc5e8cd848b0ef1a4df0488795d59 not found: ID does not exist" containerID="65746e6b65d6979b0c02ccc00e09c241e5dcc5e8cd848b0ef1a4df0488795d59"
Sep 18 16:36:08.309241 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:08.309197    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65746e6b65d6979b0c02ccc00e09c241e5dcc5e8cd848b0ef1a4df0488795d59"} err="failed to get container status \"65746e6b65d6979b0c02ccc00e09c241e5dcc5e8cd848b0ef1a4df0488795d59\": rpc error: code = NotFound desc = could not find container \"65746e6b65d6979b0c02ccc00e09c241e5dcc5e8cd848b0ef1a4df0488795d59\": container with ID starting with 65746e6b65d6979b0c02ccc00e09c241e5dcc5e8cd848b0ef1a4df0488795d59 not found: ID does not exist"
Sep 18 16:36:08.309241 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:08.309210    2566 scope.go:117] "RemoveContainer" containerID="8352dff08ffadd238bf469135c759aee4777ebd4319e2424922c0d9f84e54a63"
Sep 18 16:36:08.309408 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:36:08.309393    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8352dff08ffadd238bf469135c759aee4777ebd4319e2424922c0d9f84e54a63\": container with ID starting with 8352dff08ffadd238bf469135c759aee4777ebd4319e2424922c0d9f84e54a63 not found: ID does not exist" containerID="8352dff08ffadd238bf469135c759aee4777ebd4319e2424922c0d9f84e54a63"
Sep 18 16:36:08.309447 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:08.309412    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8352dff08ffadd238bf469135c759aee4777ebd4319e2424922c0d9f84e54a63"} err="failed to get container status \"8352dff08ffadd238bf469135c759aee4777ebd4319e2424922c0d9f84e54a63\": rpc error: code = NotFound desc = could not find container \"8352dff08ffadd238bf469135c759aee4777ebd4319e2424922c0d9f84e54a63\": container with ID starting with 8352dff08ffadd238bf469135c759aee4777ebd4319e2424922c0d9f84e54a63 not found: ID does not exist"
Sep 18 16:36:09.685812 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:09.685773    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b73b611-6608-4cb4-9d17-986e245e01ec" path="/var/lib/kubelet/pods/6b73b611-6608-4cb4-9d17-986e245e01ec/volumes"
Sep 18 16:36:15.165982 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:15.165945    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv" podUID="03ed76b6-52be-4ecd-bbb3-755f5627619f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 18 16:36:25.165456 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:25.165420    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv" podUID="03ed76b6-52be-4ecd-bbb3-755f5627619f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 18 16:36:35.165414 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:35.165370    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv" podUID="03ed76b6-52be-4ecd-bbb3-755f5627619f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 18 16:36:45.165848 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:45.165811    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv" podUID="03ed76b6-52be-4ecd-bbb3-755f5627619f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 18 16:36:55.166009 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:36:55.165976    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"
Sep 18 16:37:11.990601 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:11.990568    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"]
Sep 18 16:37:11.991138 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:11.990796    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv" podUID="03ed76b6-52be-4ecd-bbb3-755f5627619f" containerName="kserve-container" containerID="cri-o://f1faf44b3771e5e5ef27375b51a2f564a5c62212483882e373d995c9ec24096e" gracePeriod=30
Sep 18 16:37:11.991138 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:11.990842    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv" podUID="03ed76b6-52be-4ecd-bbb3-755f5627619f" containerName="kube-rbac-proxy" containerID="cri-o://5d115a680eaf143f72e770c534099ab53f934898a126e8f313c80e5f740af697" gracePeriod=30
Sep 18 16:37:12.540321 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:12.540290    2566 generic.go:356] "Generic (PLEG): container finished" podID="03ed76b6-52be-4ecd-bbb3-755f5627619f" containerID="5d115a680eaf143f72e770c534099ab53f934898a126e8f313c80e5f740af697" exitCode=2
Sep 18 16:37:12.540490 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:12.540361    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv" event={"ID":"03ed76b6-52be-4ecd-bbb3-755f5627619f","Type":"ContainerDied","Data":"5d115a680eaf143f72e770c534099ab53f934898a126e8f313c80e5f740af697"}
Sep 18 16:37:15.159861 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:15.159815    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv" podUID="03ed76b6-52be-4ecd-bbb3-755f5627619f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 18 16:37:15.165229 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:15.165201    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv" podUID="03ed76b6-52be-4ecd-bbb3-755f5627619f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 18 16:37:16.733648 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:16.733623    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"
Sep 18 16:37:16.766147 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:16.766119    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03ed76b6-52be-4ecd-bbb3-755f5627619f-cabundle-cert\") pod \"03ed76b6-52be-4ecd-bbb3-755f5627619f\" (UID: \"03ed76b6-52be-4ecd-bbb3-755f5627619f\") "
Sep 18 16:37:16.766304 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:16.766157    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03ed76b6-52be-4ecd-bbb3-755f5627619f-kube-rbac-proxy-sar-config\") pod \"03ed76b6-52be-4ecd-bbb3-755f5627619f\" (UID: \"03ed76b6-52be-4ecd-bbb3-755f5627619f\") "
Sep 18 16:37:16.766304 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:16.766201    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7zw7\" (UniqueName: \"kubernetes.io/projected/03ed76b6-52be-4ecd-bbb3-755f5627619f-kube-api-access-x7zw7\") pod \"03ed76b6-52be-4ecd-bbb3-755f5627619f\" (UID: \"03ed76b6-52be-4ecd-bbb3-755f5627619f\") "
Sep 18 16:37:16.766304 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:16.766273    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03ed76b6-52be-4ecd-bbb3-755f5627619f-kserve-provision-location\") pod \"03ed76b6-52be-4ecd-bbb3-755f5627619f\" (UID: \"03ed76b6-52be-4ecd-bbb3-755f5627619f\") "
Sep 18 16:37:16.766482 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:16.766307    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03ed76b6-52be-4ecd-bbb3-755f5627619f-proxy-tls\") pod \"03ed76b6-52be-4ecd-bbb3-755f5627619f\" (UID: \"03ed76b6-52be-4ecd-bbb3-755f5627619f\") "
Sep 18 16:37:16.766669 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:16.766639    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ed76b6-52be-4ecd-bbb3-755f5627619f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "03ed76b6-52be-4ecd-bbb3-755f5627619f" (UID: "03ed76b6-52be-4ecd-bbb3-755f5627619f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:37:16.766669 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:16.766650    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ed76b6-52be-4ecd-bbb3-755f5627619f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "03ed76b6-52be-4ecd-bbb3-755f5627619f" (UID: "03ed76b6-52be-4ecd-bbb3-755f5627619f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:37:16.766669 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:16.766641    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ed76b6-52be-4ecd-bbb3-755f5627619f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "03ed76b6-52be-4ecd-bbb3-755f5627619f" (UID: "03ed76b6-52be-4ecd-bbb3-755f5627619f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:37:16.768479 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:16.768451    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ed76b6-52be-4ecd-bbb3-755f5627619f-kube-api-access-x7zw7" (OuterVolumeSpecName: "kube-api-access-x7zw7") pod "03ed76b6-52be-4ecd-bbb3-755f5627619f" (UID: "03ed76b6-52be-4ecd-bbb3-755f5627619f"). InnerVolumeSpecName "kube-api-access-x7zw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:37:16.768601 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:16.768477    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03ed76b6-52be-4ecd-bbb3-755f5627619f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "03ed76b6-52be-4ecd-bbb3-755f5627619f" (UID: "03ed76b6-52be-4ecd-bbb3-755f5627619f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:37:16.867033 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:16.867003    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03ed76b6-52be-4ecd-bbb3-755f5627619f-proxy-tls\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:37:16.867033 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:16.867032    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03ed76b6-52be-4ecd-bbb3-755f5627619f-cabundle-cert\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:37:16.867205 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:16.867041    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03ed76b6-52be-4ecd-bbb3-755f5627619f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:37:16.867205 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:16.867052    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7zw7\" (UniqueName: \"kubernetes.io/projected/03ed76b6-52be-4ecd-bbb3-755f5627619f-kube-api-access-x7zw7\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:37:16.867205 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:16.867060    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03ed76b6-52be-4ecd-bbb3-755f5627619f-kserve-provision-location\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:37:17.560495 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:17.560459    2566 generic.go:356] "Generic (PLEG): container finished" podID="03ed76b6-52be-4ecd-bbb3-755f5627619f" containerID="f1faf44b3771e5e5ef27375b51a2f564a5c62212483882e373d995c9ec24096e" exitCode=0
Sep 18 16:37:17.560672 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:17.560516    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"
Sep 18 16:37:17.560672 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:17.560551    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv" event={"ID":"03ed76b6-52be-4ecd-bbb3-755f5627619f","Type":"ContainerDied","Data":"f1faf44b3771e5e5ef27375b51a2f564a5c62212483882e373d995c9ec24096e"}
Sep 18 16:37:17.560672 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:17.560628    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv" event={"ID":"03ed76b6-52be-4ecd-bbb3-755f5627619f","Type":"ContainerDied","Data":"eff1cabfe1ad8cdad43412d34910b8b7b5b3edbcb4e15fc9725652454adf99e9"}
Sep 18 16:37:17.560672 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:17.560658    2566 scope.go:117] "RemoveContainer" containerID="5d115a680eaf143f72e770c534099ab53f934898a126e8f313c80e5f740af697"
Sep 18 16:37:17.576186 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:17.576166    2566 scope.go:117] "RemoveContainer" containerID="f1faf44b3771e5e5ef27375b51a2f564a5c62212483882e373d995c9ec24096e"
Sep 18 16:37:17.583038 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:17.583018    2566 scope.go:117] "RemoveContainer" containerID="1d0ff445b4967aea7e3ab06e63d30e67bd36b883330db7a51954fede9c052dc7"
Sep 18 16:37:17.587467 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:17.587443    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"]
Sep 18 16:37:17.590298 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:17.590281    2566 scope.go:117] "RemoveContainer" containerID="5d115a680eaf143f72e770c534099ab53f934898a126e8f313c80e5f740af697"
Sep 18 16:37:17.590736 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:37:17.590700    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d115a680eaf143f72e770c534099ab53f934898a126e8f313c80e5f740af697\": container with ID starting with 5d115a680eaf143f72e770c534099ab53f934898a126e8f313c80e5f740af697 not found: ID does not exist" containerID="5d115a680eaf143f72e770c534099ab53f934898a126e8f313c80e5f740af697"
Sep 18 16:37:17.590841 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:17.590755    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d115a680eaf143f72e770c534099ab53f934898a126e8f313c80e5f740af697"} err="failed to get container status \"5d115a680eaf143f72e770c534099ab53f934898a126e8f313c80e5f740af697\": rpc error: code = NotFound desc = could not find container \"5d115a680eaf143f72e770c534099ab53f934898a126e8f313c80e5f740af697\": container with ID starting with 5d115a680eaf143f72e770c534099ab53f934898a126e8f313c80e5f740af697 not found: ID does not exist"
Sep 18 16:37:17.590841 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:17.590780    2566 scope.go:117] "RemoveContainer" containerID="f1faf44b3771e5e5ef27375b51a2f564a5c62212483882e373d995c9ec24096e"
Sep 18 16:37:17.591167 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:37:17.591146    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1faf44b3771e5e5ef27375b51a2f564a5c62212483882e373d995c9ec24096e\": container with ID starting with f1faf44b3771e5e5ef27375b51a2f564a5c62212483882e373d995c9ec24096e not found: ID does not exist" containerID="f1faf44b3771e5e5ef27375b51a2f564a5c62212483882e373d995c9ec24096e"
Sep 18 16:37:17.591237 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:17.591174    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1faf44b3771e5e5ef27375b51a2f564a5c62212483882e373d995c9ec24096e"} err="failed to get container status \"f1faf44b3771e5e5ef27375b51a2f564a5c62212483882e373d995c9ec24096e\": rpc error: code = NotFound desc = could not find container \"f1faf44b3771e5e5ef27375b51a2f564a5c62212483882e373d995c9ec24096e\": container with ID starting with f1faf44b3771e5e5ef27375b51a2f564a5c62212483882e373d995c9ec24096e not found: ID does not exist"
Sep 18 16:37:17.591237 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:17.591190    2566 scope.go:117] "RemoveContainer" containerID="1d0ff445b4967aea7e3ab06e63d30e67bd36b883330db7a51954fede9c052dc7"
Sep 18 16:37:17.591445 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:37:17.591429    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d0ff445b4967aea7e3ab06e63d30e67bd36b883330db7a51954fede9c052dc7\": container with ID starting with 1d0ff445b4967aea7e3ab06e63d30e67bd36b883330db7a51954fede9c052dc7 not found: ID does not exist" containerID="1d0ff445b4967aea7e3ab06e63d30e67bd36b883330db7a51954fede9c052dc7"
Sep 18 16:37:17.591490 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:17.591449    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d0ff445b4967aea7e3ab06e63d30e67bd36b883330db7a51954fede9c052dc7"} err="failed to get container status \"1d0ff445b4967aea7e3ab06e63d30e67bd36b883330db7a51954fede9c052dc7\": rpc error: code = NotFound desc = could not find container \"1d0ff445b4967aea7e3ab06e63d30e67bd36b883330db7a51954fede9c052dc7\": container with ID starting with 1d0ff445b4967aea7e3ab06e63d30e67bd36b883330db7a51954fede9c052dc7 not found: ID does not exist"
Sep 18 16:37:17.592236 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:17.592219    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-q8mwv"]
Sep 18 16:37:17.686758 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:17.686723    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03ed76b6-52be-4ecd-bbb3-755f5627619f" path="/var/lib/kubelet/pods/03ed76b6-52be-4ecd-bbb3-755f5627619f/volumes"
Sep 18 16:37:22.253965 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.253931    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"]
Sep 18 16:37:22.254345 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.254241    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65f04b2a-0186-41a3-87d8-44be882ad432" containerName="kserve-container"
Sep 18 16:37:22.254345 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.254257    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="65f04b2a-0186-41a3-87d8-44be882ad432" containerName="kserve-container"
Sep 18 16:37:22.254345 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.254271    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65f04b2a-0186-41a3-87d8-44be882ad432" containerName="kube-rbac-proxy"
Sep 18 16:37:22.254345 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.254276    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="65f04b2a-0186-41a3-87d8-44be882ad432" containerName="kube-rbac-proxy"
Sep 18 16:37:22.254345 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.254288    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03ed76b6-52be-4ecd-bbb3-755f5627619f" containerName="kube-rbac-proxy"
Sep 18 16:37:22.254345 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.254294    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="03ed76b6-52be-4ecd-bbb3-755f5627619f" containerName="kube-rbac-proxy"
Sep 18 16:37:22.254345 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.254304    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65f04b2a-0186-41a3-87d8-44be882ad432" containerName="storage-initializer"
Sep 18 16:37:22.254345 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.254309    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="65f04b2a-0186-41a3-87d8-44be882ad432" containerName="storage-initializer"
Sep 18 16:37:22.254345 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.254317    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b73b611-6608-4cb4-9d17-986e245e01ec" containerName="kserve-container"
Sep 18 16:37:22.254345 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.254328    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b73b611-6608-4cb4-9d17-986e245e01ec" containerName="kserve-container"
Sep 18 16:37:22.254345 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.254334    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b73b611-6608-4cb4-9d17-986e245e01ec" containerName="kube-rbac-proxy"
Sep 18 16:37:22.254345 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.254340    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b73b611-6608-4cb4-9d17-986e245e01ec" containerName="kube-rbac-proxy"
Sep 18 16:37:22.254345 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.254346    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03ed76b6-52be-4ecd-bbb3-755f5627619f" containerName="kserve-container"
Sep 18 16:37:22.254345 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.254352    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="03ed76b6-52be-4ecd-bbb3-755f5627619f" containerName="kserve-container"
Sep 18 16:37:22.254770 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.254360    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b73b611-6608-4cb4-9d17-986e245e01ec" containerName="storage-initializer"
Sep 18 16:37:22.254770 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.254365    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b73b611-6608-4cb4-9d17-986e245e01ec" containerName="storage-initializer"
Sep 18 16:37:22.254770 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.254370    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03ed76b6-52be-4ecd-bbb3-755f5627619f" containerName="storage-initializer"
Sep 18 16:37:22.254770 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.254375    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="03ed76b6-52be-4ecd-bbb3-755f5627619f" containerName="storage-initializer"
Sep 18 16:37:22.254770 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.254423    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="03ed76b6-52be-4ecd-bbb3-755f5627619f" containerName="kube-rbac-proxy"
Sep 18 16:37:22.254770 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.254430    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b73b611-6608-4cb4-9d17-986e245e01ec" containerName="kserve-container"
Sep 18 16:37:22.254770 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.254439    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="65f04b2a-0186-41a3-87d8-44be882ad432" containerName="kube-rbac-proxy"
Sep 18 16:37:22.254770 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.254444    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="65f04b2a-0186-41a3-87d8-44be882ad432" containerName="kserve-container"
Sep 18 16:37:22.254770 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.254450    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="03ed76b6-52be-4ecd-bbb3-755f5627619f" containerName="kserve-container"
Sep 18 16:37:22.254770 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.254455    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b73b611-6608-4cb4-9d17-986e245e01ec" containerName="kube-rbac-proxy"
Sep 18 16:37:22.257183 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.257168    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"
Sep 18 16:37:22.260609 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.260586    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 16:37:22.260778 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.260756    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 16:37:22.260836 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.260790    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-4e8a5a-predictor-serving-cert\""
Sep 18 16:37:22.260836 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.260812    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-4e8a5a-kube-rbac-proxy-sar-config\""
Sep 18 16:37:22.261003 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.260982    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 16:37:22.261300 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.261278    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 16:37:22.261722 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.261706    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6lh7v\""
Sep 18 16:37:22.271132 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.271112    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"]
Sep 18 16:37:22.308883 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.308858    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af63257f-9ae7-477a-99c0-0e09227687f4-kserve-provision-location\") pod \"isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt\" (UID: \"af63257f-9ae7-477a-99c0-0e09227687f4\") " pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"
Sep 18 16:37:22.308980 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.308891    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af63257f-9ae7-477a-99c0-0e09227687f4-kube-rbac-proxy-sar-config\") pod \"isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt\" (UID: \"af63257f-9ae7-477a-99c0-0e09227687f4\") " pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"
Sep 18 16:37:22.308980 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.308910    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndlts\" (UniqueName: \"kubernetes.io/projected/af63257f-9ae7-477a-99c0-0e09227687f4-kube-api-access-ndlts\") pod \"isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt\" (UID: \"af63257f-9ae7-477a-99c0-0e09227687f4\") " pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"
Sep 18 16:37:22.308980 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.308930    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af63257f-9ae7-477a-99c0-0e09227687f4-proxy-tls\") pod \"isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt\" (UID: \"af63257f-9ae7-477a-99c0-0e09227687f4\") " pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"
Sep 18 16:37:22.309113 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.308999    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af63257f-9ae7-477a-99c0-0e09227687f4-cabundle-cert\") pod \"isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt\" (UID: \"af63257f-9ae7-477a-99c0-0e09227687f4\") " pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"
Sep 18 16:37:22.410038 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.410009    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af63257f-9ae7-477a-99c0-0e09227687f4-kube-rbac-proxy-sar-config\") pod \"isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt\" (UID: \"af63257f-9ae7-477a-99c0-0e09227687f4\") " pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"
Sep 18 16:37:22.410038 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.410041    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndlts\" (UniqueName: \"kubernetes.io/projected/af63257f-9ae7-477a-99c0-0e09227687f4-kube-api-access-ndlts\") pod \"isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt\" (UID: \"af63257f-9ae7-477a-99c0-0e09227687f4\") " pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"
Sep 18 16:37:22.410227 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.410057    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af63257f-9ae7-477a-99c0-0e09227687f4-proxy-tls\") pod \"isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt\" (UID: \"af63257f-9ae7-477a-99c0-0e09227687f4\") " pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"
Sep 18 16:37:22.410227 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.410146    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af63257f-9ae7-477a-99c0-0e09227687f4-cabundle-cert\") pod \"isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt\" (UID: \"af63257f-9ae7-477a-99c0-0e09227687f4\") " pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"
Sep 18 16:37:22.410227 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.410186    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af63257f-9ae7-477a-99c0-0e09227687f4-kserve-provision-location\") pod \"isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt\" (UID: \"af63257f-9ae7-477a-99c0-0e09227687f4\") " pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"
Sep 18 16:37:22.410380 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:37:22.410278    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-serving-cert: secret "isvc-primary-4e8a5a-predictor-serving-cert" not found
Sep 18 16:37:22.410380 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:37:22.410354    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af63257f-9ae7-477a-99c0-0e09227687f4-proxy-tls podName:af63257f-9ae7-477a-99c0-0e09227687f4 nodeName:}" failed. No retries permitted until 2026-09-18 16:37:22.91033293 +0000 UTC m=+1281.365785990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/af63257f-9ae7-477a-99c0-0e09227687f4-proxy-tls") pod "isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt" (UID: "af63257f-9ae7-477a-99c0-0e09227687f4") : secret "isvc-primary-4e8a5a-predictor-serving-cert" not found
Sep 18 16:37:22.410556 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.410539    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af63257f-9ae7-477a-99c0-0e09227687f4-kserve-provision-location\") pod \"isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt\" (UID: \"af63257f-9ae7-477a-99c0-0e09227687f4\") " pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"
Sep 18 16:37:22.410837 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.410817    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af63257f-9ae7-477a-99c0-0e09227687f4-cabundle-cert\") pod \"isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt\" (UID: \"af63257f-9ae7-477a-99c0-0e09227687f4\") " pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"
Sep 18 16:37:22.410874 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.410817    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af63257f-9ae7-477a-99c0-0e09227687f4-kube-rbac-proxy-sar-config\") pod \"isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt\" (UID: \"af63257f-9ae7-477a-99c0-0e09227687f4\") " pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"
Sep 18 16:37:22.418512 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.418494    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndlts\" (UniqueName: \"kubernetes.io/projected/af63257f-9ae7-477a-99c0-0e09227687f4-kube-api-access-ndlts\") pod \"isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt\" (UID: \"af63257f-9ae7-477a-99c0-0e09227687f4\") " pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"
Sep 18 16:37:22.914458 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.914422    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af63257f-9ae7-477a-99c0-0e09227687f4-proxy-tls\") pod \"isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt\" (UID: \"af63257f-9ae7-477a-99c0-0e09227687f4\") " pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"
Sep 18 16:37:22.916853 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:22.916830    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af63257f-9ae7-477a-99c0-0e09227687f4-proxy-tls\") pod \"isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt\" (UID: \"af63257f-9ae7-477a-99c0-0e09227687f4\") " pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"
Sep 18 16:37:23.167345 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:23.167267    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"
Sep 18 16:37:23.438147 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:23.438050    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"]
Sep 18 16:37:23.441426 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:37:23.441391    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf63257f_9ae7_477a_99c0_0e09227687f4.slice/crio-9fce8439af8727b9d3f9f5fadae8bb34b32caef88272a0cd907192ab8673dabe WatchSource:0}: Error finding container 9fce8439af8727b9d3f9f5fadae8bb34b32caef88272a0cd907192ab8673dabe: Status 404 returned error can't find the container with id 9fce8439af8727b9d3f9f5fadae8bb34b32caef88272a0cd907192ab8673dabe
Sep 18 16:37:23.586313 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:23.586284    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt" event={"ID":"af63257f-9ae7-477a-99c0-0e09227687f4","Type":"ContainerStarted","Data":"f036d84e34d962743b5ae74392cb6c2bd9279277fe86e41d5e32097e98669ebf"}
Sep 18 16:37:23.586507 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:23.586337    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt" event={"ID":"af63257f-9ae7-477a-99c0-0e09227687f4","Type":"ContainerStarted","Data":"9fce8439af8727b9d3f9f5fadae8bb34b32caef88272a0cd907192ab8673dabe"}
Sep 18 16:37:27.602992 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:27.602959    2566 generic.go:356] "Generic (PLEG): container finished" podID="af63257f-9ae7-477a-99c0-0e09227687f4" containerID="f036d84e34d962743b5ae74392cb6c2bd9279277fe86e41d5e32097e98669ebf" exitCode=0
Sep 18 16:37:27.603359 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:27.603001    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt" event={"ID":"af63257f-9ae7-477a-99c0-0e09227687f4","Type":"ContainerDied","Data":"f036d84e34d962743b5ae74392cb6c2bd9279277fe86e41d5e32097e98669ebf"}
Sep 18 16:37:28.608601 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:28.608568    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt" event={"ID":"af63257f-9ae7-477a-99c0-0e09227687f4","Type":"ContainerStarted","Data":"f3ee04e06d055208145561bb201a71f01c6cbf40f7d2d13622ad3f15f02b0bcb"}
Sep 18 16:37:28.608973 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:28.608629    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt" event={"ID":"af63257f-9ae7-477a-99c0-0e09227687f4","Type":"ContainerStarted","Data":"4c0781941c6c03000890adbaf8095f15e94d942d5012e576e872866c50357548"}
Sep 18 16:37:28.608973 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:28.608870    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"
Sep 18 16:37:28.628583 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:28.628536    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt" podStartSLOduration=2.872735821 podStartE2EDuration="6.628521128s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.755785307s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:37:28.626793639 +0000 UTC m=+1287.082246716" watchObservedRunningTime="2026-09-18 16:37:28.628521128 +0000 UTC m=+1287.083974276"
Sep 18 16:37:29.612240 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:29.612207    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"
Sep 18 16:37:29.613526 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:29.613502    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt" podUID="af63257f-9ae7-477a-99c0-0e09227687f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 18 16:37:30.616027 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:30.615984    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt" podUID="af63257f-9ae7-477a-99c0-0e09227687f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 18 16:37:35.620981 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:35.620953    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"
Sep 18 16:37:35.621456 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:35.621430    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt" podUID="af63257f-9ae7-477a-99c0-0e09227687f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 18 16:37:45.622183 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:45.622138    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt" podUID="af63257f-9ae7-477a-99c0-0e09227687f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 18 16:37:55.622322 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:37:55.622278    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt" podUID="af63257f-9ae7-477a-99c0-0e09227687f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 18 16:38:02.819426 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:02.819382    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m"]
Sep 18 16:38:02.822862 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:02.822838    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m"
Sep 18 16:38:02.825744 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:02.825721    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-031b1-kube-rbac-proxy-sar-config\""
Sep 18 16:38:02.825861 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:02.825719    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-031b1-predictor-serving-cert\""
Sep 18 16:38:02.833195 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:02.833173    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m"]
Sep 18 16:38:02.950636 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:02.950600    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpcfh\" (UniqueName: \"kubernetes.io/projected/ab1b734f-69ba-40f0-b82e-c6a9a72c4711-kube-api-access-zpcfh\") pod \"message-dumper-raw-031b1-predictor-57d74fc557-tcw9m\" (UID: \"ab1b734f-69ba-40f0-b82e-c6a9a72c4711\") " pod="kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m"
Sep 18 16:38:02.950636 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:02.950641    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab1b734f-69ba-40f0-b82e-c6a9a72c4711-proxy-tls\") pod \"message-dumper-raw-031b1-predictor-57d74fc557-tcw9m\" (UID: \"ab1b734f-69ba-40f0-b82e-c6a9a72c4711\") " pod="kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m"
Sep 18 16:38:02.950859 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:02.950701    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab1b734f-69ba-40f0-b82e-c6a9a72c4711-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-031b1-predictor-57d74fc557-tcw9m\" (UID: \"ab1b734f-69ba-40f0-b82e-c6a9a72c4711\") " pod="kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m"
Sep 18 16:38:03.051798 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:03.051766    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpcfh\" (UniqueName: \"kubernetes.io/projected/ab1b734f-69ba-40f0-b82e-c6a9a72c4711-kube-api-access-zpcfh\") pod \"message-dumper-raw-031b1-predictor-57d74fc557-tcw9m\" (UID: \"ab1b734f-69ba-40f0-b82e-c6a9a72c4711\") " pod="kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m"
Sep 18 16:38:03.051976 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:03.051812    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab1b734f-69ba-40f0-b82e-c6a9a72c4711-proxy-tls\") pod \"message-dumper-raw-031b1-predictor-57d74fc557-tcw9m\" (UID: \"ab1b734f-69ba-40f0-b82e-c6a9a72c4711\") " pod="kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m"
Sep 18 16:38:03.051976 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:03.051843    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab1b734f-69ba-40f0-b82e-c6a9a72c4711-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-031b1-predictor-57d74fc557-tcw9m\" (UID: \"ab1b734f-69ba-40f0-b82e-c6a9a72c4711\") " pod="kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m"
Sep 18 16:38:03.052465 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:03.052439    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab1b734f-69ba-40f0-b82e-c6a9a72c4711-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-031b1-predictor-57d74fc557-tcw9m\" (UID: \"ab1b734f-69ba-40f0-b82e-c6a9a72c4711\") " pod="kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m"
Sep 18 16:38:03.054301 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:03.054283    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab1b734f-69ba-40f0-b82e-c6a9a72c4711-proxy-tls\") pod \"message-dumper-raw-031b1-predictor-57d74fc557-tcw9m\" (UID: \"ab1b734f-69ba-40f0-b82e-c6a9a72c4711\") " pod="kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m"
Sep 18 16:38:03.059801 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:03.059778    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpcfh\" (UniqueName: \"kubernetes.io/projected/ab1b734f-69ba-40f0-b82e-c6a9a72c4711-kube-api-access-zpcfh\") pod \"message-dumper-raw-031b1-predictor-57d74fc557-tcw9m\" (UID: \"ab1b734f-69ba-40f0-b82e-c6a9a72c4711\") " pod="kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m"
Sep 18 16:38:03.135525 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:03.135437    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m"
Sep 18 16:38:03.407060 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:03.406981    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m"]
Sep 18 16:38:03.409828 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:38:03.409800    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1b734f_69ba_40f0_b82e_c6a9a72c4711.slice/crio-be641f56b4cc3787ac67f8110bf7cee1ef3b5a8900ce2951f05321383df0eda4 WatchSource:0}: Error finding container be641f56b4cc3787ac67f8110bf7cee1ef3b5a8900ce2951f05321383df0eda4: Status 404 returned error can't find the container with id be641f56b4cc3787ac67f8110bf7cee1ef3b5a8900ce2951f05321383df0eda4
Sep 18 16:38:03.746116 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:03.746067    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m" event={"ID":"ab1b734f-69ba-40f0-b82e-c6a9a72c4711","Type":"ContainerStarted","Data":"be641f56b4cc3787ac67f8110bf7cee1ef3b5a8900ce2951f05321383df0eda4"}
Sep 18 16:38:05.622245 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:05.622202    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt" podUID="af63257f-9ae7-477a-99c0-0e09227687f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 18 16:38:05.755979 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:05.755942    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m" event={"ID":"ab1b734f-69ba-40f0-b82e-c6a9a72c4711","Type":"ContainerStarted","Data":"0daf6d936eab75e097c5fd60d6fc8210b5d227cf7b1523cb23f108ffcf710432"}
Sep 18 16:38:05.756174 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:05.756014    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m" event={"ID":"ab1b734f-69ba-40f0-b82e-c6a9a72c4711","Type":"ContainerStarted","Data":"6ee202d69509909000f07678438c1bbf126adfcdb888386b071f60fc064b4e4a"}
Sep 18 16:38:05.756232 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:05.756202    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m"
Sep 18 16:38:05.756271 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:05.756249    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m"
Sep 18 16:38:05.757946 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:05.757926    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m"
Sep 18 16:38:05.776906 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:05.776853    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m" podStartSLOduration=2.353060993 podStartE2EDuration="3.776839956s" totalImagesPullingTime="1.423778963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:38:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:38:05.775488479 +0000 UTC m=+1324.230941573" watchObservedRunningTime="2026-09-18 16:38:05.776839956 +0000 UTC m=+1324.232293033"
Sep 18 16:38:12.769242 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:12.769212    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m"
Sep 18 16:38:15.622247 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:15.622155    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt" podUID="af63257f-9ae7-477a-99c0-0e09227687f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 18 16:38:25.622338 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:25.622296    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt" podUID="af63257f-9ae7-477a-99c0-0e09227687f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 18 16:38:35.622362 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:35.622319    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt" podUID="af63257f-9ae7-477a-99c0-0e09227687f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 18 16:38:45.622382 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:38:45.622350    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"
Sep 18 16:39:25.494120 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:25.494053    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"]
Sep 18 16:39:25.494707 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:25.494305    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt" podUID="af63257f-9ae7-477a-99c0-0e09227687f4" containerName="kserve-container" containerID="cri-o://4c0781941c6c03000890adbaf8095f15e94d942d5012e576e872866c50357548" gracePeriod=30
Sep 18 16:39:25.494707 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:25.494345    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt" podUID="af63257f-9ae7-477a-99c0-0e09227687f4" containerName="kube-rbac-proxy" containerID="cri-o://f3ee04e06d055208145561bb201a71f01c6cbf40f7d2d13622ad3f15f02b0bcb" gracePeriod=30
Sep 18 16:39:25.617319 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:25.617280    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt" podUID="af63257f-9ae7-477a-99c0-0e09227687f4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 18 16:39:25.621651 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:25.621623    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt" podUID="af63257f-9ae7-477a-99c0-0e09227687f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 18 16:39:26.074039 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:26.074007    2566 generic.go:356] "Generic (PLEG): container finished" podID="af63257f-9ae7-477a-99c0-0e09227687f4" containerID="f3ee04e06d055208145561bb201a71f01c6cbf40f7d2d13622ad3f15f02b0bcb" exitCode=2
Sep 18 16:39:26.074230 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:26.074092    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt" event={"ID":"af63257f-9ae7-477a-99c0-0e09227687f4","Type":"ContainerDied","Data":"f3ee04e06d055208145561bb201a71f01c6cbf40f7d2d13622ad3f15f02b0bcb"}
Sep 18 16:39:30.239102 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:30.239064    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"
Sep 18 16:39:30.390184 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:30.390152    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af63257f-9ae7-477a-99c0-0e09227687f4-kserve-provision-location\") pod \"af63257f-9ae7-477a-99c0-0e09227687f4\" (UID: \"af63257f-9ae7-477a-99c0-0e09227687f4\") "
Sep 18 16:39:30.390342 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:30.390209    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af63257f-9ae7-477a-99c0-0e09227687f4-kube-rbac-proxy-sar-config\") pod \"af63257f-9ae7-477a-99c0-0e09227687f4\" (UID: \"af63257f-9ae7-477a-99c0-0e09227687f4\") "
Sep 18 16:39:30.390342 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:30.390227    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af63257f-9ae7-477a-99c0-0e09227687f4-cabundle-cert\") pod \"af63257f-9ae7-477a-99c0-0e09227687f4\" (UID: \"af63257f-9ae7-477a-99c0-0e09227687f4\") "
Sep 18 16:39:30.390342 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:30.390248    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndlts\" (UniqueName: \"kubernetes.io/projected/af63257f-9ae7-477a-99c0-0e09227687f4-kube-api-access-ndlts\") pod \"af63257f-9ae7-477a-99c0-0e09227687f4\" (UID: \"af63257f-9ae7-477a-99c0-0e09227687f4\") "
Sep 18 16:39:30.390342 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:30.390268    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af63257f-9ae7-477a-99c0-0e09227687f4-proxy-tls\") pod \"af63257f-9ae7-477a-99c0-0e09227687f4\" (UID: \"af63257f-9ae7-477a-99c0-0e09227687f4\") "
Sep 18 16:39:30.390525 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:30.390500    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af63257f-9ae7-477a-99c0-0e09227687f4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "af63257f-9ae7-477a-99c0-0e09227687f4" (UID: "af63257f-9ae7-477a-99c0-0e09227687f4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:39:30.390679 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:30.390640    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af63257f-9ae7-477a-99c0-0e09227687f4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "af63257f-9ae7-477a-99c0-0e09227687f4" (UID: "af63257f-9ae7-477a-99c0-0e09227687f4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:39:30.390791 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:30.390699    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af63257f-9ae7-477a-99c0-0e09227687f4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "af63257f-9ae7-477a-99c0-0e09227687f4" (UID: "af63257f-9ae7-477a-99c0-0e09227687f4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:39:30.392429 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:30.392404    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af63257f-9ae7-477a-99c0-0e09227687f4-kube-api-access-ndlts" (OuterVolumeSpecName: "kube-api-access-ndlts") pod "af63257f-9ae7-477a-99c0-0e09227687f4" (UID: "af63257f-9ae7-477a-99c0-0e09227687f4"). InnerVolumeSpecName "kube-api-access-ndlts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:39:30.392521 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:30.392423    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af63257f-9ae7-477a-99c0-0e09227687f4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "af63257f-9ae7-477a-99c0-0e09227687f4" (UID: "af63257f-9ae7-477a-99c0-0e09227687f4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:39:30.490926 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:30.490894    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af63257f-9ae7-477a-99c0-0e09227687f4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:39:30.490926 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:30.490921    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af63257f-9ae7-477a-99c0-0e09227687f4-cabundle-cert\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:39:30.490926 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:30.490932    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndlts\" (UniqueName: \"kubernetes.io/projected/af63257f-9ae7-477a-99c0-0e09227687f4-kube-api-access-ndlts\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:39:30.491153 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:30.490941    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af63257f-9ae7-477a-99c0-0e09227687f4-proxy-tls\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:39:30.491153 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:30.490949    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af63257f-9ae7-477a-99c0-0e09227687f4-kserve-provision-location\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:39:31.096648 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:31.096612    2566 generic.go:356] "Generic (PLEG): container finished" podID="af63257f-9ae7-477a-99c0-0e09227687f4" containerID="4c0781941c6c03000890adbaf8095f15e94d942d5012e576e872866c50357548" exitCode=0
Sep 18 16:39:31.096919 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:31.096672    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"
Sep 18 16:39:31.096919 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:31.096698    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt" event={"ID":"af63257f-9ae7-477a-99c0-0e09227687f4","Type":"ContainerDied","Data":"4c0781941c6c03000890adbaf8095f15e94d942d5012e576e872866c50357548"}
Sep 18 16:39:31.096919 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:31.096761    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt" event={"ID":"af63257f-9ae7-477a-99c0-0e09227687f4","Type":"ContainerDied","Data":"9fce8439af8727b9d3f9f5fadae8bb34b32caef88272a0cd907192ab8673dabe"}
Sep 18 16:39:31.096919 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:31.096789    2566 scope.go:117] "RemoveContainer" containerID="f3ee04e06d055208145561bb201a71f01c6cbf40f7d2d13622ad3f15f02b0bcb"
Sep 18 16:39:31.105255 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:31.105238    2566 scope.go:117] "RemoveContainer" containerID="4c0781941c6c03000890adbaf8095f15e94d942d5012e576e872866c50357548"
Sep 18 16:39:31.112404 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:31.112208    2566 scope.go:117] "RemoveContainer" containerID="f036d84e34d962743b5ae74392cb6c2bd9279277fe86e41d5e32097e98669ebf"
Sep 18 16:39:31.119089 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:31.119063    2566 scope.go:117] "RemoveContainer" containerID="f3ee04e06d055208145561bb201a71f01c6cbf40f7d2d13622ad3f15f02b0bcb"
Sep 18 16:39:31.119391 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:39:31.119372    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3ee04e06d055208145561bb201a71f01c6cbf40f7d2d13622ad3f15f02b0bcb\": container with ID starting with f3ee04e06d055208145561bb201a71f01c6cbf40f7d2d13622ad3f15f02b0bcb not found: ID does not exist" containerID="f3ee04e06d055208145561bb201a71f01c6cbf40f7d2d13622ad3f15f02b0bcb"
Sep 18 16:39:31.119443 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:31.119399    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3ee04e06d055208145561bb201a71f01c6cbf40f7d2d13622ad3f15f02b0bcb"} err="failed to get container status \"f3ee04e06d055208145561bb201a71f01c6cbf40f7d2d13622ad3f15f02b0bcb\": rpc error: code = NotFound desc = could not find container \"f3ee04e06d055208145561bb201a71f01c6cbf40f7d2d13622ad3f15f02b0bcb\": container with ID starting with f3ee04e06d055208145561bb201a71f01c6cbf40f7d2d13622ad3f15f02b0bcb not found: ID does not exist"
Sep 18 16:39:31.119443 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:31.119415    2566 scope.go:117] "RemoveContainer" containerID="4c0781941c6c03000890adbaf8095f15e94d942d5012e576e872866c50357548"
Sep 18 16:39:31.119668 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:39:31.119649    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c0781941c6c03000890adbaf8095f15e94d942d5012e576e872866c50357548\": container with ID starting with 4c0781941c6c03000890adbaf8095f15e94d942d5012e576e872866c50357548 not found: ID does not exist" containerID="4c0781941c6c03000890adbaf8095f15e94d942d5012e576e872866c50357548"
Sep 18 16:39:31.119709 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:31.119674    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c0781941c6c03000890adbaf8095f15e94d942d5012e576e872866c50357548"} err="failed to get container status \"4c0781941c6c03000890adbaf8095f15e94d942d5012e576e872866c50357548\": rpc error: code = NotFound desc = could not find container \"4c0781941c6c03000890adbaf8095f15e94d942d5012e576e872866c50357548\": container with ID starting with 4c0781941c6c03000890adbaf8095f15e94d942d5012e576e872866c50357548 not found: ID does not exist"
Sep 18 16:39:31.119709 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:31.119689    2566 scope.go:117] "RemoveContainer" containerID="f036d84e34d962743b5ae74392cb6c2bd9279277fe86e41d5e32097e98669ebf"
Sep 18 16:39:31.119877 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:39:31.119853    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f036d84e34d962743b5ae74392cb6c2bd9279277fe86e41d5e32097e98669ebf\": container with ID starting with f036d84e34d962743b5ae74392cb6c2bd9279277fe86e41d5e32097e98669ebf not found: ID does not exist" containerID="f036d84e34d962743b5ae74392cb6c2bd9279277fe86e41d5e32097e98669ebf"
Sep 18 16:39:31.119971 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:31.119880    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f036d84e34d962743b5ae74392cb6c2bd9279277fe86e41d5e32097e98669ebf"} err="failed to get container status \"f036d84e34d962743b5ae74392cb6c2bd9279277fe86e41d5e32097e98669ebf\": rpc error: code = NotFound desc = could not find container \"f036d84e34d962743b5ae74392cb6c2bd9279277fe86e41d5e32097e98669ebf\": container with ID starting with f036d84e34d962743b5ae74392cb6c2bd9279277fe86e41d5e32097e98669ebf not found: ID does not exist"
Sep 18 16:39:31.123416 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:31.123396    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"]
Sep 18 16:39:31.129854 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:31.129835    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-4e8a5a-predictor-567f87f6cf-vpnbt"]
Sep 18 16:39:31.686358 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:31.686321    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af63257f-9ae7-477a-99c0-0e09227687f4" path="/var/lib/kubelet/pods/af63257f-9ae7-477a-99c0-0e09227687f4/volumes"
Sep 18 16:39:58.448311 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.448267    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-031b1-predictor-57d74fc557-tcw9m_ab1b734f-69ba-40f0-b82e-c6a9a72c4711/kserve-container/0.log"
Sep 18 16:39:58.657085 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.657042    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m"]
Sep 18 16:39:58.657340 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.657295    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m" podUID="ab1b734f-69ba-40f0-b82e-c6a9a72c4711" containerName="kserve-container" containerID="cri-o://6ee202d69509909000f07678438c1bbf126adfcdb888386b071f60fc064b4e4a" gracePeriod=30
Sep 18 16:39:58.657411 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.657341    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m" podUID="ab1b734f-69ba-40f0-b82e-c6a9a72c4711" containerName="kube-rbac-proxy" containerID="cri-o://0daf6d936eab75e097c5fd60d6fc8210b5d227cf7b1523cb23f108ffcf710432" gracePeriod=30
Sep 18 16:39:58.762094 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.761991    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"]
Sep 18 16:39:58.762472 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.762457    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af63257f-9ae7-477a-99c0-0e09227687f4" containerName="kube-rbac-proxy"
Sep 18 16:39:58.762535 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.762476    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="af63257f-9ae7-477a-99c0-0e09227687f4" containerName="kube-rbac-proxy"
Sep 18 16:39:58.762535 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.762512    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af63257f-9ae7-477a-99c0-0e09227687f4" containerName="storage-initializer"
Sep 18 16:39:58.762535 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.762522    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="af63257f-9ae7-477a-99c0-0e09227687f4" containerName="storage-initializer"
Sep 18 16:39:58.762644 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.762538    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af63257f-9ae7-477a-99c0-0e09227687f4" containerName="kserve-container"
Sep 18 16:39:58.762644 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.762546    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="af63257f-9ae7-477a-99c0-0e09227687f4" containerName="kserve-container"
Sep 18 16:39:58.762644 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.762609    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="af63257f-9ae7-477a-99c0-0e09227687f4" containerName="kserve-container"
Sep 18 16:39:58.762644 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.762620    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="af63257f-9ae7-477a-99c0-0e09227687f4" containerName="kube-rbac-proxy"
Sep 18 16:39:58.765764 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.765745    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"
Sep 18 16:39:58.768328 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.768065    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-036a5-predictor-serving-cert\""
Sep 18 16:39:58.768328 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.768103    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 16:39:58.768328 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.768199    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-036a5-kube-rbac-proxy-sar-config\""
Sep 18 16:39:58.768558 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.768402    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 16:39:58.774944 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.774923    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"]
Sep 18 16:39:58.812832 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.812799    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-proxy-tls\") pod \"isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f\" (UID: \"6e52ac96-fa9e-4b45-8301-60f253b3d3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"
Sep 18 16:39:58.813007 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.812972    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f\" (UID: \"6e52ac96-fa9e-4b45-8301-60f253b3d3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"
Sep 18 16:39:58.813102 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.813027    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-cabundle-cert\") pod \"isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f\" (UID: \"6e52ac96-fa9e-4b45-8301-60f253b3d3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"
Sep 18 16:39:58.813102 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.813070    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4ccn\" (UniqueName: \"kubernetes.io/projected/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-kube-api-access-v4ccn\") pod \"isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f\" (UID: \"6e52ac96-fa9e-4b45-8301-60f253b3d3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"
Sep 18 16:39:58.813219 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.813136    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f\" (UID: \"6e52ac96-fa9e-4b45-8301-60f253b3d3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"
Sep 18 16:39:58.904119 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.904096    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m"
Sep 18 16:39:58.913525 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.913506    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpcfh\" (UniqueName: \"kubernetes.io/projected/ab1b734f-69ba-40f0-b82e-c6a9a72c4711-kube-api-access-zpcfh\") pod \"ab1b734f-69ba-40f0-b82e-c6a9a72c4711\" (UID: \"ab1b734f-69ba-40f0-b82e-c6a9a72c4711\") "
Sep 18 16:39:58.913608 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.913535    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab1b734f-69ba-40f0-b82e-c6a9a72c4711-proxy-tls\") pod \"ab1b734f-69ba-40f0-b82e-c6a9a72c4711\" (UID: \"ab1b734f-69ba-40f0-b82e-c6a9a72c4711\") "
Sep 18 16:39:58.913608 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.913563    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab1b734f-69ba-40f0-b82e-c6a9a72c4711-kube-rbac-proxy-sar-config\") pod \"ab1b734f-69ba-40f0-b82e-c6a9a72c4711\" (UID: \"ab1b734f-69ba-40f0-b82e-c6a9a72c4711\") "
Sep 18 16:39:58.913769 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.913749    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-proxy-tls\") pod \"isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f\" (UID: \"6e52ac96-fa9e-4b45-8301-60f253b3d3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"
Sep 18 16:39:58.913875 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.913840    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f\" (UID: \"6e52ac96-fa9e-4b45-8301-60f253b3d3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"
Sep 18 16:39:58.913948 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.913896    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-cabundle-cert\") pod \"isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f\" (UID: \"6e52ac96-fa9e-4b45-8301-60f253b3d3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"
Sep 18 16:39:58.913948 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:39:58.913905    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-serving-cert: secret "isvc-sklearn-scale-raw-036a5-predictor-serving-cert" not found
Sep 18 16:39:58.913948 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.913939    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4ccn\" (UniqueName: \"kubernetes.io/projected/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-kube-api-access-v4ccn\") pod \"isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f\" (UID: \"6e52ac96-fa9e-4b45-8301-60f253b3d3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"
Sep 18 16:39:58.914125 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.913967    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab1b734f-69ba-40f0-b82e-c6a9a72c4711-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ab1b734f-69ba-40f0-b82e-c6a9a72c4711" (UID: "ab1b734f-69ba-40f0-b82e-c6a9a72c4711"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:39:58.914125 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:39:58.913978    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-proxy-tls podName:6e52ac96-fa9e-4b45-8301-60f253b3d3b5 nodeName:}" failed. No retries permitted until 2026-09-18 16:39:59.413958428 +0000 UTC m=+1437.869411506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-proxy-tls") pod "isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" (UID: "6e52ac96-fa9e-4b45-8301-60f253b3d3b5") : secret "isvc-sklearn-scale-raw-036a5-predictor-serving-cert" not found
Sep 18 16:39:58.914125 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.914005    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f\" (UID: \"6e52ac96-fa9e-4b45-8301-60f253b3d3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"
Sep 18 16:39:58.914363 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.914312    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f\" (UID: \"6e52ac96-fa9e-4b45-8301-60f253b3d3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"
Sep 18 16:39:58.914768 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.914748    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f\" (UID: \"6e52ac96-fa9e-4b45-8301-60f253b3d3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"
Sep 18 16:39:58.914768 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.914758    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-cabundle-cert\") pod \"isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f\" (UID: \"6e52ac96-fa9e-4b45-8301-60f253b3d3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"
Sep 18 16:39:58.915967 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.915942    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1b734f-69ba-40f0-b82e-c6a9a72c4711-kube-api-access-zpcfh" (OuterVolumeSpecName: "kube-api-access-zpcfh") pod "ab1b734f-69ba-40f0-b82e-c6a9a72c4711" (UID: "ab1b734f-69ba-40f0-b82e-c6a9a72c4711"). InnerVolumeSpecName "kube-api-access-zpcfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:39:58.915967 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.915952    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab1b734f-69ba-40f0-b82e-c6a9a72c4711-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ab1b734f-69ba-40f0-b82e-c6a9a72c4711" (UID: "ab1b734f-69ba-40f0-b82e-c6a9a72c4711"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:39:58.925211 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:58.925192    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4ccn\" (UniqueName: \"kubernetes.io/projected/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-kube-api-access-v4ccn\") pod \"isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f\" (UID: \"6e52ac96-fa9e-4b45-8301-60f253b3d3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"
Sep 18 16:39:59.014798 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:59.014727    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpcfh\" (UniqueName: \"kubernetes.io/projected/ab1b734f-69ba-40f0-b82e-c6a9a72c4711-kube-api-access-zpcfh\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:39:59.014798 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:59.014762    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab1b734f-69ba-40f0-b82e-c6a9a72c4711-proxy-tls\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:39:59.014798 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:59.014773    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab1b734f-69ba-40f0-b82e-c6a9a72c4711-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:39:59.206613 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:59.206581    2566 generic.go:356] "Generic (PLEG): container finished" podID="ab1b734f-69ba-40f0-b82e-c6a9a72c4711" containerID="0daf6d936eab75e097c5fd60d6fc8210b5d227cf7b1523cb23f108ffcf710432" exitCode=2
Sep 18 16:39:59.206613 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:59.206608    2566 generic.go:356] "Generic (PLEG): container finished" podID="ab1b734f-69ba-40f0-b82e-c6a9a72c4711" containerID="6ee202d69509909000f07678438c1bbf126adfcdb888386b071f60fc064b4e4a" exitCode=2
Sep 18 16:39:59.206788 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:59.206633    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m"
Sep 18 16:39:59.206788 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:59.206647    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m" event={"ID":"ab1b734f-69ba-40f0-b82e-c6a9a72c4711","Type":"ContainerDied","Data":"0daf6d936eab75e097c5fd60d6fc8210b5d227cf7b1523cb23f108ffcf710432"}
Sep 18 16:39:59.206788 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:59.206695    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m" event={"ID":"ab1b734f-69ba-40f0-b82e-c6a9a72c4711","Type":"ContainerDied","Data":"6ee202d69509909000f07678438c1bbf126adfcdb888386b071f60fc064b4e4a"}
Sep 18 16:39:59.206788 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:59.206717    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m" event={"ID":"ab1b734f-69ba-40f0-b82e-c6a9a72c4711","Type":"ContainerDied","Data":"be641f56b4cc3787ac67f8110bf7cee1ef3b5a8900ce2951f05321383df0eda4"}
Sep 18 16:39:59.206788 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:59.206739    2566 scope.go:117] "RemoveContainer" containerID="0daf6d936eab75e097c5fd60d6fc8210b5d227cf7b1523cb23f108ffcf710432"
Sep 18 16:39:59.215211 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:59.215191    2566 scope.go:117] "RemoveContainer" containerID="6ee202d69509909000f07678438c1bbf126adfcdb888386b071f60fc064b4e4a"
Sep 18 16:39:59.222547 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:59.222532    2566 scope.go:117] "RemoveContainer" containerID="0daf6d936eab75e097c5fd60d6fc8210b5d227cf7b1523cb23f108ffcf710432"
Sep 18 16:39:59.222793 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:39:59.222762    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0daf6d936eab75e097c5fd60d6fc8210b5d227cf7b1523cb23f108ffcf710432\": container with ID starting with 0daf6d936eab75e097c5fd60d6fc8210b5d227cf7b1523cb23f108ffcf710432 not found: ID does not exist" containerID="0daf6d936eab75e097c5fd60d6fc8210b5d227cf7b1523cb23f108ffcf710432"
Sep 18 16:39:59.222889 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:59.222799    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0daf6d936eab75e097c5fd60d6fc8210b5d227cf7b1523cb23f108ffcf710432"} err="failed to get container status \"0daf6d936eab75e097c5fd60d6fc8210b5d227cf7b1523cb23f108ffcf710432\": rpc error: code = NotFound desc = could not find container \"0daf6d936eab75e097c5fd60d6fc8210b5d227cf7b1523cb23f108ffcf710432\": container with ID starting with 0daf6d936eab75e097c5fd60d6fc8210b5d227cf7b1523cb23f108ffcf710432 not found: ID does not exist"
Sep 18 16:39:59.222889 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:59.222814    2566 scope.go:117] "RemoveContainer" containerID="6ee202d69509909000f07678438c1bbf126adfcdb888386b071f60fc064b4e4a"
Sep 18 16:39:59.223099 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:39:59.223064    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ee202d69509909000f07678438c1bbf126adfcdb888386b071f60fc064b4e4a\": container with ID starting with 6ee202d69509909000f07678438c1bbf126adfcdb888386b071f60fc064b4e4a not found: ID does not exist" containerID="6ee202d69509909000f07678438c1bbf126adfcdb888386b071f60fc064b4e4a"
Sep 18 16:39:59.223170 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:59.223105    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee202d69509909000f07678438c1bbf126adfcdb888386b071f60fc064b4e4a"} err="failed to get container status \"6ee202d69509909000f07678438c1bbf126adfcdb888386b071f60fc064b4e4a\": rpc error: code = NotFound desc = could not find container \"6ee202d69509909000f07678438c1bbf126adfcdb888386b071f60fc064b4e4a\": container with ID starting with 6ee202d69509909000f07678438c1bbf126adfcdb888386b071f60fc064b4e4a not found: ID does not exist"
Sep 18 16:39:59.223170 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:59.223120    2566 scope.go:117] "RemoveContainer" containerID="0daf6d936eab75e097c5fd60d6fc8210b5d227cf7b1523cb23f108ffcf710432"
Sep 18 16:39:59.223379 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:59.223362    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0daf6d936eab75e097c5fd60d6fc8210b5d227cf7b1523cb23f108ffcf710432"} err="failed to get container status \"0daf6d936eab75e097c5fd60d6fc8210b5d227cf7b1523cb23f108ffcf710432\": rpc error: code = NotFound desc = could not find container \"0daf6d936eab75e097c5fd60d6fc8210b5d227cf7b1523cb23f108ffcf710432\": container with ID starting with 0daf6d936eab75e097c5fd60d6fc8210b5d227cf7b1523cb23f108ffcf710432 not found: ID does not exist"
Sep 18 16:39:59.223452 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:59.223381    2566 scope.go:117] "RemoveContainer" containerID="6ee202d69509909000f07678438c1bbf126adfcdb888386b071f60fc064b4e4a"
Sep 18 16:39:59.223637 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:59.223618    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee202d69509909000f07678438c1bbf126adfcdb888386b071f60fc064b4e4a"} err="failed to get container status \"6ee202d69509909000f07678438c1bbf126adfcdb888386b071f60fc064b4e4a\": rpc error: code = NotFound desc = could not find container \"6ee202d69509909000f07678438c1bbf126adfcdb888386b071f60fc064b4e4a\": container with ID starting with 6ee202d69509909000f07678438c1bbf126adfcdb888386b071f60fc064b4e4a not found: ID does not exist"
Sep 18 16:39:59.228985 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:59.228963    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m"]
Sep 18 16:39:59.231326 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:59.231306    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-031b1-predictor-57d74fc557-tcw9m"]
Sep 18 16:39:59.416917 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:59.416889    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-proxy-tls\") pod \"isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f\" (UID: \"6e52ac96-fa9e-4b45-8301-60f253b3d3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"
Sep 18 16:39:59.417108 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:39:59.417089    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-serving-cert: secret "isvc-sklearn-scale-raw-036a5-predictor-serving-cert" not found
Sep 18 16:39:59.417183 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:39:59.417173    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-proxy-tls podName:6e52ac96-fa9e-4b45-8301-60f253b3d3b5 nodeName:}" failed. No retries permitted until 2026-09-18 16:40:00.417154192 +0000 UTC m=+1438.872607254 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-proxy-tls") pod "isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" (UID: "6e52ac96-fa9e-4b45-8301-60f253b3d3b5") : secret "isvc-sklearn-scale-raw-036a5-predictor-serving-cert" not found
Sep 18 16:39:59.685719 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:39:59.685651    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab1b734f-69ba-40f0-b82e-c6a9a72c4711" path="/var/lib/kubelet/pods/ab1b734f-69ba-40f0-b82e-c6a9a72c4711/volumes"
Sep 18 16:40:00.425045 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:40:00.425002    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-proxy-tls\") pod \"isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f\" (UID: \"6e52ac96-fa9e-4b45-8301-60f253b3d3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"
Sep 18 16:40:00.427578 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:40:00.427543    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-proxy-tls\") pod \"isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f\" (UID: \"6e52ac96-fa9e-4b45-8301-60f253b3d3b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"
Sep 18 16:40:00.577215 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:40:00.577169    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"
Sep 18 16:40:00.853282 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:40:00.853233    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"]
Sep 18 16:40:00.857034 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:40:00.857006    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e52ac96_fa9e_4b45_8301_60f253b3d3b5.slice/crio-a5c7bb7b002b47fb414543ee434469f4cb67c6bbde2c5b386ab2e80c68219e92 WatchSource:0}: Error finding container a5c7bb7b002b47fb414543ee434469f4cb67c6bbde2c5b386ab2e80c68219e92: Status 404 returned error can't find the container with id a5c7bb7b002b47fb414543ee434469f4cb67c6bbde2c5b386ab2e80c68219e92
Sep 18 16:40:01.216630 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:40:01.216543    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" event={"ID":"6e52ac96-fa9e-4b45-8301-60f253b3d3b5","Type":"ContainerStarted","Data":"ab1c79ddbe74b921db94ef80c09c8c4b2ae3270ad5a684e6a35716351ac72b3b"}
Sep 18 16:40:01.216630 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:40:01.216605    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" event={"ID":"6e52ac96-fa9e-4b45-8301-60f253b3d3b5","Type":"ContainerStarted","Data":"a5c7bb7b002b47fb414543ee434469f4cb67c6bbde2c5b386ab2e80c68219e92"}
Sep 18 16:40:05.234303 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:40:05.234212    2566 generic.go:356] "Generic (PLEG): container finished" podID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerID="ab1c79ddbe74b921db94ef80c09c8c4b2ae3270ad5a684e6a35716351ac72b3b" exitCode=0
Sep 18 16:40:05.234303 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:40:05.234289    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" event={"ID":"6e52ac96-fa9e-4b45-8301-60f253b3d3b5","Type":"ContainerDied","Data":"ab1c79ddbe74b921db94ef80c09c8c4b2ae3270ad5a684e6a35716351ac72b3b"}
Sep 18 16:40:06.240378 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:40:06.240342    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" event={"ID":"6e52ac96-fa9e-4b45-8301-60f253b3d3b5","Type":"ContainerStarted","Data":"54bb94a2e34dc52ae7eb64fdf5b29093c6422897daf16c58a0288609c08bfaa3"}
Sep 18 16:40:06.240853 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:40:06.240413    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" event={"ID":"6e52ac96-fa9e-4b45-8301-60f253b3d3b5","Type":"ContainerStarted","Data":"caa92fbbcfce3912314dc956e7343316bbc2842d0fed146413a8da9e65ca5b0c"}
Sep 18 16:40:06.240853 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:40:06.240635    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"
Sep 18 16:40:06.262411 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:40:06.262362    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" podStartSLOduration=4.245918252 podStartE2EDuration="8.262345927s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.016427675s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:40:06.26106782 +0000 UTC m=+1444.716520899" watchObservedRunningTime="2026-09-18 16:40:06.262345927 +0000 UTC m=+1444.717799004"
Sep 18 16:40:07.244062 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:40:07.244026    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"
Sep 18 16:40:07.245455 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:40:07.245425    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 16:40:08.248176 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:40:08.248130    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 16:40:13.253554 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:40:13.253521    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"
Sep 18 16:40:13.254258 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:40:13.254223    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 16:40:23.254370 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:40:23.254328    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 16:40:33.254602 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:40:33.254562    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 16:40:43.254598 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:40:43.254557    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 16:40:53.254166 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:40:53.254124    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 16:41:03.254474 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:41:03.254424    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 16:41:03.669907 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:41:03.669881    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-vfmdv_1e3ca1ec-6856-4afb-850e-85e3fa968b3f/console-operator/1.log"
Sep 18 16:41:03.687006 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:41:03.686985    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-vfmdv_1e3ca1ec-6856-4afb-850e-85e3fa968b3f/console-operator/1.log"
Sep 18 16:41:13.254628 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:41:13.254589    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 16:41:23.254391 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:41:23.254308    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 16:41:33.254291 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:41:33.254252    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 16:41:42.681985 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:41:42.681939    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 16:41:52.682227 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:41:52.682187    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 16:42:02.682069 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:02.682029    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 16:42:12.681957 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:12.681916    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 16:42:22.682587 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:22.682547    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 16:42:32.682252 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:32.682211    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 16:42:42.682638 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:42.682597    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 16:42:52.682790 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:52.682757    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"
Sep 18 16:42:57.499753 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:57.499715    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"]
Sep 18 16:42:57.500187 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:57.500171    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab1b734f-69ba-40f0-b82e-c6a9a72c4711" containerName="kserve-container"
Sep 18 16:42:57.500315 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:57.500189    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab1b734f-69ba-40f0-b82e-c6a9a72c4711" containerName="kserve-container"
Sep 18 16:42:57.500315 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:57.500208    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab1b734f-69ba-40f0-b82e-c6a9a72c4711" containerName="kube-rbac-proxy"
Sep 18 16:42:57.500315 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:57.500219    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab1b734f-69ba-40f0-b82e-c6a9a72c4711" containerName="kube-rbac-proxy"
Sep 18 16:42:57.500315 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:57.500308    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab1b734f-69ba-40f0-b82e-c6a9a72c4711" containerName="kube-rbac-proxy"
Sep 18 16:42:57.500475 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:57.500318    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab1b734f-69ba-40f0-b82e-c6a9a72c4711" containerName="kserve-container"
Sep 18 16:42:57.503883 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:57.503864    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"
Sep 18 16:42:57.506518 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:57.506498    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 18 16:42:57.506615 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:57.506498    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 18 16:42:57.514380 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:57.514355    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"]
Sep 18 16:42:57.569704 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:57.569674    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkxfd\" (UniqueName: \"kubernetes.io/projected/e42b2760-c8a8-43f2-abc6-413d29f7c34c-kube-api-access-lkxfd\") pod \"isvc-canary-create-predictor-76b74f4d6c-xs4ct\" (UID: \"e42b2760-c8a8-43f2-abc6-413d29f7c34c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"
Sep 18 16:42:57.569871 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:57.569713    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e42b2760-c8a8-43f2-abc6-413d29f7c34c-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-xs4ct\" (UID: \"e42b2760-c8a8-43f2-abc6-413d29f7c34c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"
Sep 18 16:42:57.569871 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:57.569746    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e42b2760-c8a8-43f2-abc6-413d29f7c34c-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-xs4ct\" (UID: \"e42b2760-c8a8-43f2-abc6-413d29f7c34c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"
Sep 18 16:42:57.569871 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:57.569762    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e42b2760-c8a8-43f2-abc6-413d29f7c34c-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-xs4ct\" (UID: \"e42b2760-c8a8-43f2-abc6-413d29f7c34c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"
Sep 18 16:42:57.569871 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:57.569800    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e42b2760-c8a8-43f2-abc6-413d29f7c34c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-xs4ct\" (UID: \"e42b2760-c8a8-43f2-abc6-413d29f7c34c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"
Sep 18 16:42:57.671197 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:57.671158    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e42b2760-c8a8-43f2-abc6-413d29f7c34c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-xs4ct\" (UID: \"e42b2760-c8a8-43f2-abc6-413d29f7c34c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"
Sep 18 16:42:57.671409 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:57.671231    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkxfd\" (UniqueName: \"kubernetes.io/projected/e42b2760-c8a8-43f2-abc6-413d29f7c34c-kube-api-access-lkxfd\") pod \"isvc-canary-create-predictor-76b74f4d6c-xs4ct\" (UID: \"e42b2760-c8a8-43f2-abc6-413d29f7c34c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"
Sep 18 16:42:57.671409 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:57.671258    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e42b2760-c8a8-43f2-abc6-413d29f7c34c-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-xs4ct\" (UID: \"e42b2760-c8a8-43f2-abc6-413d29f7c34c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"
Sep 18 16:42:57.671409 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:57.671290    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e42b2760-c8a8-43f2-abc6-413d29f7c34c-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-xs4ct\" (UID: \"e42b2760-c8a8-43f2-abc6-413d29f7c34c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"
Sep 18 16:42:57.671409 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:57.671307    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e42b2760-c8a8-43f2-abc6-413d29f7c34c-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-xs4ct\" (UID: \"e42b2760-c8a8-43f2-abc6-413d29f7c34c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"
Sep 18 16:42:57.671791 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:57.671761    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e42b2760-c8a8-43f2-abc6-413d29f7c34c-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-xs4ct\" (UID: \"e42b2760-c8a8-43f2-abc6-413d29f7c34c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"
Sep 18 16:42:57.671925 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:57.671904    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e42b2760-c8a8-43f2-abc6-413d29f7c34c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-xs4ct\" (UID: \"e42b2760-c8a8-43f2-abc6-413d29f7c34c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"
Sep 18 16:42:57.672001 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:57.671938    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e42b2760-c8a8-43f2-abc6-413d29f7c34c-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-xs4ct\" (UID: \"e42b2760-c8a8-43f2-abc6-413d29f7c34c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"
Sep 18 16:42:57.673717 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:57.673687    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e42b2760-c8a8-43f2-abc6-413d29f7c34c-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-xs4ct\" (UID: \"e42b2760-c8a8-43f2-abc6-413d29f7c34c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"
Sep 18 16:42:57.682449 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:57.682425    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkxfd\" (UniqueName: \"kubernetes.io/projected/e42b2760-c8a8-43f2-abc6-413d29f7c34c-kube-api-access-lkxfd\") pod \"isvc-canary-create-predictor-76b74f4d6c-xs4ct\" (UID: \"e42b2760-c8a8-43f2-abc6-413d29f7c34c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"
Sep 18 16:42:57.814911 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:57.814833    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"
Sep 18 16:42:58.087966 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:58.087905    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"]
Sep 18 16:42:58.090724 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:42:58.090692    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42b2760_c8a8_43f2_abc6_413d29f7c34c.slice/crio-156b35fe0fad0a7c4575ac8de9cb880cf2aba9f701404fafae7cd171f756ca49 WatchSource:0}: Error finding container 156b35fe0fad0a7c4575ac8de9cb880cf2aba9f701404fafae7cd171f756ca49: Status 404 returned error can't find the container with id 156b35fe0fad0a7c4575ac8de9cb880cf2aba9f701404fafae7cd171f756ca49
Sep 18 16:42:58.901647 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:58.901612    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct" event={"ID":"e42b2760-c8a8-43f2-abc6-413d29f7c34c","Type":"ContainerStarted","Data":"68510d47da13740cd01f23da18f52f5178d22c5c2569bac4a7e97f0306847f49"}
Sep 18 16:42:58.902009 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:58.901670    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct" event={"ID":"e42b2760-c8a8-43f2-abc6-413d29f7c34c","Type":"ContainerStarted","Data":"156b35fe0fad0a7c4575ac8de9cb880cf2aba9f701404fafae7cd171f756ca49"}
Sep 18 16:42:59.828464 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:59.828431    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"]
Sep 18 16:42:59.828705 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:59.828684    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kserve-container" containerID="cri-o://caa92fbbcfce3912314dc956e7343316bbc2842d0fed146413a8da9e65ca5b0c" gracePeriod=30
Sep 18 16:42:59.828779 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:42:59.828751    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kube-rbac-proxy" containerID="cri-o://54bb94a2e34dc52ae7eb64fdf5b29093c6422897daf16c58a0288609c08bfaa3" gracePeriod=30
Sep 18 16:43:00.911934 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:00.911902    2566 generic.go:356] "Generic (PLEG): container finished" podID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerID="54bb94a2e34dc52ae7eb64fdf5b29093c6422897daf16c58a0288609c08bfaa3" exitCode=2
Sep 18 16:43:00.912309 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:00.911969    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" event={"ID":"6e52ac96-fa9e-4b45-8301-60f253b3d3b5","Type":"ContainerDied","Data":"54bb94a2e34dc52ae7eb64fdf5b29093c6422897daf16c58a0288609c08bfaa3"}
Sep 18 16:43:02.682200 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:02.682154    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 16:43:02.921914 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:02.921882    2566 generic.go:356] "Generic (PLEG): container finished" podID="e42b2760-c8a8-43f2-abc6-413d29f7c34c" containerID="68510d47da13740cd01f23da18f52f5178d22c5c2569bac4a7e97f0306847f49" exitCode=0
Sep 18 16:43:02.922117 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:02.921958    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct" event={"ID":"e42b2760-c8a8-43f2-abc6-413d29f7c34c","Type":"ContainerDied","Data":"68510d47da13740cd01f23da18f52f5178d22c5c2569bac4a7e97f0306847f49"}
Sep 18 16:43:03.248436 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:03.248341    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 18 16:43:03.928916 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:03.928883    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct" event={"ID":"e42b2760-c8a8-43f2-abc6-413d29f7c34c","Type":"ContainerStarted","Data":"faafe2e219fa1bafb1914198ed3e533c72864b5cafbff5f3ae8bacc2aace5a7c"}
Sep 18 16:43:03.929382 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:03.928946    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct" event={"ID":"e42b2760-c8a8-43f2-abc6-413d29f7c34c","Type":"ContainerStarted","Data":"a1e8d846a18ea69ce242d309bab3ef6b9d6449a6191b90c3ed2d5950eca99064"}
Sep 18 16:43:03.929382 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:03.929198    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"
Sep 18 16:43:03.952450 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:03.952397    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct" podStartSLOduration=3.077613638 podStartE2EDuration="6.952379782s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.874766144s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:43:03.949882751 +0000 UTC m=+1622.405335828" watchObservedRunningTime="2026-09-18 16:43:03.952379782 +0000 UTC m=+1622.407832860"
Sep 18 16:43:04.938243 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:04.938207    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"
Sep 18 16:43:04.939750 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:04.939721    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct" podUID="e42b2760-c8a8-43f2-abc6-413d29f7c34c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 18 16:43:05.942265 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:05.942225    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct" podUID="e42b2760-c8a8-43f2-abc6-413d29f7c34c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 18 16:43:08.249300 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:08.249256    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 18 16:43:10.368164 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.368130    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"
Sep 18 16:43:10.479269 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.479182    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-kserve-provision-location\") pod \"6e52ac96-fa9e-4b45-8301-60f253b3d3b5\" (UID: \"6e52ac96-fa9e-4b45-8301-60f253b3d3b5\") "
Sep 18 16:43:10.479445 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.479271    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4ccn\" (UniqueName: \"kubernetes.io/projected/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-kube-api-access-v4ccn\") pod \"6e52ac96-fa9e-4b45-8301-60f253b3d3b5\" (UID: \"6e52ac96-fa9e-4b45-8301-60f253b3d3b5\") "
Sep 18 16:43:10.479445 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.479306    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-proxy-tls\") pod \"6e52ac96-fa9e-4b45-8301-60f253b3d3b5\" (UID: \"6e52ac96-fa9e-4b45-8301-60f253b3d3b5\") "
Sep 18 16:43:10.479445 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.479345    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-cabundle-cert\") pod \"6e52ac96-fa9e-4b45-8301-60f253b3d3b5\" (UID: \"6e52ac96-fa9e-4b45-8301-60f253b3d3b5\") "
Sep 18 16:43:10.479633 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.479443    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-kube-rbac-proxy-sar-config\") pod \"6e52ac96-fa9e-4b45-8301-60f253b3d3b5\" (UID: \"6e52ac96-fa9e-4b45-8301-60f253b3d3b5\") "
Sep 18 16:43:10.479633 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.479536    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6e52ac96-fa9e-4b45-8301-60f253b3d3b5" (UID: "6e52ac96-fa9e-4b45-8301-60f253b3d3b5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:43:10.479745 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.479728    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-kserve-provision-location\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:43:10.479811 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.479665    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6e52ac96-fa9e-4b45-8301-60f253b3d3b5" (UID: "6e52ac96-fa9e-4b45-8301-60f253b3d3b5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:43:10.479867 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.479826    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6e52ac96-fa9e-4b45-8301-60f253b3d3b5" (UID: "6e52ac96-fa9e-4b45-8301-60f253b3d3b5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:43:10.481369 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.481344    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6e52ac96-fa9e-4b45-8301-60f253b3d3b5" (UID: "6e52ac96-fa9e-4b45-8301-60f253b3d3b5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:43:10.481468 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.481457    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-kube-api-access-v4ccn" (OuterVolumeSpecName: "kube-api-access-v4ccn") pod "6e52ac96-fa9e-4b45-8301-60f253b3d3b5" (UID: "6e52ac96-fa9e-4b45-8301-60f253b3d3b5"). InnerVolumeSpecName "kube-api-access-v4ccn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:43:10.580485 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.580456    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-proxy-tls\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:43:10.580485 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.580478    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-cabundle-cert\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:43:10.580485 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.580488    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:43:10.580695 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.580497    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4ccn\" (UniqueName: \"kubernetes.io/projected/6e52ac96-fa9e-4b45-8301-60f253b3d3b5-kube-api-access-v4ccn\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:43:10.947232 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.947206    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"
Sep 18 16:43:10.947850 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.947814    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct" podUID="e42b2760-c8a8-43f2-abc6-413d29f7c34c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 18 16:43:10.963612 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.963588    2566 generic.go:356] "Generic (PLEG): container finished" podID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerID="caa92fbbcfce3912314dc956e7343316bbc2842d0fed146413a8da9e65ca5b0c" exitCode=0
Sep 18 16:43:10.963718 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.963639    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"
Sep 18 16:43:10.963765 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.963641    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" event={"ID":"6e52ac96-fa9e-4b45-8301-60f253b3d3b5","Type":"ContainerDied","Data":"caa92fbbcfce3912314dc956e7343316bbc2842d0fed146413a8da9e65ca5b0c"}
Sep 18 16:43:10.963805 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.963765    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f" event={"ID":"6e52ac96-fa9e-4b45-8301-60f253b3d3b5","Type":"ContainerDied","Data":"a5c7bb7b002b47fb414543ee434469f4cb67c6bbde2c5b386ab2e80c68219e92"}
Sep 18 16:43:10.963805 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.963802    2566 scope.go:117] "RemoveContainer" containerID="54bb94a2e34dc52ae7eb64fdf5b29093c6422897daf16c58a0288609c08bfaa3"
Sep 18 16:43:10.973305 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.973289    2566 scope.go:117] "RemoveContainer" containerID="caa92fbbcfce3912314dc956e7343316bbc2842d0fed146413a8da9e65ca5b0c"
Sep 18 16:43:10.980533 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.980517    2566 scope.go:117] "RemoveContainer" containerID="ab1c79ddbe74b921db94ef80c09c8c4b2ae3270ad5a684e6a35716351ac72b3b"
Sep 18 16:43:10.986621 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.986599    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"]
Sep 18 16:43:10.987734 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.987718    2566 scope.go:117] "RemoveContainer" containerID="54bb94a2e34dc52ae7eb64fdf5b29093c6422897daf16c58a0288609c08bfaa3"
Sep 18 16:43:10.987998 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:43:10.987978    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54bb94a2e34dc52ae7eb64fdf5b29093c6422897daf16c58a0288609c08bfaa3\": container with ID starting with 54bb94a2e34dc52ae7eb64fdf5b29093c6422897daf16c58a0288609c08bfaa3 not found: ID does not exist" containerID="54bb94a2e34dc52ae7eb64fdf5b29093c6422897daf16c58a0288609c08bfaa3"
Sep 18 16:43:10.988057 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.988008    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54bb94a2e34dc52ae7eb64fdf5b29093c6422897daf16c58a0288609c08bfaa3"} err="failed to get container status \"54bb94a2e34dc52ae7eb64fdf5b29093c6422897daf16c58a0288609c08bfaa3\": rpc error: code = NotFound desc = could not find container \"54bb94a2e34dc52ae7eb64fdf5b29093c6422897daf16c58a0288609c08bfaa3\": container with ID starting with 54bb94a2e34dc52ae7eb64fdf5b29093c6422897daf16c58a0288609c08bfaa3 not found: ID does not exist"
Sep 18 16:43:10.988057 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.988025    2566 scope.go:117] "RemoveContainer" containerID="caa92fbbcfce3912314dc956e7343316bbc2842d0fed146413a8da9e65ca5b0c"
Sep 18 16:43:10.988302 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:43:10.988282    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caa92fbbcfce3912314dc956e7343316bbc2842d0fed146413a8da9e65ca5b0c\": container with ID starting with caa92fbbcfce3912314dc956e7343316bbc2842d0fed146413a8da9e65ca5b0c not found: ID does not exist" containerID="caa92fbbcfce3912314dc956e7343316bbc2842d0fed146413a8da9e65ca5b0c"
Sep 18 16:43:10.988357 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.988306    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caa92fbbcfce3912314dc956e7343316bbc2842d0fed146413a8da9e65ca5b0c"} err="failed to get container status \"caa92fbbcfce3912314dc956e7343316bbc2842d0fed146413a8da9e65ca5b0c\": rpc error: code = NotFound desc = could not find container \"caa92fbbcfce3912314dc956e7343316bbc2842d0fed146413a8da9e65ca5b0c\": container with ID starting with caa92fbbcfce3912314dc956e7343316bbc2842d0fed146413a8da9e65ca5b0c not found: ID does not exist"
Sep 18 16:43:10.988357 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.988320    2566 scope.go:117] "RemoveContainer" containerID="ab1c79ddbe74b921db94ef80c09c8c4b2ae3270ad5a684e6a35716351ac72b3b"
Sep 18 16:43:10.988601 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:43:10.988582    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab1c79ddbe74b921db94ef80c09c8c4b2ae3270ad5a684e6a35716351ac72b3b\": container with ID starting with ab1c79ddbe74b921db94ef80c09c8c4b2ae3270ad5a684e6a35716351ac72b3b not found: ID does not exist" containerID="ab1c79ddbe74b921db94ef80c09c8c4b2ae3270ad5a684e6a35716351ac72b3b"
Sep 18 16:43:10.988667 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.988607    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab1c79ddbe74b921db94ef80c09c8c4b2ae3270ad5a684e6a35716351ac72b3b"} err="failed to get container status \"ab1c79ddbe74b921db94ef80c09c8c4b2ae3270ad5a684e6a35716351ac72b3b\": rpc error: code = NotFound desc = could not find container \"ab1c79ddbe74b921db94ef80c09c8c4b2ae3270ad5a684e6a35716351ac72b3b\": container with ID starting with ab1c79ddbe74b921db94ef80c09c8c4b2ae3270ad5a684e6a35716351ac72b3b not found: ID does not exist"
Sep 18 16:43:10.990970 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:10.990948    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-036a5-predictor-74b5966d96-rrz2f"]
Sep 18 16:43:11.685875 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:11.685845    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" path="/var/lib/kubelet/pods/6e52ac96-fa9e-4b45-8301-60f253b3d3b5/volumes"
Sep 18 16:43:20.948619 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:20.948581    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct" podUID="e42b2760-c8a8-43f2-abc6-413d29f7c34c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 18 16:43:30.948468 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:30.948430    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct" podUID="e42b2760-c8a8-43f2-abc6-413d29f7c34c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 18 16:43:40.948319 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:40.948280    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct" podUID="e42b2760-c8a8-43f2-abc6-413d29f7c34c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 18 16:43:50.948774 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:43:50.948721    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct" podUID="e42b2760-c8a8-43f2-abc6-413d29f7c34c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 18 16:44:00.947911 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:00.947871    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct" podUID="e42b2760-c8a8-43f2-abc6-413d29f7c34c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 18 16:44:10.947776 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:10.947729    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct" podUID="e42b2760-c8a8-43f2-abc6-413d29f7c34c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 18 16:44:20.351231 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:20.351152    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"]
Sep 18 16:44:20.351737 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:20.351416    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct" podUID="e42b2760-c8a8-43f2-abc6-413d29f7c34c" containerName="kserve-container" containerID="cri-o://a1e8d846a18ea69ce242d309bab3ef6b9d6449a6191b90c3ed2d5950eca99064" gracePeriod=30
Sep 18 16:44:20.351737 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:20.351453    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct" podUID="e42b2760-c8a8-43f2-abc6-413d29f7c34c" containerName="kube-rbac-proxy" containerID="cri-o://faafe2e219fa1bafb1914198ed3e533c72864b5cafbff5f3ae8bacc2aace5a7c" gracePeriod=30
Sep 18 16:44:20.352258 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:20.352240    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"
Sep 18 16:44:20.943246 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:20.943199    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct" podUID="e42b2760-c8a8-43f2-abc6-413d29f7c34c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 18 16:44:21.240050 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:21.239959    2566 generic.go:356] "Generic (PLEG): container finished" podID="e42b2760-c8a8-43f2-abc6-413d29f7c34c" containerID="faafe2e219fa1bafb1914198ed3e533c72864b5cafbff5f3ae8bacc2aace5a7c" exitCode=2
Sep 18 16:44:21.240050 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:21.240031    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct" event={"ID":"e42b2760-c8a8-43f2-abc6-413d29f7c34c","Type":"ContainerDied","Data":"faafe2e219fa1bafb1914198ed3e533c72864b5cafbff5f3ae8bacc2aace5a7c"}
Sep 18 16:44:24.887708 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:24.887683    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"
Sep 18 16:44:24.997826 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:24.997748    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkxfd\" (UniqueName: \"kubernetes.io/projected/e42b2760-c8a8-43f2-abc6-413d29f7c34c-kube-api-access-lkxfd\") pod \"e42b2760-c8a8-43f2-abc6-413d29f7c34c\" (UID: \"e42b2760-c8a8-43f2-abc6-413d29f7c34c\") "
Sep 18 16:44:24.997826 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:24.997798    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e42b2760-c8a8-43f2-abc6-413d29f7c34c-kserve-provision-location\") pod \"e42b2760-c8a8-43f2-abc6-413d29f7c34c\" (UID: \"e42b2760-c8a8-43f2-abc6-413d29f7c34c\") "
Sep 18 16:44:24.998029 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:24.997831    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e42b2760-c8a8-43f2-abc6-413d29f7c34c-proxy-tls\") pod \"e42b2760-c8a8-43f2-abc6-413d29f7c34c\" (UID: \"e42b2760-c8a8-43f2-abc6-413d29f7c34c\") "
Sep 18 16:44:24.998029 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:24.997850    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e42b2760-c8a8-43f2-abc6-413d29f7c34c-kube-rbac-proxy-sar-config\") pod \"e42b2760-c8a8-43f2-abc6-413d29f7c34c\" (UID: \"e42b2760-c8a8-43f2-abc6-413d29f7c34c\") "
Sep 18 16:44:24.998029 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:24.997923    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e42b2760-c8a8-43f2-abc6-413d29f7c34c-cabundle-cert\") pod \"e42b2760-c8a8-43f2-abc6-413d29f7c34c\" (UID: \"e42b2760-c8a8-43f2-abc6-413d29f7c34c\") "
Sep 18 16:44:24.998280 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:24.998195    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42b2760-c8a8-43f2-abc6-413d29f7c34c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e42b2760-c8a8-43f2-abc6-413d29f7c34c" (UID: "e42b2760-c8a8-43f2-abc6-413d29f7c34c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:44:24.998346 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:24.998320    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e42b2760-c8a8-43f2-abc6-413d29f7c34c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e42b2760-c8a8-43f2-abc6-413d29f7c34c" (UID: "e42b2760-c8a8-43f2-abc6-413d29f7c34c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:44:24.998400 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:24.998332    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e42b2760-c8a8-43f2-abc6-413d29f7c34c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e42b2760-c8a8-43f2-abc6-413d29f7c34c" (UID: "e42b2760-c8a8-43f2-abc6-413d29f7c34c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:44:24.999749 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:24.999722    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42b2760-c8a8-43f2-abc6-413d29f7c34c-kube-api-access-lkxfd" (OuterVolumeSpecName: "kube-api-access-lkxfd") pod "e42b2760-c8a8-43f2-abc6-413d29f7c34c" (UID: "e42b2760-c8a8-43f2-abc6-413d29f7c34c"). InnerVolumeSpecName "kube-api-access-lkxfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:44:24.999856 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:24.999810    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e42b2760-c8a8-43f2-abc6-413d29f7c34c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e42b2760-c8a8-43f2-abc6-413d29f7c34c" (UID: "e42b2760-c8a8-43f2-abc6-413d29f7c34c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:44:25.098624 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:25.098578    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkxfd\" (UniqueName: \"kubernetes.io/projected/e42b2760-c8a8-43f2-abc6-413d29f7c34c-kube-api-access-lkxfd\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:44:25.098624 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:25.098618    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e42b2760-c8a8-43f2-abc6-413d29f7c34c-kserve-provision-location\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:44:25.098624 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:25.098628    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e42b2760-c8a8-43f2-abc6-413d29f7c34c-proxy-tls\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:44:25.098624 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:25.098636    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e42b2760-c8a8-43f2-abc6-413d29f7c34c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:44:25.098887 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:25.098645    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e42b2760-c8a8-43f2-abc6-413d29f7c34c-cabundle-cert\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:44:25.262721 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:25.262638    2566 generic.go:356] "Generic (PLEG): container finished" podID="e42b2760-c8a8-43f2-abc6-413d29f7c34c" containerID="a1e8d846a18ea69ce242d309bab3ef6b9d6449a6191b90c3ed2d5950eca99064" exitCode=0
Sep 18 16:44:25.262721 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:25.262703    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"
Sep 18 16:44:25.262900 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:25.262722    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct" event={"ID":"e42b2760-c8a8-43f2-abc6-413d29f7c34c","Type":"ContainerDied","Data":"a1e8d846a18ea69ce242d309bab3ef6b9d6449a6191b90c3ed2d5950eca99064"}
Sep 18 16:44:25.262900 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:25.262805    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct" event={"ID":"e42b2760-c8a8-43f2-abc6-413d29f7c34c","Type":"ContainerDied","Data":"156b35fe0fad0a7c4575ac8de9cb880cf2aba9f701404fafae7cd171f756ca49"}
Sep 18 16:44:25.262900 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:25.262833    2566 scope.go:117] "RemoveContainer" containerID="faafe2e219fa1bafb1914198ed3e533c72864b5cafbff5f3ae8bacc2aace5a7c"
Sep 18 16:44:25.271624 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:25.271603    2566 scope.go:117] "RemoveContainer" containerID="a1e8d846a18ea69ce242d309bab3ef6b9d6449a6191b90c3ed2d5950eca99064"
Sep 18 16:44:25.278529 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:25.278514    2566 scope.go:117] "RemoveContainer" containerID="68510d47da13740cd01f23da18f52f5178d22c5c2569bac4a7e97f0306847f49"
Sep 18 16:44:25.285459 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:25.285436    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"]
Sep 18 16:44:25.285856 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:25.285842    2566 scope.go:117] "RemoveContainer" containerID="faafe2e219fa1bafb1914198ed3e533c72864b5cafbff5f3ae8bacc2aace5a7c"
Sep 18 16:44:25.286142 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:44:25.286122    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faafe2e219fa1bafb1914198ed3e533c72864b5cafbff5f3ae8bacc2aace5a7c\": container with ID starting with faafe2e219fa1bafb1914198ed3e533c72864b5cafbff5f3ae8bacc2aace5a7c not found: ID does not exist" containerID="faafe2e219fa1bafb1914198ed3e533c72864b5cafbff5f3ae8bacc2aace5a7c"
Sep 18 16:44:25.286206 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:25.286150    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faafe2e219fa1bafb1914198ed3e533c72864b5cafbff5f3ae8bacc2aace5a7c"} err="failed to get container status \"faafe2e219fa1bafb1914198ed3e533c72864b5cafbff5f3ae8bacc2aace5a7c\": rpc error: code = NotFound desc = could not find container \"faafe2e219fa1bafb1914198ed3e533c72864b5cafbff5f3ae8bacc2aace5a7c\": container with ID starting with faafe2e219fa1bafb1914198ed3e533c72864b5cafbff5f3ae8bacc2aace5a7c not found: ID does not exist"
Sep 18 16:44:25.286206 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:25.286169    2566 scope.go:117] "RemoveContainer" containerID="a1e8d846a18ea69ce242d309bab3ef6b9d6449a6191b90c3ed2d5950eca99064"
Sep 18 16:44:25.286490 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:44:25.286457    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1e8d846a18ea69ce242d309bab3ef6b9d6449a6191b90c3ed2d5950eca99064\": container with ID starting with a1e8d846a18ea69ce242d309bab3ef6b9d6449a6191b90c3ed2d5950eca99064 not found: ID does not exist" containerID="a1e8d846a18ea69ce242d309bab3ef6b9d6449a6191b90c3ed2d5950eca99064"
Sep 18 16:44:25.286490 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:25.286483    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1e8d846a18ea69ce242d309bab3ef6b9d6449a6191b90c3ed2d5950eca99064"} err="failed to get container status \"a1e8d846a18ea69ce242d309bab3ef6b9d6449a6191b90c3ed2d5950eca99064\": rpc error: code = NotFound desc = could not find container \"a1e8d846a18ea69ce242d309bab3ef6b9d6449a6191b90c3ed2d5950eca99064\": container with ID starting with a1e8d846a18ea69ce242d309bab3ef6b9d6449a6191b90c3ed2d5950eca99064 not found: ID does not exist"
Sep 18 16:44:25.286621 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:25.286498    2566 scope.go:117] "RemoveContainer" containerID="68510d47da13740cd01f23da18f52f5178d22c5c2569bac4a7e97f0306847f49"
Sep 18 16:44:25.286928 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:44:25.286832    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68510d47da13740cd01f23da18f52f5178d22c5c2569bac4a7e97f0306847f49\": container with ID starting with 68510d47da13740cd01f23da18f52f5178d22c5c2569bac4a7e97f0306847f49 not found: ID does not exist" containerID="68510d47da13740cd01f23da18f52f5178d22c5c2569bac4a7e97f0306847f49"
Sep 18 16:44:25.286928 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:25.286859    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68510d47da13740cd01f23da18f52f5178d22c5c2569bac4a7e97f0306847f49"} err="failed to get container status \"68510d47da13740cd01f23da18f52f5178d22c5c2569bac4a7e97f0306847f49\": rpc error: code = NotFound desc = could not find container \"68510d47da13740cd01f23da18f52f5178d22c5c2569bac4a7e97f0306847f49\": container with ID starting with 68510d47da13740cd01f23da18f52f5178d22c5c2569bac4a7e97f0306847f49 not found: ID does not exist"
Sep 18 16:44:25.288585 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:25.288563    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-xs4ct"]
Sep 18 16:44:25.686417 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:25.686386    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e42b2760-c8a8-43f2-abc6-413d29f7c34c" path="/var/lib/kubelet/pods/e42b2760-c8a8-43f2-abc6-413d29f7c34c/volumes"
Sep 18 16:44:42.731168 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.731130    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"]
Sep 18 16:44:42.731640 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.731438    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e42b2760-c8a8-43f2-abc6-413d29f7c34c" containerName="kube-rbac-proxy"
Sep 18 16:44:42.731640 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.731450    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e42b2760-c8a8-43f2-abc6-413d29f7c34c" containerName="kube-rbac-proxy"
Sep 18 16:44:42.731640 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.731465    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e42b2760-c8a8-43f2-abc6-413d29f7c34c" containerName="storage-initializer"
Sep 18 16:44:42.731640 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.731471    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e42b2760-c8a8-43f2-abc6-413d29f7c34c" containerName="storage-initializer"
Sep 18 16:44:42.731640 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.731478    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="storage-initializer"
Sep 18 16:44:42.731640 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.731484    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="storage-initializer"
Sep 18 16:44:42.731640 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.731491    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kube-rbac-proxy"
Sep 18 16:44:42.731640 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.731498    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kube-rbac-proxy"
Sep 18 16:44:42.731640 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.731507    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e42b2760-c8a8-43f2-abc6-413d29f7c34c" containerName="kserve-container"
Sep 18 16:44:42.731640 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.731515    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e42b2760-c8a8-43f2-abc6-413d29f7c34c" containerName="kserve-container"
Sep 18 16:44:42.731640 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.731525    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kserve-container"
Sep 18 16:44:42.731640 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.731530    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kserve-container"
Sep 18 16:44:42.731640 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.731571    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e42b2760-c8a8-43f2-abc6-413d29f7c34c" containerName="kserve-container"
Sep 18 16:44:42.731640 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.731579    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kserve-container"
Sep 18 16:44:42.731640 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.731593    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e52ac96-fa9e-4b45-8301-60f253b3d3b5" containerName="kube-rbac-proxy"
Sep 18 16:44:42.731640 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.731602    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e42b2760-c8a8-43f2-abc6-413d29f7c34c" containerName="kube-rbac-proxy"
Sep 18 16:44:42.736270 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.736253    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"
Sep 18 16:44:42.738781 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.738762    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 16:44:42.738892 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.738801    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 18 16:44:42.739698 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.739670    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 18 16:44:42.739815 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.739716    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 16:44:42.739815 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.739725    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 16:44:42.739815 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.739720    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 16:44:42.739815 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.739722    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6lh7v\""
Sep 18 16:44:42.744433 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.744410    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"]
Sep 18 16:44:42.800146 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.800109    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"]
Sep 18 16:44:42.803291 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.803273    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"
Sep 18 16:44:42.806148 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.806111    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 18 16:44:42.812040 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.812018    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"]
Sep 18 16:44:42.831347 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.831305    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b927b2ba-a937-4713-b063-a04091ea1c91-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4\" (UID: \"b927b2ba-a937-4713-b063-a04091ea1c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"
Sep 18 16:44:42.831347 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.831349    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b927b2ba-a937-4713-b063-a04091ea1c91-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4\" (UID: \"b927b2ba-a937-4713-b063-a04091ea1c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"
Sep 18 16:44:42.831581 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.831408    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcf475c8-5832-4bd2-b57e-5e04415df9ba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-ddmfm\" (UID: \"dcf475c8-5832-4bd2-b57e-5e04415df9ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"
Sep 18 16:44:42.831581 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.831456    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b927b2ba-a937-4713-b063-a04091ea1c91-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4\" (UID: \"b927b2ba-a937-4713-b063-a04091ea1c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"
Sep 18 16:44:42.831581 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.831478    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f752r\" (UniqueName: \"kubernetes.io/projected/b927b2ba-a937-4713-b063-a04091ea1c91-kube-api-access-f752r\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4\" (UID: \"b927b2ba-a937-4713-b063-a04091ea1c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"
Sep 18 16:44:42.831581 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.831510    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcf475c8-5832-4bd2-b57e-5e04415df9ba-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-ddmfm\" (UID: \"dcf475c8-5832-4bd2-b57e-5e04415df9ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"
Sep 18 16:44:42.831581 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.831552    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcf475c8-5832-4bd2-b57e-5e04415df9ba-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-ddmfm\" (UID: \"dcf475c8-5832-4bd2-b57e-5e04415df9ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"
Sep 18 16:44:42.831581 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.831579    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b927b2ba-a937-4713-b063-a04091ea1c91-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4\" (UID: \"b927b2ba-a937-4713-b063-a04091ea1c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"
Sep 18 16:44:42.831790 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.831637    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dcf475c8-5832-4bd2-b57e-5e04415df9ba-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-ddmfm\" (UID: \"dcf475c8-5832-4bd2-b57e-5e04415df9ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"
Sep 18 16:44:42.831790 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.831684    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r2l8\" (UniqueName: \"kubernetes.io/projected/dcf475c8-5832-4bd2-b57e-5e04415df9ba-kube-api-access-4r2l8\") pod \"isvc-canary-promote-predictor-6546f8697f-ddmfm\" (UID: \"dcf475c8-5832-4bd2-b57e-5e04415df9ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"
Sep 18 16:44:42.932843 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.932804    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b927b2ba-a937-4713-b063-a04091ea1c91-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4\" (UID: \"b927b2ba-a937-4713-b063-a04091ea1c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"
Sep 18 16:44:42.932843 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.932844    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b927b2ba-a937-4713-b063-a04091ea1c91-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4\" (UID: \"b927b2ba-a937-4713-b063-a04091ea1c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"
Sep 18 16:44:42.933209 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.932866    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcf475c8-5832-4bd2-b57e-5e04415df9ba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-ddmfm\" (UID: \"dcf475c8-5832-4bd2-b57e-5e04415df9ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"
Sep 18 16:44:42.933209 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.932893    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b927b2ba-a937-4713-b063-a04091ea1c91-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4\" (UID: \"b927b2ba-a937-4713-b063-a04091ea1c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"
Sep 18 16:44:42.933209 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.932911    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f752r\" (UniqueName: \"kubernetes.io/projected/b927b2ba-a937-4713-b063-a04091ea1c91-kube-api-access-f752r\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4\" (UID: \"b927b2ba-a937-4713-b063-a04091ea1c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"
Sep 18 16:44:42.933209 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.932932    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcf475c8-5832-4bd2-b57e-5e04415df9ba-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-ddmfm\" (UID: \"dcf475c8-5832-4bd2-b57e-5e04415df9ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"
Sep 18 16:44:42.933209 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.932979    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcf475c8-5832-4bd2-b57e-5e04415df9ba-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-ddmfm\" (UID: \"dcf475c8-5832-4bd2-b57e-5e04415df9ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"
Sep 18 16:44:42.933209 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.933009    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b927b2ba-a937-4713-b063-a04091ea1c91-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4\" (UID: \"b927b2ba-a937-4713-b063-a04091ea1c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"
Sep 18 16:44:42.933209 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.933046    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dcf475c8-5832-4bd2-b57e-5e04415df9ba-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-ddmfm\" (UID: \"dcf475c8-5832-4bd2-b57e-5e04415df9ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"
Sep 18 16:44:42.933209 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.933071    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r2l8\" (UniqueName: \"kubernetes.io/projected/dcf475c8-5832-4bd2-b57e-5e04415df9ba-kube-api-access-4r2l8\") pod \"isvc-canary-promote-predictor-6546f8697f-ddmfm\" (UID: \"dcf475c8-5832-4bd2-b57e-5e04415df9ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"
Sep 18 16:44:42.933627 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.933340    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b927b2ba-a937-4713-b063-a04091ea1c91-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4\" (UID: \"b927b2ba-a937-4713-b063-a04091ea1c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"
Sep 18 16:44:42.933686 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.933632    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcf475c8-5832-4bd2-b57e-5e04415df9ba-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-ddmfm\" (UID: \"dcf475c8-5832-4bd2-b57e-5e04415df9ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"
Sep 18 16:44:42.933775 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.933753    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcf475c8-5832-4bd2-b57e-5e04415df9ba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-ddmfm\" (UID: \"dcf475c8-5832-4bd2-b57e-5e04415df9ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"
Sep 18 16:44:42.933828 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.933781    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b927b2ba-a937-4713-b063-a04091ea1c91-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4\" (UID: \"b927b2ba-a937-4713-b063-a04091ea1c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"
Sep 18 16:44:42.933865 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.933829    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b927b2ba-a937-4713-b063-a04091ea1c91-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4\" (UID: \"b927b2ba-a937-4713-b063-a04091ea1c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"
Sep 18 16:44:42.933904 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.933883    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dcf475c8-5832-4bd2-b57e-5e04415df9ba-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-ddmfm\" (UID: \"dcf475c8-5832-4bd2-b57e-5e04415df9ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"
Sep 18 16:44:42.935944 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.935921    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcf475c8-5832-4bd2-b57e-5e04415df9ba-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-ddmfm\" (UID: \"dcf475c8-5832-4bd2-b57e-5e04415df9ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"
Sep 18 16:44:42.936046 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.936024    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b927b2ba-a937-4713-b063-a04091ea1c91-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4\" (UID: \"b927b2ba-a937-4713-b063-a04091ea1c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"
Sep 18 16:44:42.941030 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.941007    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r2l8\" (UniqueName: \"kubernetes.io/projected/dcf475c8-5832-4bd2-b57e-5e04415df9ba-kube-api-access-4r2l8\") pod \"isvc-canary-promote-predictor-6546f8697f-ddmfm\" (UID: \"dcf475c8-5832-4bd2-b57e-5e04415df9ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"
Sep 18 16:44:42.941314 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:42.941293    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f752r\" (UniqueName: \"kubernetes.io/projected/b927b2ba-a937-4713-b063-a04091ea1c91-kube-api-access-f752r\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4\" (UID: \"b927b2ba-a937-4713-b063-a04091ea1c91\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"
Sep 18 16:44:43.047451 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:43.047359    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"
Sep 18 16:44:43.113615 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:43.113590    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"
Sep 18 16:44:43.320525 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:43.320472    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"]
Sep 18 16:44:43.322655 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:44:43.322627    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb927b2ba_a937_4713_b063_a04091ea1c91.slice/crio-ff886b3d47d9a02a4538fbc9711bd3b9b7f0f2e696653a8c63f9e7972814bb43 WatchSource:0}: Error finding container ff886b3d47d9a02a4538fbc9711bd3b9b7f0f2e696653a8c63f9e7972814bb43: Status 404 returned error can't find the container with id ff886b3d47d9a02a4538fbc9711bd3b9b7f0f2e696653a8c63f9e7972814bb43
Sep 18 16:44:43.333125 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:43.333100    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4" event={"ID":"b927b2ba-a937-4713-b063-a04091ea1c91","Type":"ContainerStarted","Data":"ff886b3d47d9a02a4538fbc9711bd3b9b7f0f2e696653a8c63f9e7972814bb43"}
Sep 18 16:44:43.386667 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:43.386639    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"]
Sep 18 16:44:43.389677 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:44:43.389651    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf475c8_5832_4bd2_b57e_5e04415df9ba.slice/crio-8505e5f9e789692f5648809e77182488911b6c01d226498414eb90db4c6874ff WatchSource:0}: Error finding container 8505e5f9e789692f5648809e77182488911b6c01d226498414eb90db4c6874ff: Status 404 returned error can't find the container with id 8505e5f9e789692f5648809e77182488911b6c01d226498414eb90db4c6874ff
Sep 18 16:44:44.338418 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:44.338379    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4" event={"ID":"b927b2ba-a937-4713-b063-a04091ea1c91","Type":"ContainerStarted","Data":"b343fac2638f0ccc7ee7e606ac5930d66408b223b70643455f9ee9b6acf3a9e6"}
Sep 18 16:44:44.339939 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:44.339903    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm" event={"ID":"dcf475c8-5832-4bd2-b57e-5e04415df9ba","Type":"ContainerStarted","Data":"e30e3d47f011cf3cf69035bbfadbfd3ed812a3ed9cb631b36e594514438e22f7"}
Sep 18 16:44:44.340044 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:44.339964    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm" event={"ID":"dcf475c8-5832-4bd2-b57e-5e04415df9ba","Type":"ContainerStarted","Data":"8505e5f9e789692f5648809e77182488911b6c01d226498414eb90db4c6874ff"}
Sep 18 16:44:47.354035 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:47.354004    2566 generic.go:356] "Generic (PLEG): container finished" podID="b927b2ba-a937-4713-b063-a04091ea1c91" containerID="b343fac2638f0ccc7ee7e606ac5930d66408b223b70643455f9ee9b6acf3a9e6" exitCode=0
Sep 18 16:44:47.354423 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:47.354096    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4" event={"ID":"b927b2ba-a937-4713-b063-a04091ea1c91","Type":"ContainerDied","Data":"b343fac2638f0ccc7ee7e606ac5930d66408b223b70643455f9ee9b6acf3a9e6"}
Sep 18 16:44:47.355854 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:47.355834    2566 generic.go:356] "Generic (PLEG): container finished" podID="dcf475c8-5832-4bd2-b57e-5e04415df9ba" containerID="e30e3d47f011cf3cf69035bbfadbfd3ed812a3ed9cb631b36e594514438e22f7" exitCode=0
Sep 18 16:44:47.355951 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:47.355903    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm" event={"ID":"dcf475c8-5832-4bd2-b57e-5e04415df9ba","Type":"ContainerDied","Data":"e30e3d47f011cf3cf69035bbfadbfd3ed812a3ed9cb631b36e594514438e22f7"}
Sep 18 16:44:48.361503 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:48.361468    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4" event={"ID":"b927b2ba-a937-4713-b063-a04091ea1c91","Type":"ContainerStarted","Data":"cf70fa8f112405b3cec795ce57f1cc50d26f74d7e3349244f6a374734400ed88"}
Sep 18 16:44:48.361922 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:48.361572    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4" event={"ID":"b927b2ba-a937-4713-b063-a04091ea1c91","Type":"ContainerStarted","Data":"57fdea05e1fa92db9918331459454638c57b1363a8d46256e68d3ef0e9cf2f20"}
Sep 18 16:44:48.361922 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:48.361787    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"
Sep 18 16:44:48.363552 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:48.363528    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm" event={"ID":"dcf475c8-5832-4bd2-b57e-5e04415df9ba","Type":"ContainerStarted","Data":"b6f5cf043a3f8a71bbeb50e848caf91d676c93304a0d11b87c3e73197410d641"}
Sep 18 16:44:48.363681 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:48.363595    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm" event={"ID":"dcf475c8-5832-4bd2-b57e-5e04415df9ba","Type":"ContainerStarted","Data":"2d795be73168cb516ec23774edde14e00a892ff35be57cc5b9bf52eb69318c55"}
Sep 18 16:44:48.363819 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:48.363802    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"
Sep 18 16:44:48.383004 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:48.382956    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4" podStartSLOduration=2.477254091 podStartE2EDuration="6.38293926s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.905685169s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:44:48.381377724 +0000 UTC m=+1726.836830801" watchObservedRunningTime="2026-09-18 16:44:48.38293926 +0000 UTC m=+1726.838392340"
Sep 18 16:44:49.367867 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:49.367829    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"
Sep 18 16:44:49.368237 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:49.367892    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"
Sep 18 16:44:49.368904 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:49.368881    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm" podUID="dcf475c8-5832-4bd2-b57e-5e04415df9ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 16:44:49.368974 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:49.368877    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4" podUID="b927b2ba-a937-4713-b063-a04091ea1c91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 16:44:50.372248 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:50.372205    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4" podUID="b927b2ba-a937-4713-b063-a04091ea1c91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 16:44:50.372618 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:50.372213    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm" podUID="dcf475c8-5832-4bd2-b57e-5e04415df9ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 16:44:55.377142 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:55.377109    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"
Sep 18 16:44:55.377605 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:55.377420    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"
Sep 18 16:44:55.377672 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:55.377645    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm" podUID="dcf475c8-5832-4bd2-b57e-5e04415df9ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 16:44:55.378218 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:55.378195    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4" podUID="b927b2ba-a937-4713-b063-a04091ea1c91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 16:44:55.400394 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:44:55.400341    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm" podStartSLOduration=9.506487983 podStartE2EDuration="13.400326471s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.893838488s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:44:48.41651312 +0000 UTC m=+1726.871966197" watchObservedRunningTime="2026-09-18 16:44:55.400326471 +0000 UTC m=+1733.855779548"
Sep 18 16:45:05.378299 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:45:05.378252    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm" podUID="dcf475c8-5832-4bd2-b57e-5e04415df9ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 16:45:05.378772 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:45:05.378250    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4" podUID="b927b2ba-a937-4713-b063-a04091ea1c91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 16:45:15.378143 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:45:15.378102    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm" podUID="dcf475c8-5832-4bd2-b57e-5e04415df9ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 16:45:15.378549 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:45:15.378099    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4" podUID="b927b2ba-a937-4713-b063-a04091ea1c91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 16:45:25.378502 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:45:25.378461    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm" podUID="dcf475c8-5832-4bd2-b57e-5e04415df9ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 16:45:25.378947 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:45:25.378469    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4" podUID="b927b2ba-a937-4713-b063-a04091ea1c91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 16:45:35.378252 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:45:35.378210    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4" podUID="b927b2ba-a937-4713-b063-a04091ea1c91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 16:45:35.378726 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:45:35.378213    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm" podUID="dcf475c8-5832-4bd2-b57e-5e04415df9ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 16:45:45.378210 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:45:45.378123    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4" podUID="b927b2ba-a937-4713-b063-a04091ea1c91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 16:45:45.378210 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:45:45.378143    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm" podUID="dcf475c8-5832-4bd2-b57e-5e04415df9ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 16:45:55.377700 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:45:55.377662    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm" podUID="dcf475c8-5832-4bd2-b57e-5e04415df9ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 16:45:55.378147 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:45:55.378053    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4" podUID="b927b2ba-a937-4713-b063-a04091ea1c91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 16:46:03.703593 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:03.703563    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-vfmdv_1e3ca1ec-6856-4afb-850e-85e3fa968b3f/console-operator/1.log"
Sep 18 16:46:03.712159 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:03.712137    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-vfmdv_1e3ca1ec-6856-4afb-850e-85e3fa968b3f/console-operator/1.log"
Sep 18 16:46:05.379276 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:05.379250    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"
Sep 18 16:46:05.379656 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:05.379333    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"
Sep 18 16:46:12.843266 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:12.843233    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"]
Sep 18 16:46:12.843702 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:12.843451    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm" podUID="dcf475c8-5832-4bd2-b57e-5e04415df9ba" containerName="kserve-container" containerID="cri-o://2d795be73168cb516ec23774edde14e00a892ff35be57cc5b9bf52eb69318c55" gracePeriod=30
Sep 18 16:46:12.843702 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:12.843507    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm" podUID="dcf475c8-5832-4bd2-b57e-5e04415df9ba" containerName="kube-rbac-proxy" containerID="cri-o://b6f5cf043a3f8a71bbeb50e848caf91d676c93304a0d11b87c3e73197410d641" gracePeriod=30
Sep 18 16:46:13.721547 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:13.721508    2566 generic.go:356] "Generic (PLEG): container finished" podID="dcf475c8-5832-4bd2-b57e-5e04415df9ba" containerID="b6f5cf043a3f8a71bbeb50e848caf91d676c93304a0d11b87c3e73197410d641" exitCode=2
Sep 18 16:46:13.721733 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:13.721579    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm" event={"ID":"dcf475c8-5832-4bd2-b57e-5e04415df9ba","Type":"ContainerDied","Data":"b6f5cf043a3f8a71bbeb50e848caf91d676c93304a0d11b87c3e73197410d641"}
Sep 18 16:46:15.372814 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:15.372776    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm" podUID="dcf475c8-5832-4bd2-b57e-5e04415df9ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 18 16:46:15.378206 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:15.378168    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm" podUID="dcf475c8-5832-4bd2-b57e-5e04415df9ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 16:46:17.486419 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.486397    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"
Sep 18 16:46:17.549548 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.549517    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcf475c8-5832-4bd2-b57e-5e04415df9ba-kube-rbac-proxy-sar-config\") pod \"dcf475c8-5832-4bd2-b57e-5e04415df9ba\" (UID: \"dcf475c8-5832-4bd2-b57e-5e04415df9ba\") "
Sep 18 16:46:17.549548 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.549552    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcf475c8-5832-4bd2-b57e-5e04415df9ba-proxy-tls\") pod \"dcf475c8-5832-4bd2-b57e-5e04415df9ba\" (UID: \"dcf475c8-5832-4bd2-b57e-5e04415df9ba\") "
Sep 18 16:46:17.549783 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.549588    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4r2l8\" (UniqueName: \"kubernetes.io/projected/dcf475c8-5832-4bd2-b57e-5e04415df9ba-kube-api-access-4r2l8\") pod \"dcf475c8-5832-4bd2-b57e-5e04415df9ba\" (UID: \"dcf475c8-5832-4bd2-b57e-5e04415df9ba\") "
Sep 18 16:46:17.549783 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.549618    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcf475c8-5832-4bd2-b57e-5e04415df9ba-kserve-provision-location\") pod \"dcf475c8-5832-4bd2-b57e-5e04415df9ba\" (UID: \"dcf475c8-5832-4bd2-b57e-5e04415df9ba\") "
Sep 18 16:46:17.549783 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.549688    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dcf475c8-5832-4bd2-b57e-5e04415df9ba-cabundle-cert\") pod \"dcf475c8-5832-4bd2-b57e-5e04415df9ba\" (UID: \"dcf475c8-5832-4bd2-b57e-5e04415df9ba\") "
Sep 18 16:46:17.549953 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.549935    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf475c8-5832-4bd2-b57e-5e04415df9ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dcf475c8-5832-4bd2-b57e-5e04415df9ba" (UID: "dcf475c8-5832-4bd2-b57e-5e04415df9ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:46:17.550024 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.549940    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf475c8-5832-4bd2-b57e-5e04415df9ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dcf475c8-5832-4bd2-b57e-5e04415df9ba" (UID: "dcf475c8-5832-4bd2-b57e-5e04415df9ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:46:17.550139 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.550122    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf475c8-5832-4bd2-b57e-5e04415df9ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dcf475c8-5832-4bd2-b57e-5e04415df9ba" (UID: "dcf475c8-5832-4bd2-b57e-5e04415df9ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:46:17.551697 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.551671    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf475c8-5832-4bd2-b57e-5e04415df9ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dcf475c8-5832-4bd2-b57e-5e04415df9ba" (UID: "dcf475c8-5832-4bd2-b57e-5e04415df9ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:46:17.551796 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.551753    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf475c8-5832-4bd2-b57e-5e04415df9ba-kube-api-access-4r2l8" (OuterVolumeSpecName: "kube-api-access-4r2l8") pod "dcf475c8-5832-4bd2-b57e-5e04415df9ba" (UID: "dcf475c8-5832-4bd2-b57e-5e04415df9ba"). InnerVolumeSpecName "kube-api-access-4r2l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:46:17.650839 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.650760    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dcf475c8-5832-4bd2-b57e-5e04415df9ba-cabundle-cert\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:46:17.650839 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.650787    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcf475c8-5832-4bd2-b57e-5e04415df9ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:46:17.650839 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.650798    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcf475c8-5832-4bd2-b57e-5e04415df9ba-proxy-tls\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:46:17.650839 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.650808    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r2l8\" (UniqueName: \"kubernetes.io/projected/dcf475c8-5832-4bd2-b57e-5e04415df9ba-kube-api-access-4r2l8\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:46:17.650839 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.650817    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcf475c8-5832-4bd2-b57e-5e04415df9ba-kserve-provision-location\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:46:17.738237 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.738205    2566 generic.go:356] "Generic (PLEG): container finished" podID="dcf475c8-5832-4bd2-b57e-5e04415df9ba" containerID="2d795be73168cb516ec23774edde14e00a892ff35be57cc5b9bf52eb69318c55" exitCode=0
Sep 18 16:46:17.738388 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.738259    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm" event={"ID":"dcf475c8-5832-4bd2-b57e-5e04415df9ba","Type":"ContainerDied","Data":"2d795be73168cb516ec23774edde14e00a892ff35be57cc5b9bf52eb69318c55"}
Sep 18 16:46:17.738388 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.738273    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"
Sep 18 16:46:17.738388 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.738307    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm" event={"ID":"dcf475c8-5832-4bd2-b57e-5e04415df9ba","Type":"ContainerDied","Data":"8505e5f9e789692f5648809e77182488911b6c01d226498414eb90db4c6874ff"}
Sep 18 16:46:17.738388 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.738332    2566 scope.go:117] "RemoveContainer" containerID="b6f5cf043a3f8a71bbeb50e848caf91d676c93304a0d11b87c3e73197410d641"
Sep 18 16:46:17.746880 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.746862    2566 scope.go:117] "RemoveContainer" containerID="2d795be73168cb516ec23774edde14e00a892ff35be57cc5b9bf52eb69318c55"
Sep 18 16:46:17.756282 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.756265    2566 scope.go:117] "RemoveContainer" containerID="e30e3d47f011cf3cf69035bbfadbfd3ed812a3ed9cb631b36e594514438e22f7"
Sep 18 16:46:17.756984 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.756961    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"]
Sep 18 16:46:17.762010 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.761986    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-ddmfm"]
Sep 18 16:46:17.763863 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.763843    2566 scope.go:117] "RemoveContainer" containerID="b6f5cf043a3f8a71bbeb50e848caf91d676c93304a0d11b87c3e73197410d641"
Sep 18 16:46:17.764144 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:46:17.764125    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6f5cf043a3f8a71bbeb50e848caf91d676c93304a0d11b87c3e73197410d641\": container with ID starting with b6f5cf043a3f8a71bbeb50e848caf91d676c93304a0d11b87c3e73197410d641 not found: ID does not exist" containerID="b6f5cf043a3f8a71bbeb50e848caf91d676c93304a0d11b87c3e73197410d641"
Sep 18 16:46:17.764204 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.764152    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6f5cf043a3f8a71bbeb50e848caf91d676c93304a0d11b87c3e73197410d641"} err="failed to get container status \"b6f5cf043a3f8a71bbeb50e848caf91d676c93304a0d11b87c3e73197410d641\": rpc error: code = NotFound desc = could not find container \"b6f5cf043a3f8a71bbeb50e848caf91d676c93304a0d11b87c3e73197410d641\": container with ID starting with b6f5cf043a3f8a71bbeb50e848caf91d676c93304a0d11b87c3e73197410d641 not found: ID does not exist"
Sep 18 16:46:17.764204 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.764171    2566 scope.go:117] "RemoveContainer" containerID="2d795be73168cb516ec23774edde14e00a892ff35be57cc5b9bf52eb69318c55"
Sep 18 16:46:17.764426 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:46:17.764401    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d795be73168cb516ec23774edde14e00a892ff35be57cc5b9bf52eb69318c55\": container with ID starting with 2d795be73168cb516ec23774edde14e00a892ff35be57cc5b9bf52eb69318c55 not found: ID does not exist" containerID="2d795be73168cb516ec23774edde14e00a892ff35be57cc5b9bf52eb69318c55"
Sep 18 16:46:17.764471 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.764436    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d795be73168cb516ec23774edde14e00a892ff35be57cc5b9bf52eb69318c55"} err="failed to get container status \"2d795be73168cb516ec23774edde14e00a892ff35be57cc5b9bf52eb69318c55\": rpc error: code = NotFound desc = could not find container \"2d795be73168cb516ec23774edde14e00a892ff35be57cc5b9bf52eb69318c55\": container with ID starting with 2d795be73168cb516ec23774edde14e00a892ff35be57cc5b9bf52eb69318c55 not found: ID does not exist"
Sep 18 16:46:17.764471 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.764452    2566 scope.go:117] "RemoveContainer" containerID="e30e3d47f011cf3cf69035bbfadbfd3ed812a3ed9cb631b36e594514438e22f7"
Sep 18 16:46:17.764690 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:46:17.764671    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e30e3d47f011cf3cf69035bbfadbfd3ed812a3ed9cb631b36e594514438e22f7\": container with ID starting with e30e3d47f011cf3cf69035bbfadbfd3ed812a3ed9cb631b36e594514438e22f7 not found: ID does not exist" containerID="e30e3d47f011cf3cf69035bbfadbfd3ed812a3ed9cb631b36e594514438e22f7"
Sep 18 16:46:17.764741 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:17.764700    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e30e3d47f011cf3cf69035bbfadbfd3ed812a3ed9cb631b36e594514438e22f7"} err="failed to get container status \"e30e3d47f011cf3cf69035bbfadbfd3ed812a3ed9cb631b36e594514438e22f7\": rpc error: code = NotFound desc = could not find container \"e30e3d47f011cf3cf69035bbfadbfd3ed812a3ed9cb631b36e594514438e22f7\": container with ID starting with e30e3d47f011cf3cf69035bbfadbfd3ed812a3ed9cb631b36e594514438e22f7 not found: ID does not exist"
Sep 18 16:46:19.686338 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:46:19.686304    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcf475c8-5832-4bd2-b57e-5e04415df9ba" path="/var/lib/kubelet/pods/dcf475c8-5832-4bd2-b57e-5e04415df9ba/volumes"
Sep 18 16:47:43.056377 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:43.056345    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"]
Sep 18 16:47:43.056889 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:43.056601    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4" podUID="b927b2ba-a937-4713-b063-a04091ea1c91" containerName="kserve-container" containerID="cri-o://57fdea05e1fa92db9918331459454638c57b1363a8d46256e68d3ef0e9cf2f20" gracePeriod=30
Sep 18 16:47:43.056889 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:43.056619    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4" podUID="b927b2ba-a937-4713-b063-a04091ea1c91" containerName="kube-rbac-proxy" containerID="cri-o://cf70fa8f112405b3cec795ce57f1cc50d26f74d7e3349244f6a374734400ed88" gracePeriod=30
Sep 18 16:47:44.072836 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:44.072802    2566 generic.go:356] "Generic (PLEG): container finished" podID="b927b2ba-a937-4713-b063-a04091ea1c91" containerID="cf70fa8f112405b3cec795ce57f1cc50d26f74d7e3349244f6a374734400ed88" exitCode=2
Sep 18 16:47:44.073224 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:44.072874    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4" event={"ID":"b927b2ba-a937-4713-b063-a04091ea1c91","Type":"ContainerDied","Data":"cf70fa8f112405b3cec795ce57f1cc50d26f74d7e3349244f6a374734400ed88"}
Sep 18 16:47:45.372645 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:45.372596    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4" podUID="b927b2ba-a937-4713-b063-a04091ea1c91" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 18 16:47:45.378448 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:45.378417    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4" podUID="b927b2ba-a937-4713-b063-a04091ea1c91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 16:47:48.001137 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.001111    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"
Sep 18 16:47:48.089462 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.089373    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b927b2ba-a937-4713-b063-a04091ea1c91-proxy-tls\") pod \"b927b2ba-a937-4713-b063-a04091ea1c91\" (UID: \"b927b2ba-a937-4713-b063-a04091ea1c91\") "
Sep 18 16:47:48.089462 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.089426    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b927b2ba-a937-4713-b063-a04091ea1c91-kube-rbac-proxy-sar-config\") pod \"b927b2ba-a937-4713-b063-a04091ea1c91\" (UID: \"b927b2ba-a937-4713-b063-a04091ea1c91\") "
Sep 18 16:47:48.089713 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.089472    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b927b2ba-a937-4713-b063-a04091ea1c91-kserve-provision-location\") pod \"b927b2ba-a937-4713-b063-a04091ea1c91\" (UID: \"b927b2ba-a937-4713-b063-a04091ea1c91\") "
Sep 18 16:47:48.089713 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.089513    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b927b2ba-a937-4713-b063-a04091ea1c91-cabundle-cert\") pod \"b927b2ba-a937-4713-b063-a04091ea1c91\" (UID: \"b927b2ba-a937-4713-b063-a04091ea1c91\") "
Sep 18 16:47:48.089713 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.089693    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f752r\" (UniqueName: \"kubernetes.io/projected/b927b2ba-a937-4713-b063-a04091ea1c91-kube-api-access-f752r\") pod \"b927b2ba-a937-4713-b063-a04091ea1c91\" (UID: \"b927b2ba-a937-4713-b063-a04091ea1c91\") "
Sep 18 16:47:48.089876 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.089795    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b927b2ba-a937-4713-b063-a04091ea1c91-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b927b2ba-a937-4713-b063-a04091ea1c91" (UID: "b927b2ba-a937-4713-b063-a04091ea1c91"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:47:48.089876 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.089832    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b927b2ba-a937-4713-b063-a04091ea1c91-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b927b2ba-a937-4713-b063-a04091ea1c91" (UID: "b927b2ba-a937-4713-b063-a04091ea1c91"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:47:48.089991 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.089869    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b927b2ba-a937-4713-b063-a04091ea1c91-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b927b2ba-a937-4713-b063-a04091ea1c91" (UID: "b927b2ba-a937-4713-b063-a04091ea1c91"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:47:48.090060 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.090041    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b927b2ba-a937-4713-b063-a04091ea1c91-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:47:48.090141 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.090068    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b927b2ba-a937-4713-b063-a04091ea1c91-kserve-provision-location\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:47:48.090141 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.090100    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b927b2ba-a937-4713-b063-a04091ea1c91-cabundle-cert\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:47:48.090799 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.090771    2566 generic.go:356] "Generic (PLEG): container finished" podID="b927b2ba-a937-4713-b063-a04091ea1c91" containerID="57fdea05e1fa92db9918331459454638c57b1363a8d46256e68d3ef0e9cf2f20" exitCode=0
Sep 18 16:47:48.090932 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.090806    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4" event={"ID":"b927b2ba-a937-4713-b063-a04091ea1c91","Type":"ContainerDied","Data":"57fdea05e1fa92db9918331459454638c57b1363a8d46256e68d3ef0e9cf2f20"}
Sep 18 16:47:48.090932 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.090847    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"
Sep 18 16:47:48.090932 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.090888    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4" event={"ID":"b927b2ba-a937-4713-b063-a04091ea1c91","Type":"ContainerDied","Data":"ff886b3d47d9a02a4538fbc9711bd3b9b7f0f2e696653a8c63f9e7972814bb43"}
Sep 18 16:47:48.090932 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.090917    2566 scope.go:117] "RemoveContainer" containerID="cf70fa8f112405b3cec795ce57f1cc50d26f74d7e3349244f6a374734400ed88"
Sep 18 16:47:48.091745 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.091720    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b927b2ba-a937-4713-b063-a04091ea1c91-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b927b2ba-a937-4713-b063-a04091ea1c91" (UID: "b927b2ba-a937-4713-b063-a04091ea1c91"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:47:48.091993 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.091968    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b927b2ba-a937-4713-b063-a04091ea1c91-kube-api-access-f752r" (OuterVolumeSpecName: "kube-api-access-f752r") pod "b927b2ba-a937-4713-b063-a04091ea1c91" (UID: "b927b2ba-a937-4713-b063-a04091ea1c91"). InnerVolumeSpecName "kube-api-access-f752r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:47:48.105097 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.105067    2566 scope.go:117] "RemoveContainer" containerID="57fdea05e1fa92db9918331459454638c57b1363a8d46256e68d3ef0e9cf2f20"
Sep 18 16:47:48.111944 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.111927    2566 scope.go:117] "RemoveContainer" containerID="b343fac2638f0ccc7ee7e606ac5930d66408b223b70643455f9ee9b6acf3a9e6"
Sep 18 16:47:48.118662 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.118642    2566 scope.go:117] "RemoveContainer" containerID="cf70fa8f112405b3cec795ce57f1cc50d26f74d7e3349244f6a374734400ed88"
Sep 18 16:47:48.118927 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:47:48.118908    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf70fa8f112405b3cec795ce57f1cc50d26f74d7e3349244f6a374734400ed88\": container with ID starting with cf70fa8f112405b3cec795ce57f1cc50d26f74d7e3349244f6a374734400ed88 not found: ID does not exist" containerID="cf70fa8f112405b3cec795ce57f1cc50d26f74d7e3349244f6a374734400ed88"
Sep 18 16:47:48.118977 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.118936    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf70fa8f112405b3cec795ce57f1cc50d26f74d7e3349244f6a374734400ed88"} err="failed to get container status \"cf70fa8f112405b3cec795ce57f1cc50d26f74d7e3349244f6a374734400ed88\": rpc error: code = NotFound desc = could not find container \"cf70fa8f112405b3cec795ce57f1cc50d26f74d7e3349244f6a374734400ed88\": container with ID starting with cf70fa8f112405b3cec795ce57f1cc50d26f74d7e3349244f6a374734400ed88 not found: ID does not exist"
Sep 18 16:47:48.118977 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.118951    2566 scope.go:117] "RemoveContainer" containerID="57fdea05e1fa92db9918331459454638c57b1363a8d46256e68d3ef0e9cf2f20"
Sep 18 16:47:48.119256 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:47:48.119235    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57fdea05e1fa92db9918331459454638c57b1363a8d46256e68d3ef0e9cf2f20\": container with ID starting with 57fdea05e1fa92db9918331459454638c57b1363a8d46256e68d3ef0e9cf2f20 not found: ID does not exist" containerID="57fdea05e1fa92db9918331459454638c57b1363a8d46256e68d3ef0e9cf2f20"
Sep 18 16:47:48.119335 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.119263    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57fdea05e1fa92db9918331459454638c57b1363a8d46256e68d3ef0e9cf2f20"} err="failed to get container status \"57fdea05e1fa92db9918331459454638c57b1363a8d46256e68d3ef0e9cf2f20\": rpc error: code = NotFound desc = could not find container \"57fdea05e1fa92db9918331459454638c57b1363a8d46256e68d3ef0e9cf2f20\": container with ID starting with 57fdea05e1fa92db9918331459454638c57b1363a8d46256e68d3ef0e9cf2f20 not found: ID does not exist"
Sep 18 16:47:48.119335 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.119284    2566 scope.go:117] "RemoveContainer" containerID="b343fac2638f0ccc7ee7e606ac5930d66408b223b70643455f9ee9b6acf3a9e6"
Sep 18 16:47:48.119570 ip-10-0-138-142 kubenswrapper[2566]: E0918 16:47:48.119551    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b343fac2638f0ccc7ee7e606ac5930d66408b223b70643455f9ee9b6acf3a9e6\": container with ID starting with b343fac2638f0ccc7ee7e606ac5930d66408b223b70643455f9ee9b6acf3a9e6 not found: ID does not exist" containerID="b343fac2638f0ccc7ee7e606ac5930d66408b223b70643455f9ee9b6acf3a9e6"
Sep 18 16:47:48.119613 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.119574    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b343fac2638f0ccc7ee7e606ac5930d66408b223b70643455f9ee9b6acf3a9e6"} err="failed to get container status \"b343fac2638f0ccc7ee7e606ac5930d66408b223b70643455f9ee9b6acf3a9e6\": rpc error: code = NotFound desc = could not find container \"b343fac2638f0ccc7ee7e606ac5930d66408b223b70643455f9ee9b6acf3a9e6\": container with ID starting with b343fac2638f0ccc7ee7e606ac5930d66408b223b70643455f9ee9b6acf3a9e6 not found: ID does not exist"
Sep 18 16:47:48.190587 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.190552    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b927b2ba-a937-4713-b063-a04091ea1c91-proxy-tls\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:47:48.190587 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.190581    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f752r\" (UniqueName: \"kubernetes.io/projected/b927b2ba-a937-4713-b063-a04091ea1c91-kube-api-access-f752r\") on node \"ip-10-0-138-142.ec2.internal\" DevicePath \"\""
Sep 18 16:47:48.417425 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.417394    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"]
Sep 18 16:47:48.421895 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:48.421871    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-ksdk4"]
Sep 18 16:47:49.686293 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:47:49.686258    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b927b2ba-a937-4713-b063-a04091ea1c91" path="/var/lib/kubelet/pods/b927b2ba-a937-4713-b063-a04091ea1c91/volumes"
Sep 18 16:50:16.142738 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.142658    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g45m6/must-gather-sqsxg"]
Sep 18 16:50:16.143227 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.142973    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b927b2ba-a937-4713-b063-a04091ea1c91" containerName="kserve-container"
Sep 18 16:50:16.143227 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.142983    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b927b2ba-a937-4713-b063-a04091ea1c91" containerName="kserve-container"
Sep 18 16:50:16.143227 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.142992    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b927b2ba-a937-4713-b063-a04091ea1c91" containerName="storage-initializer"
Sep 18 16:50:16.143227 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.142998    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b927b2ba-a937-4713-b063-a04091ea1c91" containerName="storage-initializer"
Sep 18 16:50:16.143227 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.143006    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b927b2ba-a937-4713-b063-a04091ea1c91" containerName="kube-rbac-proxy"
Sep 18 16:50:16.143227 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.143012    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b927b2ba-a937-4713-b063-a04091ea1c91" containerName="kube-rbac-proxy"
Sep 18 16:50:16.143227 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.143018    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf475c8-5832-4bd2-b57e-5e04415df9ba" containerName="kserve-container"
Sep 18 16:50:16.143227 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.143024    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf475c8-5832-4bd2-b57e-5e04415df9ba" containerName="kserve-container"
Sep 18 16:50:16.143227 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.143035    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf475c8-5832-4bd2-b57e-5e04415df9ba" containerName="storage-initializer"
Sep 18 16:50:16.143227 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.143040    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf475c8-5832-4bd2-b57e-5e04415df9ba" containerName="storage-initializer"
Sep 18 16:50:16.143227 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.143047    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf475c8-5832-4bd2-b57e-5e04415df9ba" containerName="kube-rbac-proxy"
Sep 18 16:50:16.143227 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.143053    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf475c8-5832-4bd2-b57e-5e04415df9ba" containerName="kube-rbac-proxy"
Sep 18 16:50:16.143227 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.143123    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf475c8-5832-4bd2-b57e-5e04415df9ba" containerName="kserve-container"
Sep 18 16:50:16.143227 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.143133    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b927b2ba-a937-4713-b063-a04091ea1c91" containerName="kube-rbac-proxy"
Sep 18 16:50:16.143227 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.143141    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b927b2ba-a937-4713-b063-a04091ea1c91" containerName="kserve-container"
Sep 18 16:50:16.143227 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.143146    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf475c8-5832-4bd2-b57e-5e04415df9ba" containerName="kube-rbac-proxy"
Sep 18 16:50:16.145994 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.145979    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g45m6/must-gather-sqsxg"
Sep 18 16:50:16.152056 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.151971    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g45m6\"/\"default-dockercfg-wgwnv\""
Sep 18 16:50:16.152176 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.152063    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g45m6\"/\"openshift-service-ca.crt\""
Sep 18 16:50:16.152317 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.152304    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g45m6\"/\"kube-root-ca.crt\""
Sep 18 16:50:16.158358 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.158339    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g45m6/must-gather-sqsxg"]
Sep 18 16:50:16.205019 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.204991    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pmrk\" (UniqueName: \"kubernetes.io/projected/48cc1d04-d03b-43fd-ab85-afe13c1650c7-kube-api-access-7pmrk\") pod \"must-gather-sqsxg\" (UID: \"48cc1d04-d03b-43fd-ab85-afe13c1650c7\") " pod="openshift-must-gather-g45m6/must-gather-sqsxg"
Sep 18 16:50:16.205171 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.205093    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/48cc1d04-d03b-43fd-ab85-afe13c1650c7-must-gather-output\") pod \"must-gather-sqsxg\" (UID: \"48cc1d04-d03b-43fd-ab85-afe13c1650c7\") " pod="openshift-must-gather-g45m6/must-gather-sqsxg"
Sep 18 16:50:16.305833 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.305803    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/48cc1d04-d03b-43fd-ab85-afe13c1650c7-must-gather-output\") pod \"must-gather-sqsxg\" (UID: \"48cc1d04-d03b-43fd-ab85-afe13c1650c7\") " pod="openshift-must-gather-g45m6/must-gather-sqsxg"
Sep 18 16:50:16.305969 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.305849    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pmrk\" (UniqueName: \"kubernetes.io/projected/48cc1d04-d03b-43fd-ab85-afe13c1650c7-kube-api-access-7pmrk\") pod \"must-gather-sqsxg\" (UID: \"48cc1d04-d03b-43fd-ab85-afe13c1650c7\") " pod="openshift-must-gather-g45m6/must-gather-sqsxg"
Sep 18 16:50:16.306147 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.306128    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/48cc1d04-d03b-43fd-ab85-afe13c1650c7-must-gather-output\") pod \"must-gather-sqsxg\" (UID: \"48cc1d04-d03b-43fd-ab85-afe13c1650c7\") " pod="openshift-must-gather-g45m6/must-gather-sqsxg"
Sep 18 16:50:16.314017 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.313990    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pmrk\" (UniqueName: \"kubernetes.io/projected/48cc1d04-d03b-43fd-ab85-afe13c1650c7-kube-api-access-7pmrk\") pod \"must-gather-sqsxg\" (UID: \"48cc1d04-d03b-43fd-ab85-afe13c1650c7\") " pod="openshift-must-gather-g45m6/must-gather-sqsxg"
Sep 18 16:50:16.476221 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.476149    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g45m6/must-gather-sqsxg"
Sep 18 16:50:16.742642 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.742561    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g45m6/must-gather-sqsxg"]
Sep 18 16:50:16.746113 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:50:16.746062    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48cc1d04_d03b_43fd_ab85_afe13c1650c7.slice/crio-6553d31052a26efbdd1058d83ff7f78d87a86ff29b143f4aa02126c89bfec39e WatchSource:0}: Error finding container 6553d31052a26efbdd1058d83ff7f78d87a86ff29b143f4aa02126c89bfec39e: Status 404 returned error can't find the container with id 6553d31052a26efbdd1058d83ff7f78d87a86ff29b143f4aa02126c89bfec39e
Sep 18 16:50:16.747972 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:16.747954    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:50:17.660037 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:17.660004    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g45m6/must-gather-sqsxg" event={"ID":"48cc1d04-d03b-43fd-ab85-afe13c1650c7","Type":"ContainerStarted","Data":"bb4bf692ade9fba62a7dd78e929fc5a48e7bd5f701e5151d5e40489dbe44102e"}
Sep 18 16:50:17.660421 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:17.660044    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g45m6/must-gather-sqsxg" event={"ID":"48cc1d04-d03b-43fd-ab85-afe13c1650c7","Type":"ContainerStarted","Data":"c9daa63f8c3214a84ef4658ed7ffd74debfa8a5d0522ea87742fc285a4720f21"}
Sep 18 16:50:17.660421 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:17.660057    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g45m6/must-gather-sqsxg" event={"ID":"48cc1d04-d03b-43fd-ab85-afe13c1650c7","Type":"ContainerStarted","Data":"6553d31052a26efbdd1058d83ff7f78d87a86ff29b143f4aa02126c89bfec39e"}
Sep 18 16:50:17.687777 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:17.687729    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g45m6/must-gather-sqsxg" podStartSLOduration=0.98323874 podStartE2EDuration="1.687712262s" totalImagesPullingTime="704.473522ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:50:17.687050035 +0000 UTC m=+2056.142503116" watchObservedRunningTime="2026-09-18 16:50:17.687712262 +0000 UTC m=+2056.143165338"
Sep 18 16:50:18.973896 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:18.973864    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gwgv5_94d26f9a-f3f1-417e-91bd-0e74248bbbb4/global-pull-secret-syncer/0.log"
Sep 18 16:50:19.063109 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:19.063059    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zcmfd_8434b53d-830b-4df4-8cf1-e8ed6cf92b9f/konnectivity-agent/0.log"
Sep 18 16:50:19.150918 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:19.150880    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-142.ec2.internal_61ad4b4551fb28947d10dc4793af77f5/haproxy/0.log"
Sep 18 16:50:23.550295 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:23.550196    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lfh2m_1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4/node-exporter/0.log"
Sep 18 16:50:23.574524 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:23.574495    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lfh2m_1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4/kube-rbac-proxy/0.log"
Sep 18 16:50:23.596647 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:23.596479    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lfh2m_1017f4cb-796b-4c6c-9a9a-4e96ef08c9f4/init-textfile/0.log"
Sep 18 16:50:23.794965 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:23.794887    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_66ab706e-7e5f-4f92-a967-33f822ff4b64/prometheus/0.log"
Sep 18 16:50:23.815695 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:23.815621    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_66ab706e-7e5f-4f92-a967-33f822ff4b64/config-reloader/0.log"
Sep 18 16:50:23.847265 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:23.847181    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_66ab706e-7e5f-4f92-a967-33f822ff4b64/thanos-sidecar/0.log"
Sep 18 16:50:23.872144 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:23.872117    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_66ab706e-7e5f-4f92-a967-33f822ff4b64/kube-rbac-proxy-web/0.log"
Sep 18 16:50:23.899119 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:23.899095    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_66ab706e-7e5f-4f92-a967-33f822ff4b64/kube-rbac-proxy/0.log"
Sep 18 16:50:23.933492 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:23.933461    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_66ab706e-7e5f-4f92-a967-33f822ff4b64/kube-rbac-proxy-thanos/0.log"
Sep 18 16:50:23.962309 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:23.962284    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_66ab706e-7e5f-4f92-a967-33f822ff4b64/init-config-reloader/0.log"
Sep 18 16:50:24.167034 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:24.167006    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-656bb475f9-xkpvm_455c4a22-d4ae-4d97-b6be-33f2b1b264d2/thanos-query/0.log"
Sep 18 16:50:24.194351 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:24.194319    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-656bb475f9-xkpvm_455c4a22-d4ae-4d97-b6be-33f2b1b264d2/kube-rbac-proxy-web/0.log"
Sep 18 16:50:24.217465 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:24.217434    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-656bb475f9-xkpvm_455c4a22-d4ae-4d97-b6be-33f2b1b264d2/kube-rbac-proxy/0.log"
Sep 18 16:50:24.244442 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:24.244408    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-656bb475f9-xkpvm_455c4a22-d4ae-4d97-b6be-33f2b1b264d2/prom-label-proxy/0.log"
Sep 18 16:50:24.270030 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:24.269867    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-656bb475f9-xkpvm_455c4a22-d4ae-4d97-b6be-33f2b1b264d2/kube-rbac-proxy-rules/0.log"
Sep 18 16:50:24.294506 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:24.294456    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-656bb475f9-xkpvm_455c4a22-d4ae-4d97-b6be-33f2b1b264d2/kube-rbac-proxy-metrics/0.log"
Sep 18 16:50:26.174686 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:26.174648    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-vfmdv_1e3ca1ec-6856-4afb-850e-85e3fa968b3f/console-operator/1.log"
Sep 18 16:50:26.187223 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:26.187194    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-vfmdv_1e3ca1ec-6856-4afb-850e-85e3fa968b3f/console-operator/2.log"
Sep 18 16:50:27.045320 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:27.045279    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g45m6/perf-node-gather-daemonset-85ptp"]
Sep 18 16:50:27.051661 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:27.051631    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-85ptp"
Sep 18 16:50:27.059321 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:27.059287    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g45m6/perf-node-gather-daemonset-85ptp"]
Sep 18 16:50:27.210883 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:27.210840    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q74cw\" (UniqueName: \"kubernetes.io/projected/c7c78379-f478-4e2b-b145-7a66b0f48a04-kube-api-access-q74cw\") pod \"perf-node-gather-daemonset-85ptp\" (UID: \"c7c78379-f478-4e2b-b145-7a66b0f48a04\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-85ptp"
Sep 18 16:50:27.211556 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:27.211522    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c7c78379-f478-4e2b-b145-7a66b0f48a04-podres\") pod \"perf-node-gather-daemonset-85ptp\" (UID: \"c7c78379-f478-4e2b-b145-7a66b0f48a04\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-85ptp"
Sep 18 16:50:27.211688 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:27.211649    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7c78379-f478-4e2b-b145-7a66b0f48a04-sys\") pod \"perf-node-gather-daemonset-85ptp\" (UID: \"c7c78379-f478-4e2b-b145-7a66b0f48a04\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-85ptp"
Sep 18 16:50:27.211688 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:27.211680    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c7c78379-f478-4e2b-b145-7a66b0f48a04-proc\") pod \"perf-node-gather-daemonset-85ptp\" (UID: \"c7c78379-f478-4e2b-b145-7a66b0f48a04\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-85ptp"
Sep 18 16:50:27.211824 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:27.211724    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7c78379-f478-4e2b-b145-7a66b0f48a04-lib-modules\") pod \"perf-node-gather-daemonset-85ptp\" (UID: \"c7c78379-f478-4e2b-b145-7a66b0f48a04\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-85ptp"
Sep 18 16:50:27.313190 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:27.313106    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q74cw\" (UniqueName: \"kubernetes.io/projected/c7c78379-f478-4e2b-b145-7a66b0f48a04-kube-api-access-q74cw\") pod \"perf-node-gather-daemonset-85ptp\" (UID: \"c7c78379-f478-4e2b-b145-7a66b0f48a04\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-85ptp"
Sep 18 16:50:27.313683 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:27.313646    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c7c78379-f478-4e2b-b145-7a66b0f48a04-podres\") pod \"perf-node-gather-daemonset-85ptp\" (UID: \"c7c78379-f478-4e2b-b145-7a66b0f48a04\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-85ptp"
Sep 18 16:50:27.313797 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:27.313755    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7c78379-f478-4e2b-b145-7a66b0f48a04-sys\") pod \"perf-node-gather-daemonset-85ptp\" (UID: \"c7c78379-f478-4e2b-b145-7a66b0f48a04\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-85ptp"
Sep 18 16:50:27.313797 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:27.313779    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c7c78379-f478-4e2b-b145-7a66b0f48a04-proc\") pod \"perf-node-gather-daemonset-85ptp\" (UID: \"c7c78379-f478-4e2b-b145-7a66b0f48a04\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-85ptp"
Sep 18 16:50:27.313909 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:27.313809    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7c78379-f478-4e2b-b145-7a66b0f48a04-lib-modules\") pod \"perf-node-gather-daemonset-85ptp\" (UID: \"c7c78379-f478-4e2b-b145-7a66b0f48a04\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-85ptp"
Sep 18 16:50:27.313909 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:27.313827    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c7c78379-f478-4e2b-b145-7a66b0f48a04-podres\") pod \"perf-node-gather-daemonset-85ptp\" (UID: \"c7c78379-f478-4e2b-b145-7a66b0f48a04\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-85ptp"
Sep 18 16:50:27.313909 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:27.313885    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7c78379-f478-4e2b-b145-7a66b0f48a04-sys\") pod \"perf-node-gather-daemonset-85ptp\" (UID: \"c7c78379-f478-4e2b-b145-7a66b0f48a04\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-85ptp"
Sep 18 16:50:27.314024 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:27.313887    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c7c78379-f478-4e2b-b145-7a66b0f48a04-proc\") pod \"perf-node-gather-daemonset-85ptp\" (UID: \"c7c78379-f478-4e2b-b145-7a66b0f48a04\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-85ptp"
Sep 18 16:50:27.314024 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:27.313933    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7c78379-f478-4e2b-b145-7a66b0f48a04-lib-modules\") pod \"perf-node-gather-daemonset-85ptp\" (UID: \"c7c78379-f478-4e2b-b145-7a66b0f48a04\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-85ptp"
Sep 18 16:50:27.322382 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:27.322353    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q74cw\" (UniqueName: \"kubernetes.io/projected/c7c78379-f478-4e2b-b145-7a66b0f48a04-kube-api-access-q74cw\") pod \"perf-node-gather-daemonset-85ptp\" (UID: \"c7c78379-f478-4e2b-b145-7a66b0f48a04\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-85ptp"
Sep 18 16:50:27.365248 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:27.365214    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-85ptp"
Sep 18 16:50:27.653789 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:27.653744    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g45m6/perf-node-gather-daemonset-85ptp"]
Sep 18 16:50:27.659167 ip-10-0-138-142 kubenswrapper[2566]: W0918 16:50:27.659134    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc7c78379_f478_4e2b_b145_7a66b0f48a04.slice/crio-733e120164bba0358896a6e084eb7f3cceb7b59d62c20aeab129c438aee26b9f WatchSource:0}: Error finding container 733e120164bba0358896a6e084eb7f3cceb7b59d62c20aeab129c438aee26b9f: Status 404 returned error can't find the container with id 733e120164bba0358896a6e084eb7f3cceb7b59d62c20aeab129c438aee26b9f
Sep 18 16:50:27.720544 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:27.720514    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-85ptp" event={"ID":"c7c78379-f478-4e2b-b145-7a66b0f48a04","Type":"ContainerStarted","Data":"733e120164bba0358896a6e084eb7f3cceb7b59d62c20aeab129c438aee26b9f"}
Sep 18 16:50:28.161217 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:28.161183    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-787hc_83e8eb33-e386-4b60-98c6-a95b93c408cd/dns/0.log"
Sep 18 16:50:28.182300 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:28.182273    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-787hc_83e8eb33-e386-4b60-98c6-a95b93c408cd/kube-rbac-proxy/0.log"
Sep 18 16:50:28.299552 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:28.299509    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ptbtj_1d77222a-f546-4b33-b6be-fbfa3bf2f6d0/dns-node-resolver/0.log"
Sep 18 16:50:28.726902 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:28.726855    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-85ptp" event={"ID":"c7c78379-f478-4e2b-b145-7a66b0f48a04","Type":"ContainerStarted","Data":"4c4eab5d8306f886ee45f3758184a926cba6e007d0ae5e1e920bf35b27910b71"}
Sep 18 16:50:28.727119 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:28.726957    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-85ptp"
Sep 18 16:50:28.742475 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:28.742421    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-85ptp" podStartSLOduration=1.742405802 podStartE2EDuration="1.742405802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:50:28.742031755 +0000 UTC m=+2067.197484833" watchObservedRunningTime="2026-09-18 16:50:28.742405802 +0000 UTC m=+2067.197858914"
Sep 18 16:50:28.761260 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:28.761231    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-689468cdb8-lqfkq_8367995b-3d1a-4762-8445-6cba40de5867/registry/0.log"
Sep 18 16:50:28.828929 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:28.828899    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-szd7j_9665ba03-1cdc-46c9-84f2-3ca2137b66e2/node-ca/0.log"
Sep 18 16:50:29.899238 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:29.899204    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2wd2m_711861f8-db0f-4e93-a894-0cedf9efdf89/serve-healthcheck-canary/0.log"
Sep 18 16:50:30.330677 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:30.330649    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-v2fwq_7b329cbb-a9d0-4b69-bedb-e754d774d1ac/insights-operator/0.log"
Sep 18 16:50:30.332472 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:30.332452    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-v2fwq_7b329cbb-a9d0-4b69-bedb-e754d774d1ac/insights-operator/1.log"
Sep 18 16:50:30.356397 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:30.356379    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cxv7j_14e3d714-83c0-4ef8-8468-4143f8f2852d/kube-rbac-proxy/0.log"
Sep 18 16:50:30.378843 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:30.378821    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cxv7j_14e3d714-83c0-4ef8-8468-4143f8f2852d/exporter/0.log"
Sep 18 16:50:30.402469 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:30.402441    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cxv7j_14e3d714-83c0-4ef8-8468-4143f8f2852d/extractor/0.log"
Sep 18 16:50:32.927660 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:32.927618    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-g2mv4_18591da2-4faa-4c7f-96cf-f9b9e9dce2c6/manager/0.log"
Sep 18 16:50:33.108607 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:33.108575    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-5lvl8_edb62ba8-aecc-40fa-93d7-c865713ab9e8/s3-init/0.log"
Sep 18 16:50:33.124684 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:33.124654    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-5lvl8_edb62ba8-aecc-40fa-93d7-c865713ab9e8/download-hf-model/0.log"
Sep 18 16:50:34.742124 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:34.741304    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-85ptp"
Sep 18 16:50:37.698619 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:37.698588    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-jrgv9_dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5/kube-storage-version-migrator-operator/1.log"
Sep 18 16:50:37.699832 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:37.699808    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-jrgv9_dee5cfd6-82ac-4c82-8ebc-65c97d7c68f5/kube-storage-version-migrator-operator/0.log"
Sep 18 16:50:38.648031 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:38.648001    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7hjqm_1a1fa079-6d52-4844-bded-44bf16212a8a/kube-multus-additional-cni-plugins/0.log"
Sep 18 16:50:38.669658 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:38.669589    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7hjqm_1a1fa079-6d52-4844-bded-44bf16212a8a/egress-router-binary-copy/0.log"
Sep 18 16:50:38.692829 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:38.692798    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7hjqm_1a1fa079-6d52-4844-bded-44bf16212a8a/cni-plugins/0.log"
Sep 18 16:50:38.715923 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:38.715899    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7hjqm_1a1fa079-6d52-4844-bded-44bf16212a8a/bond-cni-plugin/0.log"
Sep 18 16:50:38.744712 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:38.744684    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7hjqm_1a1fa079-6d52-4844-bded-44bf16212a8a/routeoverride-cni/0.log"
Sep 18 16:50:38.770651 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:38.770628    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7hjqm_1a1fa079-6d52-4844-bded-44bf16212a8a/whereabouts-cni-bincopy/0.log"
Sep 18 16:50:38.795020 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:38.794993    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7hjqm_1a1fa079-6d52-4844-bded-44bf16212a8a/whereabouts-cni/0.log"
Sep 18 16:50:39.159919 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:39.159872    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jzsfr_eb932b90-e784-4e7e-8f10-64421f61edaa/kube-multus/0.log"
Sep 18 16:50:39.297210 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:39.297179    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zzxwf_bbd14141-b0c9-4e6d-aab3-c1091adc5ddf/network-metrics-daemon/0.log"
Sep 18 16:50:39.315045 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:39.315021    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zzxwf_bbd14141-b0c9-4e6d-aab3-c1091adc5ddf/kube-rbac-proxy/0.log"
Sep 18 16:50:40.242147 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:40.242102    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j92qj_30c9a034-ba9f-4ead-8685-316723e6852a/ovn-controller/0.log"
Sep 18 16:50:40.257588 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:40.257567    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j92qj_30c9a034-ba9f-4ead-8685-316723e6852a/ovn-acl-logging/0.log"
Sep 18 16:50:40.288253 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:40.288200    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j92qj_30c9a034-ba9f-4ead-8685-316723e6852a/kube-rbac-proxy-node/0.log"
Sep 18 16:50:40.314596 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:40.314572    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j92qj_30c9a034-ba9f-4ead-8685-316723e6852a/kube-rbac-proxy-ovn-metrics/0.log"
Sep 18 16:50:40.332629 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:40.332602    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j92qj_30c9a034-ba9f-4ead-8685-316723e6852a/northd/0.log"
Sep 18 16:50:40.352999 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:40.352979    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j92qj_30c9a034-ba9f-4ead-8685-316723e6852a/nbdb/0.log"
Sep 18 16:50:40.372771 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:40.372748    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j92qj_30c9a034-ba9f-4ead-8685-316723e6852a/sbdb/0.log"
Sep 18 16:50:40.429348 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:40.429321    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j92qj_30c9a034-ba9f-4ead-8685-316723e6852a/ovnkube-controller/0.log"
Sep 18 16:50:41.730042 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:41.730014    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-vnmfs_48d628be-136e-4521-a2fa-77f6c20b27d8/network-check-target-container/0.log"
Sep 18 16:50:42.735655 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:42.735623    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-xt2cm_044f1a79-aadb-474d-9525-6ccdb4bcbdf0/iptables-alerter/0.log"
Sep 18 16:50:43.426250 ip-10-0-138-142 kubenswrapper[2566]: I0918 16:50:43.426224    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-v26zp_a53312cc-6d62-41f2-b3ef-c7b41da2219c/tuned/0.log"