Sep 17 17:33:44.569368 ip-10-0-142-4 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 17:33:44.908018 ip-10-0-142-4 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:33:44.908018 ip-10-0-142-4 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 17:33:44.908018 ip-10-0-142-4 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:33:44.908018 ip-10-0-142-4 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 17:33:44.908018 ip-10-0-142-4 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:33:44.909720 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.909611    2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 17:33:44.913279 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913264    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:44.913315 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913281    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:44.913315 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913286    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:44.913315 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913289    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:44.913315 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913293    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:44.913315 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913296    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:44.913315 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913300    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:44.913315 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913303    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:44.913315 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913306    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:44.913315 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913308    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:44.913315 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913311    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:44.913315 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913314    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:44.913315 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913317    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:44.913315 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913319    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:44.913618 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913323    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:44.913618 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913326    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:44.913618 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913329    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:44.913618 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913332    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:44.913618 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913335    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:44.913618 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913337    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:44.913618 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913340    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:44.913618 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913342    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:44.913618 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913345    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:44.913618 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913348    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:44.913618 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913350    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:44.913618 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913353    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:44.913618 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913356    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:44.913618 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913358    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:44.913618 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913361    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:44.913618 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913363    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:44.913618 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913366    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:44.913618 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913369    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:44.913618 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913371    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:44.913618 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913374    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:44.914107 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913376    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:44.914107 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913378    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:44.914107 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913381    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:44.914107 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913384    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:44.914107 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913386    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:44.914107 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913389    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:44.914107 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913392    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:44.914107 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913394    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:44.914107 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913399    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:44.914107 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913403    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:44.914107 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913406    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:44.914107 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913409    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:44.914107 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913412    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:44.914107 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913415    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:44.914107 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913417    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:44.914107 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913420    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:44.914107 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913422    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:44.914107 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913425    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:44.914107 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913428    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:44.914573 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913430    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:44.914573 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913433    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:44.914573 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913436    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:44.914573 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913438    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:44.914573 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913441    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:44.914573 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913443    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:44.914573 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913446    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:44.914573 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913449    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:44.914573 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913451    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:44.914573 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913454    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:44.914573 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913457    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:44.914573 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913459    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:44.914573 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913462    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:44.914573 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913464    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:44.914573 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913466    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:44.914573 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913469    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:44.914573 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913472    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:44.914573 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913475    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:44.914573 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913478    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:44.914573 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913480    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:44.915062 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913483    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:44.915062 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913485    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:44.915062 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913488    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:44.915062 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913490    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:44.915062 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913494    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:44.915062 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913499    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:44.915062 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913502    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:44.915062 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913504    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:44.915062 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913507    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:44.915062 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913510    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:44.915062 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913512    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:44.915062 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913515    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:44.915062 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913517    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:44.915062 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913520    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:44.915062 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913522    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:44.915062 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913524    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:44.915062 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913527    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:44.915062 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913529    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:44.915062 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913533    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:44.915541 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913536    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:44.915541 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913538    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:44.915541 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913541    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:44.915541 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913543    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:44.915541 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913546    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:44.915541 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913549    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:44.915541 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.913551    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:44.915541 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914232    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:44.915541 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914239    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:44.915541 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914242    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:44.915541 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914251    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:44.915541 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914255    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:44.915541 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914258    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:44.915541 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914261    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:44.915541 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914263    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:44.915541 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914266    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:44.915541 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914268    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:44.915541 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914271    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:44.915541 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914274    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:44.915541 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914277    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:44.916072 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914280    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:44.916072 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914282    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:44.916072 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914285    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:44.916072 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914288    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:44.916072 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914291    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:44.916072 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914293    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:44.916072 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914296    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:44.916072 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914299    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:44.916072 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914301    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:44.916072 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914305    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:44.916072 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914309    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:44.916072 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914313    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:44.916072 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914315    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:44.916072 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914318    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:44.916072 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914321    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:44.916072 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914325    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:44.916072 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914329    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:44.916072 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914332    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:44.916072 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914335    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:44.916533 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914337    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:44.916533 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914340    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:44.916533 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914344    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:44.916533 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914346    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:44.916533 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914350    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:44.916533 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914353    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:44.916533 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914355    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:44.916533 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914358    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:44.916533 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914361    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:44.916533 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914363    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:44.916533 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914366    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:44.916533 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914369    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:44.916533 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914371    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:44.916533 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914374    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:44.916533 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914378    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:44.916533 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914380    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:44.916533 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914383    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:44.916533 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914385    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:44.916533 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914388    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:44.916533 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914390    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:44.917021 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914393    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:44.917021 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914396    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:44.917021 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914398    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:44.917021 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914401    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:44.917021 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914404    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:44.917021 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914407    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:44.917021 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914410    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:44.917021 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914413    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:44.917021 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914416    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:44.917021 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914419    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:44.917021 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914422    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:44.917021 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914424    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:44.917021 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914427    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:44.917021 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914430    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:44.917021 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914433    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:44.917021 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914436    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:44.917021 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914439    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:44.917021 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914442    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:44.917021 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914444    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:44.917486 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914447    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:44.917486 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914449    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:44.917486 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914452    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:44.917486 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914455    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:44.917486 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914457    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:44.917486 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914460    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:44.917486 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914462    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:44.917486 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914465    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:44.917486 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914468    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:44.917486 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914470    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:44.917486 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914473    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:44.917486 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914475    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:44.917486 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914478    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:44.917486 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914481    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:44.917486 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914483    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:44.917486 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914486    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:44.917486 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914488    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:44.917486 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914490    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:44.917486 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914493    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:44.917486 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914496    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:44.917967 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914498    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:44.917967 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914501    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:44.917967 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914503    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:44.917967 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914505    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:44.917967 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914508    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:44.917967 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914510    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:44.917967 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914514    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:44.917967 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914517    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:44.917967 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914881    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:44.917967 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914886    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:44.917967 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914889    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:44.917967 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914892    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:44.917967 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914894    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:44.917967 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914897    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:44.917967 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914900    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:44.917967 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914902    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:44.917967 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914905    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:44.917967 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914909    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:44.917967 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914913    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:44.918437 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914916    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:44.918437 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914919    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:44.918437 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914922    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:44.918437 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914924    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:44.918437 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914926    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:44.918437 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914929    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:44.918437 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914931    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:44.918437 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914934    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:44.918437 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914937    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:44.918437 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914939    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:44.918437 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914942    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:44.918437 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914945    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:44.918437 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914947    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:44.918437 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914950    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:44.918437 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914953    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:44.918437 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914955    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:44.918437 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914958    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:44.918437 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914960    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:44.918437 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914963    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:44.918437 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914966    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:44.918915 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914968    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:44.918915 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914971    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:44.918915 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914974    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:44.918915 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914977    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:44.918915 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914980    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:44.918915 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914983    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:44.918915 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914985    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:44.918915 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914989    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:44.918915 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914992    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:44.918915 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914994    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:44.918915 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.914997    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:44.918915 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915000    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:44.918915 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915003    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:44.918915 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915006    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:44.918915 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915008    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:44.918915 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915011    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:44.918915 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915014    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:44.918915 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915016    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:44.918915 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915019    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:44.918915 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915021    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:44.919453 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915024    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:44.919453 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915027    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:44.919453 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915030    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:44.919453 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915033    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:44.919453 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915036    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:44.919453 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915039    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:44.919453 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915041    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:44.919453 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915044    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:44.919453 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915046    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:44.919453 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915049    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:44.919453 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915065    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:44.919453 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915070    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:44.919453 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915073    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:44.919453 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915077    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:44.919453 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915080    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:44.919453 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915082    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:44.919453 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915085    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:44.919453 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915088    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:44.919453 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915091    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:44.919453 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915093    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:44.919925 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915096    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:44.919925 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915098    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:44.919925 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915100    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:44.919925 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915103    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:44.919925 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915106    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:44.919925 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915108    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:44.919925 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915111    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:44.919925 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915113    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:44.919925 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915116    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:44.919925 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915118    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:44.919925 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915121    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:44.919925 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915123    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:44.919925 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915126    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:44.919925 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915128    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:44.919925 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915131    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:44.919925 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915134    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:44.919925 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915137    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:44.919925 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915139    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:44.919925 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915142    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:44.919925 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915144    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:44.920420 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915147    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:44.920420 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915149    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:44.920420 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915152    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:44.920420 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915156    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:44.920420 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915159    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:44.920420 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915161    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:44.920420 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915164    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:44.920420 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915168    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:44.920420 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915238    2574 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 17:33:44.920420 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915268    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 17:33:44.920420 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915275    2574 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 17:33:44.920420 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915279    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 17:33:44.920420 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915283    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 17:33:44.920420 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915286    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 17:33:44.920420 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915291    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 17:33:44.920420 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915294    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 17:33:44.920420 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915298    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 17:33:44.920420 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915301    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 17:33:44.920420 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915304    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 17:33:44.920420 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915315    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 17:33:44.920420 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915319    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 17:33:44.920420 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915322    2574 flags.go:64] FLAG: --cgroup-root=""
Sep 17 17:33:44.920975 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915325    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 17:33:44.920975 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915328    2574 flags.go:64] FLAG: --client-ca-file=""
Sep 17 17:33:44.920975 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915330    2574 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 17:33:44.920975 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915333    2574 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 17:33:44.920975 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915337    2574 flags.go:64] FLAG: --cluster-domain=""
Sep 17 17:33:44.920975 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915341    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 17:33:44.920975 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915344    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 17:33:44.920975 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915347    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 17:33:44.920975 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915350    2574 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 17:33:44.920975 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915354    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 17:33:44.920975 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915357    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 17:33:44.920975 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915360    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 17:33:44.920975 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915363    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 17:33:44.920975 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915368    2574 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 17:33:44.920975 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915371    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 17:33:44.920975 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915373    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 17:33:44.920975 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915376    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 17:33:44.920975 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915379    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 17:33:44.920975 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915383    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 17:33:44.920975 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915386    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 17:33:44.920975 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915389    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 17:33:44.920975 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915393    2574 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 17:33:44.920975 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915396    2574 flags.go:64] FLAG: --enable-server="true"
Sep 17 17:33:44.920975 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915399    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 17:33:44.920975 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915403    2574 flags.go:64] FLAG: --event-burst="100"
Sep 17 17:33:44.921596 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915407    2574 flags.go:64] FLAG: --event-qps="50"
Sep 17 17:33:44.921596 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915410    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 17:33:44.921596 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915413    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 17:33:44.921596 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915415    2574 flags.go:64] FLAG: --eviction-hard=""
Sep 17 17:33:44.921596 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915419    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 17:33:44.921596 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915422    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 17:33:44.921596 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915425    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 17:33:44.921596 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915428    2574 flags.go:64] FLAG: --eviction-soft=""
Sep 17 17:33:44.921596 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915430    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 17:33:44.921596 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915433    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 17:33:44.921596 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915436    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 17:33:44.921596 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915439    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 17:33:44.921596 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915444    2574 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 17:33:44.921596 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915447    2574 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 17:33:44.921596 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915450    2574 flags.go:64] FLAG: --feature-gates=""
Sep 17 17:33:44.921596 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915454    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 17:33:44.921596 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915456    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 17:33:44.921596 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915459    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 17:33:44.921596 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915463    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 17:33:44.921596 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915466    2574 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 17:33:44.921596 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915470    2574 flags.go:64] FLAG: --help="false"
Sep 17 17:33:44.921596 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915473    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-142-4.ec2.internal"
Sep 17 17:33:44.921596 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915476    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 17:33:44.921596 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915479    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 17:33:44.922323 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915482    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 17:33:44.922323 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915486    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 17:33:44.922323 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915489    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 17:33:44.922323 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915492    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 17:33:44.922323 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915495    2574 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 17:33:44.922323 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915498    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 17:33:44.922323 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915501    2574 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 17:33:44.922323 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915504    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 17:33:44.922323 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915507    2574 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 17:33:44.922323 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915510    2574 flags.go:64] FLAG: --kube-reserved=""
Sep 17 17:33:44.922323 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915513    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 17:33:44.922323 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915516    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 17:33:44.922323 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915519    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 17:33:44.922323 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915521    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 17:33:44.922323 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915524    2574 flags.go:64] FLAG: --lock-file=""
Sep 17 17:33:44.922323 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915527    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 17:33:44.922323 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915530    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 17:33:44.922323 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915533    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 17:33:44.922323 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915538    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 17:33:44.922323 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915541    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 17:33:44.922323 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915545    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 17:33:44.922323 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915548    2574 flags.go:64] FLAG: --logging-format="text"
Sep 17 17:33:44.922323 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915551    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 17:33:44.922883 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915555    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 17:33:44.922883 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915558    2574 flags.go:64] FLAG: --manifest-url=""
Sep 17 17:33:44.922883 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915561    2574 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 17:33:44.922883 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915565    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 17:33:44.922883 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915568    2574 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 17:33:44.922883 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915573    2574 flags.go:64] FLAG: --max-pods="110"
Sep 17 17:33:44.922883 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915576    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 17:33:44.922883 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915579    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 17:33:44.922883 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915581    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 17:33:44.922883 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915584    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 17:33:44.922883 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915587    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 17:33:44.922883 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915590    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 17:33:44.922883 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915593    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 17:33:44.922883 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915599    2574 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 17:33:44.922883 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915603    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 17:33:44.922883 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915606    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 17:33:44.922883 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915609    2574 flags.go:64] FLAG: --pod-cidr=""
Sep 17 17:33:44.922883 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915612    2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 17:33:44.922883 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915617    2574 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 17:33:44.922883 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915620    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 17:33:44.922883 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915623    2574 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 17:33:44.922883 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915626    2574 flags.go:64] FLAG: --port="10250"
Sep 17 17:33:44.922883 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915629    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 17:33:44.922883 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915632    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b440eb3c3a9f9a6a"
Sep 17 17:33:44.923496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915635    2574 flags.go:64] FLAG: --qos-reserved=""
Sep 17 17:33:44.923496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915638    2574 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 17:33:44.923496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915641    2574 flags.go:64] FLAG: --register-node="true"
Sep 17 17:33:44.923496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915643    2574 flags.go:64] FLAG: --register-with-taints=""
Sep 17 17:33:44.923496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915647    2574 flags.go:64] FLAG: --registry-burst="10"
Sep 17 17:33:44.923496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915651    2574 flags.go:64] FLAG: --registry-qps="5"
Sep 17 17:33:44.923496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915654    2574 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 17:33:44.923496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915657    2574 flags.go:64] FLAG: --reserved-memory=""
Sep 17 17:33:44.923496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915661    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 17:33:44.923496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915664    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 17:33:44.923496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915667    2574 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 17:33:44.923496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915670    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 17:33:44.923496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915672    2574 flags.go:64] FLAG: --runonce="false"
Sep 17 17:33:44.923496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915675    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 17:33:44.923496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915679    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 17:33:44.923496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915682    2574 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 17:33:44.923496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915685    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 17:33:44.923496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915688    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 17:33:44.923496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915691    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 17:33:44.923496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915694    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 17:33:44.923496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915697    2574 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 17:33:44.923496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915700    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 17:33:44.923496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915703    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 17:33:44.923496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915706    2574 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 17:33:44.923496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915709    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 17:33:44.923496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915713    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 17:33:44.924201 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915716    2574 flags.go:64] FLAG: --system-cgroups=""
Sep 17 17:33:44.924201 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915718    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 17:33:44.924201 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915724    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 17:33:44.924201 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915726    2574 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 17:33:44.924201 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915729    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 17:33:44.924201 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915733    2574 flags.go:64] FLAG: --tls-min-version=""
Sep 17 17:33:44.924201 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915735    2574 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 17:33:44.924201 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915738    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 17:33:44.924201 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915741    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 17:33:44.924201 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915744    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 17:33:44.924201 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915747    2574 flags.go:64] FLAG: --v="2"
Sep 17 17:33:44.924201 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915752    2574 flags.go:64] FLAG: --version="false"
Sep 17 17:33:44.924201 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915756    2574 flags.go:64] FLAG: --vmodule=""
Sep 17 17:33:44.924201 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915760    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 17:33:44.924201 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.915763    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 17:33:44.924201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915873    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:44.924201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915878    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:44.924201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915881    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:44.924201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915884    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:44.924201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915887    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:44.924201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915890    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:44.924201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915893    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:44.924201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915896    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:44.924891 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915899    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:44.924891 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915902    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:44.924891 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915905    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:44.924891 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915908    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:44.924891 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915912    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:44.924891 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915916    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:44.924891 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915921    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:44.924891 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915924    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:44.924891 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915928    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:44.924891 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915931    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:44.924891 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915934    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:44.924891 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915936    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:44.924891 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915939    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:44.924891 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915941    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:44.924891 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915944    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:44.924891 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915947    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:44.924891 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915949    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:44.924891 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915952    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:44.924891 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915954    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:44.925466 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915957    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:44.925466 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915961    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:44.925466 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915964    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:44.925466 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915966    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:44.925466 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915969    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:44.925466 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915971    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:44.925466 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915974    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:44.925466 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915977    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:44.925466 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915980    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:44.925466 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915982    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:44.925466 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915985    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:44.925466 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915988    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:44.925466 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915990    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:44.925466 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915993    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:44.925466 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915995    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:44.925466 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.915998    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:44.925466 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916002    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:44.925466 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916006    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:44.925466 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916009    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:44.925466 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916012    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:44.925960 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916015    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:44.925960 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916019    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:44.925960 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916022    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:44.925960 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916025    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:44.925960 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916028    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:44.925960 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916030    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:44.925960 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916033    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:44.925960 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916036    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:44.925960 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916039    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:44.925960 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916041    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:44.925960 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916044    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:44.925960 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916047    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:44.925960 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916061    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:44.925960 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916065    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:44.925960 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916069    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:44.925960 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916072    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:44.925960 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916075    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:44.925960 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916078    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:44.925960 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916081    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:44.925960 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916084    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:44.926470 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916086    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:44.926470 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916089    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:44.926470 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916092    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:44.926470 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916094    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:44.926470 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916097    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:44.926470 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916099    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:44.926470 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916102    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:44.926470 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916105    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:44.926470 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916108    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:44.926470 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916110    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:44.926470 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916113    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:44.926470 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916116    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:44.926470 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916118    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:44.926470 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916121    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:44.926470 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916123    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:44.926470 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916126    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:44.926470 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916129    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:44.926470 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916131    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:44.926470 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916134    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:44.926470 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916137    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:44.926938 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916140    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:44.926938 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916143    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:44.926938 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916147    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:44.926938 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916150    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:44.926938 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916152    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:44.926938 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916156    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:44.926938 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916160    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:44.926938 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916162    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:44.926938 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916165    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:44.926938 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916168    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:44.926938 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916170    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:44.926938 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.916173    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:44.926938 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.916633    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:33:44.926938 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.922895    2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 17:33:44.926938 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.922908    2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 17:33:44.927336 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.923521    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 17:33:44.927336 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.923531    2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 17:33:44.927336 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.923680    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:44.927336 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.923687    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:44.927336 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.923691    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:44.927336 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.923694    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:44.927336 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.923697    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:44.927336 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.923700    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:44.927336 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.923708    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:44.927336 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.923711    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:44.927336 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.923714    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:44.927336 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.923717    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:44.927336 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.923720    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:44.927336 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.923724    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:44.927336 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.923728    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:44.927336 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.923732    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:44.927336 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925256    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:44.927336 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925298    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:44.927336 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925304    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:44.927804 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925309    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:44.927804 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925313    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:44.927804 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925318    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:44.927804 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925322    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:44.927804 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925326    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:44.927804 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925331    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:44.927804 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925352    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:44.927804 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925367    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:44.927804 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925373    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:44.927804 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925381    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:44.927804 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925384    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:44.927804 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925387    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:44.927804 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925391    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:44.927804 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925396    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:44.927804 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925399    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:44.927804 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925403    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:44.927804 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925414    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:44.927804 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925420    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:44.927804 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925430    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:44.928305 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925536    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:44.928305 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925543    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:44.928305 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925547    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:44.928305 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925550    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:44.928305 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925553    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:44.928305 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925556    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:44.928305 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925559    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:44.928305 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925562    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:44.928305 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925565    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:44.928305 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925568    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:44.928305 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925570    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:44.928305 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925573    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:44.928305 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925575    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:44.928305 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925578    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:44.928305 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925580    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:44.928305 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925583    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:44.928305 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925586    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:44.928305 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925589    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:44.928305 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925591    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:44.928305 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925597    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:44.928785 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925601    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:44.928785 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925604    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:44.928785 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925606    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:44.928785 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925609    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:44.928785 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925612    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:44.928785 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925615    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:44.928785 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925617    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:44.928785 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925620    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:44.928785 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925623    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:44.928785 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925625    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:44.928785 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925629    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:44.928785 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925631    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:44.928785 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925634    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:44.928785 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925636    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:44.928785 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925639    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:44.928785 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925642    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:44.928785 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925645    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:44.928785 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925647    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:44.928785 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925650    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:44.928785 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925653    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:44.929324 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925655    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:44.929324 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925658    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:44.929324 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925661    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:44.929324 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925663    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:44.929324 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925666    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:44.929324 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925669    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:44.929324 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925671    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:44.929324 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925674    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:44.929324 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925676    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:44.929324 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925679    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:44.929324 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925681    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:44.929324 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925684    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:44.929324 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925687    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:44.929324 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925690    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:44.929324 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925692    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:44.929324 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925695    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:44.929324 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925698    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:44.929324 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925700    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:44.929324 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925703    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:44.929324 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925706    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:44.929920 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925709    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:44.929920 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925712    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:44.929920 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925714    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:44.929920 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.925720    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:33:44.929920 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925854    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:44.929920 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925859    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:44.929920 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925863    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:44.929920 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925866    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:44.929920 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925870    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:44.929920 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925874    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:44.929920 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925877    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:44.929920 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925880    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:44.929920 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925882    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:44.929920 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925885    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:44.930338 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925888    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:44.930338 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925890    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:44.930338 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925893    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:44.930338 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925895    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:44.930338 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925898    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:44.930338 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925900    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:44.930338 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925903    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:44.930338 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925906    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:44.930338 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925908    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:44.930338 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925911    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:44.930338 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925914    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:44.930338 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925916    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:44.930338 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925919    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:44.930338 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925921    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:44.930338 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925924    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:44.930338 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925926    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:44.930338 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925929    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:44.930338 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925932    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:44.930338 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925934    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:44.930811 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925937    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:44.930811 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925939    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:44.930811 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925942    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:44.930811 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925944    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:44.930811 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925947    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:44.930811 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925950    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:44.930811 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925952    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:44.930811 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925955    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:44.930811 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925958    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:44.930811 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925961    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:44.930811 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925964    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:44.930811 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925968    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:44.930811 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925971    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:44.930811 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925974    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:44.930811 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925977    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:44.930811 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925979    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:44.930811 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925982    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:44.930811 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925984    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:44.930811 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925987    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:44.930811 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925989    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:44.931375 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925993    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:44.931375 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925995    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:44.931375 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.925998    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:44.931375 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926000    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:44.931375 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926003    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:44.931375 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926006    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:44.931375 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926008    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:44.931375 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926011    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:44.931375 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926014    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:44.931375 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926017    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:44.931375 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926021    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:44.931375 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926024    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:44.931375 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926027    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:44.931375 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926030    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:44.931375 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926032    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:44.931375 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926035    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:44.931375 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926038    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:44.931375 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926041    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:44.931375 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926044    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:44.931375 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926046    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:44.932201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926063    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:44.932201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926068    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:44.932201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926071    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:44.932201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926074    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:44.932201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926076    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:44.932201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926079    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:44.932201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926081    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:44.932201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926084    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:44.932201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926086    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:44.932201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926089    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:44.932201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926091    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:44.932201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926094    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:44.932201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926096    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:44.932201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926100    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:44.932201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926102    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:44.932201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926105    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:44.932201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926107    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:44.932201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926110    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:44.932201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926113    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:44.932201 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926116    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:44.932805 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926119    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:44.932805 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926121    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:44.932805 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926124    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:44.932805 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926126    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:44.932805 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926129    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:44.932805 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926131    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:44.932805 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926134    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:44.932805 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926136    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:44.932805 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926139    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:44.932805 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:44.926142    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:44.932805 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.926147    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:33:44.932805 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.926242    2574 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 17:33:44.932805 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.928692    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 17:33:44.932805 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.929474    2574 server.go:1019] "Starting client certificate rotation"
Sep 17 17:33:44.933174 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.929582    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:33:44.933174 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.929637    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:33:44.933174 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.931662    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:33:44.933174 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.932762    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:33:44.943355 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.943339    2574 log.go:25] "Validated CRI v1 runtime API"
Sep 17 17:33:44.944750 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.944737    2574 log.go:25] "Validated CRI v1 image API"
Sep 17 17:33:44.946010 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.945993    2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 17:33:44.948473 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.948449    2574 fs.go:135] Filesystem UUIDs: map[6138fc73-de67-4aa7-ba85-76e35fe0c41f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ced195b2-7290-45f6-bb87-b03d4a2b1db4:/dev/nvme0n1p3]
Sep 17 17:33:44.948547 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.948469    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 17:33:44.954485 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.954383    2574 manager.go:216] Machine: {Timestamp:2026-09-17 17:33:44.952985448 +0000 UTC m=+0.282085062 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100642 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23b3e3b34cf55f6ddb535326ea28e0 SystemUUID:ec23b3e3-b34c-f55f-6ddb-535326ea28e0 BootID:09a56376-0d6f-405a-b2ea-3dec1c974391 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f5:9d:1a:c5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f5:9d:1a:c5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:80:1e:66:03:35 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 17:33:44.954485 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.954484    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 17:33:44.954597 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.954554    2574 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 17:33:44.956073 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.956042    2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 17:33:44.956213 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.956074    2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-4.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 17:33:44.957008 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.956999    2574 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 17:33:44.957043 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.957017    2574 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 17:33:44.957043 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.957025    2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:33:44.957676 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.957667    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:33:44.957740 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.957732    2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 17:33:44.960861 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.960850    2574 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:33:44.960984 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.960974    2574 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 17:33:44.961025 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.961006    2574 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 17:33:44.961025 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.961021    2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 17:33:44.961118 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.961038    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 17:33:44.961118 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.961047    2574 kubelet.go:398] "Adding apiserver pod source"
Sep 17 17:33:44.961118 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.961070    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 17:33:44.961516 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.961494    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:33:44.961516 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.961520    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:33:44.965269 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.965243    2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 17:33:44.967097 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.967084    2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 17:33:44.967153 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.967113    2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 17:33:44.968501 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.968489    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 17:33:44.968550 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.968507    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 17:33:44.968550 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.968513    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 17:33:44.968550 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.968518    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 17:33:44.968550 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.968526    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 17:33:44.968550 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.968531    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 17:33:44.968550 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.968543    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 17:33:44.968550 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.968548    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 17:33:44.968769 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.968555    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 17:33:44.968769 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.968561    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 17:33:44.968769 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.968569    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 17:33:44.968769 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.968574    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 17:33:44.969154 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.969145    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 17:33:44.969154 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.969154    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 17:33:44.972004 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.971986    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:33:44.972704 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.972691    2574 server.go:1268] "Started kubelet"
Sep 17 17:33:44.972821 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.972741    2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 17:33:44.972895 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.972800    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 17:33:44.972895 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.972880    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 17:33:44.973230 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:44.973194    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-4.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 17:33:44.973313 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.973236    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-4.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 17:33:44.973313 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:44.973239    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 17:33:44.973547 ip-10-0-142-4 systemd[1]: Started Kubernetes Kubelet.
Sep 17 17:33:44.974345 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.974328    2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 17:33:44.976044 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.976029    2574 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 17:33:44.977610 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:44.977592    2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 17:33:44.979914 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.979894    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 17:33:44.980432 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.980414    2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 17:33:44.980789 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:44.980764    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-4.ec2.internal\" not found"
Sep 17 17:33:44.981066 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.981034    2574 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 17:33:44.981066 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.981070    2574 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 17:33:44.981249 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.981148    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 17:33:44.981249 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.981237    2574 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 17:33:44.981249 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.981244    2574 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 17:33:44.981708 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.981692    2574 factory.go:55] Registering systemd factory
Sep 17 17:33:44.981708 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.981708    2574 factory.go:223] Registration of the systemd container factory successfully
Sep 17 17:33:44.981932 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.981921    2574 factory.go:153] Registering CRI-O factory
Sep 17 17:33:44.981932 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.981933    2574 factory.go:223] Registration of the crio container factory successfully
Sep 17 17:33:44.982004 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.981992    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 17:33:44.982045 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.982006    2574 factory.go:103] Registering Raw factory
Sep 17 17:33:44.982045 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.982015    2574 manager.go:1196] Started watching for new ooms in manager
Sep 17 17:33:44.982487 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.982469    2574 manager.go:349] Starting recovery of all containers
Sep 17 17:33:44.984369 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:44.984326    2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-4.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 17:33:44.984442 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:44.984425    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 17:33:44.985541 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:44.984529    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-4.ec2.internal.18d62c21a6f05f45  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-4.ec2.internal,UID:ip-10-0-142-4.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-4.ec2.internal,},FirstTimestamp:2026-09-17 17:33:44.972664645 +0000 UTC m=+0.301764261,LastTimestamp:2026-09-17 17:33:44.972664645 +0000 UTC m=+0.301764261,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-4.ec2.internal,}"
Sep 17 17:33:44.993382 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.993365    2574 manager.go:354] Recovery completed
Sep 17 17:33:44.997914 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.997899    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:44.999503 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.999486    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:44.999578 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.999518    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:44.999578 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:44.999533    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:45.000100 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.000083    2574 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 17:33:45.000100 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.000100    2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 17:33:45.000215 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.000115    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:45.000215 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.000154    2574 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:33:45.000853 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.000783    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:45.000853 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.000807    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:45.000853 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.000821    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:45.001955 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:45.001889    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-4.ec2.internal.18d62c21a889e5df  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-4.ec2.internal,UID:ip-10-0-142-4.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-4.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-4.ec2.internal,},FirstTimestamp:2026-09-17 17:33:44.999503327 +0000 UTC m=+0.328602946,LastTimestamp:2026-09-17 17:33:44.999503327 +0000 UTC m=+0.328602946,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-4.ec2.internal,}"
Sep 17 17:33:45.002066 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.002040    2574 policy_none.go:49] "None policy: Start"
Sep 17 17:33:45.002111 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.002107    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 17:33:45.002148 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.002119    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 17:33:45.003940 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.003928    2574 policy_none.go:47] "Start"
Sep 17 17:33:45.006705 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.006688    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4c6s7"
Sep 17 17:33:45.013326 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:45.013270    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-4.ec2.internal.18d62c21a88a3b54  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-4.ec2.internal,UID:ip-10-0-142-4.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-4.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-4.ec2.internal,},FirstTimestamp:2026-09-17 17:33:44.999525204 +0000 UTC m=+0.328624822,LastTimestamp:2026-09-17 17:33:44.999525204 +0000 UTC m=+0.328624822,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-4.ec2.internal,}"
Sep 17 17:33:45.017980 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:45.017873    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-4.ec2.internal.18d62c21a88a7237  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-4.ec2.internal,UID:ip-10-0-142-4.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-4.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-4.ec2.internal,},FirstTimestamp:2026-09-17 17:33:44.999539255 +0000 UTC m=+0.328638873,LastTimestamp:2026-09-17 17:33:44.999539255 +0000 UTC m=+0.328638873,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-4.ec2.internal,}"
Sep 17 17:33:45.019855 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.019836    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4c6s7"
Sep 17 17:33:45.026714 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.026689    2574 manager.go:337] "Starting Device Plugin manager"
Sep 17 17:33:45.026809 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:45.026725    2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 17:33:45.026809 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.026737    2574 server.go:87] "Starting device plugin registration server"
Sep 17 17:33:45.027042 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.027029    2574 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 17:33:45.027133 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.027043    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 17:33:45.027197 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.027149    2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 17:33:45.027250 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.027235    2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 17:33:45.027250 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.027245    2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 17:33:45.028107 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:45.028084    2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 17:33:45.028185 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:45.028149    2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-4.ec2.internal\" not found"
Sep 17 17:33:45.108690 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.108656    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 17:33:45.109824 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.109810    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 17:33:45.109905 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.109861    2574 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 17:33:45.109905 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.109882    2574 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 17:33:45.109999 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:45.109922    2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 17:33:45.114070 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.114039    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:33:45.128037 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.128024    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:45.128930 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.128913    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:45.128986 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.128946    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:45.128986 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.128960    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:45.128986 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.128985    2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-4.ec2.internal"
Sep 17 17:33:45.149142 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.149124    2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-4.ec2.internal"
Sep 17 17:33:45.149189 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:45.149145    2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-4.ec2.internal\": node \"ip-10-0-142-4.ec2.internal\" not found"
Sep 17 17:33:45.177190 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:45.177131    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-4.ec2.internal\" not found"
Sep 17 17:33:45.210627 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.210600    2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-4.ec2.internal"]
Sep 17 17:33:45.210700 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.210662    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:45.211593 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.211578    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:45.211645 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.211609    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:45.211645 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.211624    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:45.213905 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.213893    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:45.214016 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.214002    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal"
Sep 17 17:33:45.214078 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.214028    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:45.215718 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.215705    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:45.215781 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.215720    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:45.215781 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.215734    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:45.215781 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.215744    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:45.215781 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.215750    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:45.215781 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.215763    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:45.218096 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.218079    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-4.ec2.internal"
Sep 17 17:33:45.218182 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.218110    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:45.218859 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.218843    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:45.218921 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.218871    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:45.218921 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.218881    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:45.250585 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:45.250570    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-4.ec2.internal\" not found" node="ip-10-0-142-4.ec2.internal"
Sep 17 17:33:45.255007 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:45.254992    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-4.ec2.internal\" not found" node="ip-10-0-142-4.ec2.internal"
Sep 17 17:33:45.278079 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:45.278038    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-4.ec2.internal\" not found"
Sep 17 17:33:45.283351 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.283335    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/68edadcfa29cc8be7e22dcaea45c88a8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal\" (UID: \"68edadcfa29cc8be7e22dcaea45c88a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal"
Sep 17 17:33:45.283401 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.283360    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68edadcfa29cc8be7e22dcaea45c88a8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal\" (UID: \"68edadcfa29cc8be7e22dcaea45c88a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal"
Sep 17 17:33:45.283401 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.283377    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3771e4c220b5a223d0f484f25cdc29e8-config\") pod \"kube-apiserver-proxy-ip-10-0-142-4.ec2.internal\" (UID: \"3771e4c220b5a223d0f484f25cdc29e8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-4.ec2.internal"
Sep 17 17:33:45.379071 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:45.379038    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-4.ec2.internal\" not found"
Sep 17 17:33:45.383483 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.383467    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/68edadcfa29cc8be7e22dcaea45c88a8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal\" (UID: \"68edadcfa29cc8be7e22dcaea45c88a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal"
Sep 17 17:33:45.383548 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.383491    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68edadcfa29cc8be7e22dcaea45c88a8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal\" (UID: \"68edadcfa29cc8be7e22dcaea45c88a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal"
Sep 17 17:33:45.383548 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.383519    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3771e4c220b5a223d0f484f25cdc29e8-config\") pod \"kube-apiserver-proxy-ip-10-0-142-4.ec2.internal\" (UID: \"3771e4c220b5a223d0f484f25cdc29e8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-4.ec2.internal"
Sep 17 17:33:45.383624 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.383573    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/68edadcfa29cc8be7e22dcaea45c88a8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal\" (UID: \"68edadcfa29cc8be7e22dcaea45c88a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal"
Sep 17 17:33:45.383624 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.383583    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3771e4c220b5a223d0f484f25cdc29e8-config\") pod \"kube-apiserver-proxy-ip-10-0-142-4.ec2.internal\" (UID: \"3771e4c220b5a223d0f484f25cdc29e8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-4.ec2.internal"
Sep 17 17:33:45.383624 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.383583    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68edadcfa29cc8be7e22dcaea45c88a8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal\" (UID: \"68edadcfa29cc8be7e22dcaea45c88a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal"
Sep 17 17:33:45.479759 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:45.479701    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-4.ec2.internal\" not found"
Sep 17 17:33:45.553170 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.553149    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal"
Sep 17 17:33:45.558148 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.558014    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-4.ec2.internal"
Sep 17 17:33:45.580633 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:45.580611    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-4.ec2.internal\" not found"
Sep 17 17:33:45.681045 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:45.681021    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-4.ec2.internal\" not found"
Sep 17 17:33:45.781487 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:45.781432    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-4.ec2.internal\" not found"
Sep 17 17:33:45.881905 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:45.881878    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-4.ec2.internal\" not found"
Sep 17 17:33:45.929352 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.929328    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 17:33:45.929707 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:45.929475    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:33:45.982737 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:45.982716    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-4.ec2.internal\" not found"
Sep 17 17:33:46.008179 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.008155    2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:33:46.025600 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.025552    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:28:45 +0000 UTC" deadline="2028-02-17 14:13:55.440616119 +0000 UTC"
Sep 17 17:33:46.025678 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.025599    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12428h40m9.415020595s"
Sep 17 17:33:46.055189 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:46.055007    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68edadcfa29cc8be7e22dcaea45c88a8.slice/crio-f25bdc8b17253fd91bb464d732a236ca5f8833a0af0de840f74fb06e5430fc1c WatchSource:0}: Error finding container f25bdc8b17253fd91bb464d732a236ca5f8833a0af0de840f74fb06e5430fc1c: Status 404 returned error can't find the container with id f25bdc8b17253fd91bb464d732a236ca5f8833a0af0de840f74fb06e5430fc1c
Sep 17 17:33:46.064021 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.064003    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:33:46.069736 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:46.069716    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3771e4c220b5a223d0f484f25cdc29e8.slice/crio-0dde6ac3809c8011f2cb42ad94bffdeb3fcd19c81312d4b86d41a0271db05415 WatchSource:0}: Error finding container 0dde6ac3809c8011f2cb42ad94bffdeb3fcd19c81312d4b86d41a0271db05415: Status 404 returned error can't find the container with id 0dde6ac3809c8011f2cb42ad94bffdeb3fcd19c81312d4b86d41a0271db05415
Sep 17 17:33:46.083756 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:46.083731    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-4.ec2.internal\" not found"
Sep 17 17:33:46.094566 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.094544    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:46.095476 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.095458    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:46.095554 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.095493    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:46.095554 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.095504    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:46.112810 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.112769    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-4.ec2.internal" event={"ID":"3771e4c220b5a223d0f484f25cdc29e8","Type":"ContainerStarted","Data":"0dde6ac3809c8011f2cb42ad94bffdeb3fcd19c81312d4b86d41a0271db05415"}
Sep 17 17:33:46.114047 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.114027    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal" event={"ID":"68edadcfa29cc8be7e22dcaea45c88a8","Type":"ContainerStarted","Data":"f25bdc8b17253fd91bb464d732a236ca5f8833a0af0de840f74fb06e5430fc1c"}
Sep 17 17:33:46.184186 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:46.184163    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-4.ec2.internal\" not found"
Sep 17 17:33:46.286769 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:46.286751    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-4.ec2.internal\" not found"
Sep 17 17:33:46.303621 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.303572    2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:33:46.381731 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.381709    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal"
Sep 17 17:33:46.396589 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.396573    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:33:46.398125 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.398109    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-4.ec2.internal"
Sep 17 17:33:46.408666 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.408651    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:33:46.534605 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.534576    2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:33:46.962770 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.962737    2574 apiserver.go:52] "Watching apiserver"
Sep 17 17:33:46.971196 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.971168    2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 17:33:46.972226 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.972192    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-jvdsb","openshift-multus/multus-96ncc","openshift-network-diagnostics/network-check-target-sfmkn","openshift-network-operator/iptables-alerter-8zxbn","kube-system/global-pull-secret-syncer-n8p54","openshift-image-registry/node-ca-cbnqr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal","openshift-multus/multus-additional-cni-plugins-qxvb8","openshift-multus/network-metrics-daemon-kcglc","openshift-ovn-kubernetes/ovnkube-node-c528h","kube-system/konnectivity-agent-89m26","kube-system/kube-apiserver-proxy-ip-10-0-142-4.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj","openshift-cluster-node-tuning-operator/tuned-snzd9"]
Sep 17 17:33:46.974949 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.974925    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cbnqr"
Sep 17 17:33:46.977486 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.977354    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-96ncc"
Sep 17 17:33:46.977591 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.977558    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 17:33:46.977880 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.977857    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 17:33:46.977880 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.977866    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z9cx6\""
Sep 17 17:33:46.978573 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.978550    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 17:33:46.979730 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.979708    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:33:46.979828 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:46.979793    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcglc" podUID="c08424bd-0295-4dcb-9c57-3d5a5a6ba21d"
Sep 17 17:33:46.980293 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.980169    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 17:33:46.980293 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.980215    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 17:33:46.980293 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.980232    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8ncd2\""
Sep 17 17:33:46.980496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.980310    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 17:33:46.980496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.980476    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 17:33:46.980605 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.980562    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 17:33:46.982512 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.982494    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qxvb8"
Sep 17 17:33:46.985273 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.984761    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:33:46.985273 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:46.984832    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8p54" podUID="c58d2a73-a185-4e79-bbba-0b547390e9c7"
Sep 17 17:33:46.985273 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.985092    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-btd7d\""
Sep 17 17:33:46.985273 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.985122    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 17:33:46.985539 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.985445    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 17:33:46.987228 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.987208    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jvdsb"
Sep 17 17:33:46.989926 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.989905    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8zxbn"
Sep 17 17:33:46.990509 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.990483    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l5bcg\""
Sep 17 17:33:46.990686 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.990650    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 17:33:46.990767 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.990727    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 17:33:46.991556 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.991510    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0774cdca-c8f7-4cae-9037-633a199e0ff3-system-cni-dir\") pod \"multus-additional-cni-plugins-qxvb8\" (UID: \"0774cdca-c8f7-4cae-9037-633a199e0ff3\") " pod="openshift-multus/multus-additional-cni-plugins-qxvb8"
Sep 17 17:33:46.991671 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.991580    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0774cdca-c8f7-4cae-9037-633a199e0ff3-cnibin\") pod \"multus-additional-cni-plugins-qxvb8\" (UID: \"0774cdca-c8f7-4cae-9037-633a199e0ff3\") " pod="openshift-multus/multus-additional-cni-plugins-qxvb8"
Sep 17 17:33:46.991671 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.991622    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0774cdca-c8f7-4cae-9037-633a199e0ff3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qxvb8\" (UID: \"0774cdca-c8f7-4cae-9037-633a199e0ff3\") " pod="openshift-multus/multus-additional-cni-plugins-qxvb8"
Sep 17 17:33:46.991671 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.991652    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a811ebbe-f6a5-4bac-a036-e3d2f439f394-host\") pod \"node-ca-cbnqr\" (UID: \"a811ebbe-f6a5-4bac-a036-e3d2f439f394\") " pod="openshift-image-registry/node-ca-cbnqr"
Sep 17 17:33:46.991839 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.991678    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a811ebbe-f6a5-4bac-a036-e3d2f439f394-serviceca\") pod \"node-ca-cbnqr\" (UID: \"a811ebbe-f6a5-4bac-a036-e3d2f439f394\") " pod="openshift-image-registry/node-ca-cbnqr"
Sep 17 17:33:46.991839 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.991706    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-hostroot\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:46.991839 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.991731    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-multus-conf-dir\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:46.991839 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.991760    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-host-var-lib-cni-bin\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:46.991839 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.991802    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs\") pod \"network-metrics-daemon-kcglc\" (UID: \"c08424bd-0295-4dcb-9c57-3d5a5a6ba21d\") " pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:33:46.992011 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.991837    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d5db\" (UniqueName: \"kubernetes.io/projected/a811ebbe-f6a5-4bac-a036-e3d2f439f394-kube-api-access-5d5db\") pod \"node-ca-cbnqr\" (UID: \"a811ebbe-f6a5-4bac-a036-e3d2f439f394\") " pod="openshift-image-registry/node-ca-cbnqr"
Sep 17 17:33:46.992011 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.991865    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-system-cni-dir\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:46.992011 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.991891    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-host-run-k8s-cni-cncf-io\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:46.992011 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.991922    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbn7j\" (UniqueName: \"kubernetes.io/projected/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-kube-api-access-fbn7j\") pod \"network-metrics-daemon-kcglc\" (UID: \"c08424bd-0295-4dcb-9c57-3d5a5a6ba21d\") " pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:33:46.992011 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.991947    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0774cdca-c8f7-4cae-9037-633a199e0ff3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qxvb8\" (UID: \"0774cdca-c8f7-4cae-9037-633a199e0ff3\") " pod="openshift-multus/multus-additional-cni-plugins-qxvb8"
Sep 17 17:33:46.992011 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.991971    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-multus-socket-dir-parent\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:46.992011 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.991998    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-host-var-lib-kubelet\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:46.992422 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.992024    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0774cdca-c8f7-4cae-9037-633a199e0ff3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qxvb8\" (UID: \"0774cdca-c8f7-4cae-9037-633a199e0ff3\") " pod="openshift-multus/multus-additional-cni-plugins-qxvb8"
Sep 17 17:33:46.992422 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.992053    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-cnibin\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:46.992422 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.992093    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whxpb\" (UniqueName: \"kubernetes.io/projected/0774cdca-c8f7-4cae-9037-633a199e0ff3-kube-api-access-whxpb\") pod \"multus-additional-cni-plugins-qxvb8\" (UID: \"0774cdca-c8f7-4cae-9037-633a199e0ff3\") " pod="openshift-multus/multus-additional-cni-plugins-qxvb8"
Sep 17 17:33:46.992422 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.992118    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-cni-binary-copy\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:46.992422 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.992141    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-host-run-netns\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:46.992422 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.992174    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-host-var-lib-cni-multus\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:46.992422 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.992215    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-multus-daemon-config\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:46.992422 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.992240    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-host-run-multus-certs\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:46.992422 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.992266    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-etc-kubernetes\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:46.992422 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.992293    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0774cdca-c8f7-4cae-9037-633a199e0ff3-cni-binary-copy\") pod \"multus-additional-cni-plugins-qxvb8\" (UID: \"0774cdca-c8f7-4cae-9037-633a199e0ff3\") " pod="openshift-multus/multus-additional-cni-plugins-qxvb8"
Sep 17 17:33:46.992422 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.992325    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-multus-cni-dir\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:46.992422 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.992351    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzmtz\" (UniqueName: \"kubernetes.io/projected/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-kube-api-access-lzmtz\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:46.992964 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.992525    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:33:46.992964 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:46.992608    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfmkn" podUID="6916046b-8c0d-4394-8786-14b9ee9a6bee"
Sep 17 17:33:46.993601 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.993583    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 17:33:46.993601 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.993581    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d986h\""
Sep 17 17:33:46.994334 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.994310    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:33:46.994441 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.994310    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 17:33:46.994989 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.994965    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:46.997038 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.997017    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 17:33:46.997389 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.997372    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 17:33:46.997791 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.997770    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-89m26"
Sep 17 17:33:47.000802 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.999286    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2t488\""
Sep 17 17:33:47.000802 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:46.999711    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 17:33:47.000802 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.000027    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 17:33:47.000802 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.000626    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 17:33:47.000802 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.000647    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 17:33:47.000802 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.000714    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 17:33:47.001330 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.001248    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hs7rn\""
Sep 17 17:33:47.001330 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.001326    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 17:33:47.002971 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.002677    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:33:47.004047 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.004018    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj"
Sep 17 17:33:47.006462 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.006429    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 17:33:47.006560 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.006497    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.006560 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.006519    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9gm2l\""
Sep 17 17:33:47.006679 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.006605    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 17:33:47.006957 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.006907    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 17:33:47.008886 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.008869    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:33:47.009198 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.009149    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9rm4q\""
Sep 17 17:33:47.009831 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.009652    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 17:33:47.035032 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.035008    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6lz4v"
Sep 17 17:33:47.043314 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.043258    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6lz4v"
Sep 17 17:33:47.082447 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.082426    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 17:33:47.093185 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093158    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-cni-binary-copy\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.093288 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093201    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-host-run-netns\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.093288 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093228    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-host-var-lib-cni-multus\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.093288 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093248    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-multus-daemon-config\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.093288 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093274    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-host-run-multus-certs\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.093551 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093302    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-etc-kubernetes\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.093551 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093333    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34ac981d-6735-4aa4-a6f2-73945a8e1c22-socket-dir\") pod \"aws-ebs-csi-driver-node-xhzjj\" (UID: \"34ac981d-6735-4aa4-a6f2-73945a8e1c22\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj"
Sep 17 17:33:47.093551 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093359    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7f97\" (UniqueName: \"kubernetes.io/projected/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-kube-api-access-d7f97\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.093551 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093391    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0774cdca-c8f7-4cae-9037-633a199e0ff3-cni-binary-copy\") pod \"multus-additional-cni-plugins-qxvb8\" (UID: \"0774cdca-c8f7-4cae-9037-633a199e0ff3\") " pod="openshift-multus/multus-additional-cni-plugins-qxvb8"
Sep 17 17:33:47.093551 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093408    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-host-var-lib-cni-multus\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.093551 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093415    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-multus-cni-dir\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.093551 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093433    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-etc-kubernetes\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.093551 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093454    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-host-run-multus-certs\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.093551 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093301    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-host-run-netns\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.093551 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093456    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9b120543-c1f5-49a0-9b46-8494758bc055-ovnkube-script-lib\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.093551 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093507    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34ac981d-6735-4aa4-a6f2-73945a8e1c22-sys-fs\") pod \"aws-ebs-csi-driver-node-xhzjj\" (UID: \"34ac981d-6735-4aa4-a6f2-73945a8e1c22\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj"
Sep 17 17:33:47.094028 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093559    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-multus-cni-dir\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.094028 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093572    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-etc-sysctl-d\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.094028 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093607    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-var-lib-kubelet\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.094028 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093640    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0774cdca-c8f7-4cae-9037-633a199e0ff3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qxvb8\" (UID: \"0774cdca-c8f7-4cae-9037-633a199e0ff3\") " pod="openshift-multus/multus-additional-cni-plugins-qxvb8"
Sep 17 17:33:47.094028 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093694    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-hostroot\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.094028 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093731    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c58d2a73-a185-4e79-bbba-0b547390e9c7-kubelet-config\") pod \"global-pull-secret-syncer-n8p54\" (UID: \"c58d2a73-a185-4e79-bbba-0b547390e9c7\") " pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:33:47.094028 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093765    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34ac981d-6735-4aa4-a6f2-73945a8e1c22-device-dir\") pod \"aws-ebs-csi-driver-node-xhzjj\" (UID: \"34ac981d-6735-4aa4-a6f2-73945a8e1c22\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj"
Sep 17 17:33:47.094028 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093798    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-hostroot\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.094028 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093802    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8146ff25-6272-4f4f-bc26-f165405e2610-konnectivity-ca\") pod \"konnectivity-agent-89m26\" (UID: \"8146ff25-6272-4f4f-bc26-f165405e2610\") " pod="kube-system/konnectivity-agent-89m26"
Sep 17 17:33:47.094028 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093856    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-run\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.094028 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093883    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-lib-modules\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.094028 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093888    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-cni-binary-copy\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.094028 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093920    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-host-var-lib-cni-bin\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.094028 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.093955    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kx2w\" (UniqueName: \"kubernetes.io/projected/b5dd23d0-8b89-453c-99ee-dfd8a18c683c-kube-api-access-6kx2w\") pod \"node-resolver-jvdsb\" (UID: \"b5dd23d0-8b89-453c-99ee-dfd8a18c683c\") " pod="openshift-dns/node-resolver-jvdsb"
Sep 17 17:33:47.094028 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094005    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-multus-daemon-config\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.094028 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094004    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-host-var-lib-cni-bin\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.094028 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094028    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-etc-sysconfig\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.094756 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094061    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-tmp\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.094756 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094064    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0774cdca-c8f7-4cae-9037-633a199e0ff3-cni-binary-copy\") pod \"multus-additional-cni-plugins-qxvb8\" (UID: \"0774cdca-c8f7-4cae-9037-633a199e0ff3\") " pod="openshift-multus/multus-additional-cni-plugins-qxvb8"
Sep 17 17:33:47.094756 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094101    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs\") pod \"network-metrics-daemon-kcglc\" (UID: \"c08424bd-0295-4dcb-9c57-3d5a5a6ba21d\") " pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:33:47.094756 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094138    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c58d2a73-a185-4e79-bbba-0b547390e9c7-dbus\") pod \"global-pull-secret-syncer-n8p54\" (UID: \"c58d2a73-a185-4e79-bbba-0b547390e9c7\") " pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:33:47.094756 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094168    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/709c4e71-19d2-48bf-b0ae-65cd818ba096-host-slash\") pod \"iptables-alerter-8zxbn\" (UID: \"709c4e71-19d2-48bf-b0ae-65cd818ba096\") " pod="openshift-network-operator/iptables-alerter-8zxbn"
Sep 17 17:33:47.094756 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094195    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0774cdca-c8f7-4cae-9037-633a199e0ff3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qxvb8\" (UID: \"0774cdca-c8f7-4cae-9037-633a199e0ff3\") " pod="openshift-multus/multus-additional-cni-plugins-qxvb8"
Sep 17 17:33:47.094756 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094203    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4p7g\" (UniqueName: \"kubernetes.io/projected/709c4e71-19d2-48bf-b0ae-65cd818ba096-kube-api-access-k4p7g\") pod \"iptables-alerter-8zxbn\" (UID: \"709c4e71-19d2-48bf-b0ae-65cd818ba096\") " pod="openshift-network-operator/iptables-alerter-8zxbn"
Sep 17 17:33:47.094756 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:47.094229    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:47.094756 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094253    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-systemd-units\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.094756 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094279    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-etc-sysctl-conf\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.094756 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:47.094309    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs podName:c08424bd-0295-4dcb-9c57-3d5a5a6ba21d nodeName:}" failed. No retries permitted until 2026-09-17 17:33:47.594287445 +0000 UTC m=+2.923387053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs") pod "network-metrics-daemon-kcglc" (UID: "c08424bd-0295-4dcb-9c57-3d5a5a6ba21d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:47.094756 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094345    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-host-var-lib-kubelet\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.094756 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094376    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b5dd23d0-8b89-453c-99ee-dfd8a18c683c-hosts-file\") pod \"node-resolver-jvdsb\" (UID: \"b5dd23d0-8b89-453c-99ee-dfd8a18c683c\") " pod="openshift-dns/node-resolver-jvdsb"
Sep 17 17:33:47.094756 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094405    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0774cdca-c8f7-4cae-9037-633a199e0ff3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qxvb8\" (UID: \"0774cdca-c8f7-4cae-9037-633a199e0ff3\") " pod="openshift-multus/multus-additional-cni-plugins-qxvb8"
Sep 17 17:33:47.094756 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094408    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-host-var-lib-kubelet\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.094756 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094434    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-var-lib-openvswitch\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.095506 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094460    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-etc-openvswitch\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.095506 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094484    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-run-ovn\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.095506 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094508    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-etc-modprobe-d\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.095506 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094534    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-sys\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.095506 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094564    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whxpb\" (UniqueName: \"kubernetes.io/projected/0774cdca-c8f7-4cae-9037-633a199e0ff3-kube-api-access-whxpb\") pod \"multus-additional-cni-plugins-qxvb8\" (UID: \"0774cdca-c8f7-4cae-9037-633a199e0ff3\") " pod="openshift-multus/multus-additional-cni-plugins-qxvb8"
Sep 17 17:33:47.095506 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094625    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5dd23d0-8b89-453c-99ee-dfd8a18c683c-tmp-dir\") pod \"node-resolver-jvdsb\" (UID: \"b5dd23d0-8b89-453c-99ee-dfd8a18c683c\") " pod="openshift-dns/node-resolver-jvdsb"
Sep 17 17:33:47.095506 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094650    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-node-log\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.095506 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094668    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-etc-tuned\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.095506 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094697    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzmtz\" (UniqueName: \"kubernetes.io/projected/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-kube-api-access-lzmtz\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.095506 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094721    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-host-cni-bin\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.095506 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094738    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.095506 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094760    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8146ff25-6272-4f4f-bc26-f165405e2610-agent-certs\") pod \"konnectivity-agent-89m26\" (UID: \"8146ff25-6272-4f4f-bc26-f165405e2610\") " pod="kube-system/konnectivity-agent-89m26"
Sep 17 17:33:47.095506 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094824    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0774cdca-c8f7-4cae-9037-633a199e0ff3-system-cni-dir\") pod \"multus-additional-cni-plugins-qxvb8\" (UID: \"0774cdca-c8f7-4cae-9037-633a199e0ff3\") " pod="openshift-multus/multus-additional-cni-plugins-qxvb8"
Sep 17 17:33:47.095506 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094912    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0774cdca-c8f7-4cae-9037-633a199e0ff3-cnibin\") pod \"multus-additional-cni-plugins-qxvb8\" (UID: \"0774cdca-c8f7-4cae-9037-633a199e0ff3\") " pod="openshift-multus/multus-additional-cni-plugins-qxvb8"
Sep 17 17:33:47.095506 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094942    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a811ebbe-f6a5-4bac-a036-e3d2f439f394-host\") pod \"node-ca-cbnqr\" (UID: \"a811ebbe-f6a5-4bac-a036-e3d2f439f394\") " pod="openshift-image-registry/node-ca-cbnqr"
Sep 17 17:33:47.095506 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094952    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0774cdca-c8f7-4cae-9037-633a199e0ff3-cnibin\") pod \"multus-additional-cni-plugins-qxvb8\" (UID: \"0774cdca-c8f7-4cae-9037-633a199e0ff3\") " pod="openshift-multus/multus-additional-cni-plugins-qxvb8"
Sep 17 17:33:47.095506 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094977    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a811ebbe-f6a5-4bac-a036-e3d2f439f394-serviceca\") pod \"node-ca-cbnqr\" (UID: \"a811ebbe-f6a5-4bac-a036-e3d2f439f394\") " pod="openshift-image-registry/node-ca-cbnqr"
Sep 17 17:33:47.096300 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094912    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0774cdca-c8f7-4cae-9037-633a199e0ff3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qxvb8\" (UID: \"0774cdca-c8f7-4cae-9037-633a199e0ff3\") " pod="openshift-multus/multus-additional-cni-plugins-qxvb8"
Sep 17 17:33:47.096300 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095002    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-multus-conf-dir\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.096300 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095008    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a811ebbe-f6a5-4bac-a036-e3d2f439f394-host\") pod \"node-ca-cbnqr\" (UID: \"a811ebbe-f6a5-4bac-a036-e3d2f439f394\") " pod="openshift-image-registry/node-ca-cbnqr"
Sep 17 17:33:47.096300 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.094914    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0774cdca-c8f7-4cae-9037-633a199e0ff3-system-cni-dir\") pod \"multus-additional-cni-plugins-qxvb8\" (UID: \"0774cdca-c8f7-4cae-9037-633a199e0ff3\") " pod="openshift-multus/multus-additional-cni-plugins-qxvb8"
Sep 17 17:33:47.096300 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095040    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c58d2a73-a185-4e79-bbba-0b547390e9c7-original-pull-secret\") pod \"global-pull-secret-syncer-n8p54\" (UID: \"c58d2a73-a185-4e79-bbba-0b547390e9c7\") " pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:33:47.096300 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095045    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-multus-conf-dir\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.096300 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095083    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-host\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.096300 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095114    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d5db\" (UniqueName: \"kubernetes.io/projected/a811ebbe-f6a5-4bac-a036-e3d2f439f394-kube-api-access-5d5db\") pod \"node-ca-cbnqr\" (UID: \"a811ebbe-f6a5-4bac-a036-e3d2f439f394\") " pod="openshift-image-registry/node-ca-cbnqr"
Sep 17 17:33:47.096300 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095140    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-system-cni-dir\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.096300 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095239    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-run-systemd\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.096300 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095271    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-host-cni-netd\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.096300 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095273    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-system-cni-dir\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.096300 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095305    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-etc-kubernetes\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.096300 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095335    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-host-run-k8s-cni-cncf-io\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.096300 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095360    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-host-slash\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.096300 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095385    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-host-run-netns\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.096300 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095410    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-host-run-ovn-kubernetes\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.096924 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095434    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9b120543-c1f5-49a0-9b46-8494758bc055-ovnkube-config\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.096924 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095438    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a811ebbe-f6a5-4bac-a036-e3d2f439f394-serviceca\") pod \"node-ca-cbnqr\" (UID: \"a811ebbe-f6a5-4bac-a036-e3d2f439f394\") " pod="openshift-image-registry/node-ca-cbnqr"
Sep 17 17:33:47.096924 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095486    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cx5p\" (UniqueName: \"kubernetes.io/projected/9b120543-c1f5-49a0-9b46-8494758bc055-kube-api-access-8cx5p\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.096924 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095530    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-host-run-k8s-cni-cncf-io\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.096924 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095532    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34ac981d-6735-4aa4-a6f2-73945a8e1c22-registration-dir\") pod \"aws-ebs-csi-driver-node-xhzjj\" (UID: \"34ac981d-6735-4aa4-a6f2-73945a8e1c22\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj"
Sep 17 17:33:47.096924 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095568    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-etc-systemd\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.096924 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095605    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbn7j\" (UniqueName: \"kubernetes.io/projected/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-kube-api-access-fbn7j\") pod \"network-metrics-daemon-kcglc\" (UID: \"c08424bd-0295-4dcb-9c57-3d5a5a6ba21d\") " pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:33:47.096924 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095627    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0774cdca-c8f7-4cae-9037-633a199e0ff3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qxvb8\" (UID: \"0774cdca-c8f7-4cae-9037-633a199e0ff3\") " pod="openshift-multus/multus-additional-cni-plugins-qxvb8"
Sep 17 17:33:47.096924 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095654    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-multus-socket-dir-parent\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.096924 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095680    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b120543-c1f5-49a0-9b46-8494758bc055-ovn-node-metrics-cert\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.096924 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095706    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34ac981d-6735-4aa4-a6f2-73945a8e1c22-etc-selinux\") pod \"aws-ebs-csi-driver-node-xhzjj\" (UID: \"34ac981d-6735-4aa4-a6f2-73945a8e1c22\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj"
Sep 17 17:33:47.096924 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095730    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss2vl\" (UniqueName: \"kubernetes.io/projected/6916046b-8c0d-4394-8786-14b9ee9a6bee-kube-api-access-ss2vl\") pod \"network-check-target-sfmkn\" (UID: \"6916046b-8c0d-4394-8786-14b9ee9a6bee\") " pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:33:47.096924 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095755    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-log-socket\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.096924 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095793    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-multus-socket-dir-parent\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.096924 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095796    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-cnibin\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.096924 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095829    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0774cdca-c8f7-4cae-9037-633a199e0ff3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qxvb8\" (UID: \"0774cdca-c8f7-4cae-9037-633a199e0ff3\") " pod="openshift-multus/multus-additional-cni-plugins-qxvb8"
Sep 17 17:33:47.096924 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095846    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-cnibin\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.097500 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095846    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/709c4e71-19d2-48bf-b0ae-65cd818ba096-iptables-alerter-script\") pod \"iptables-alerter-8zxbn\" (UID: \"709c4e71-19d2-48bf-b0ae-65cd818ba096\") " pod="openshift-network-operator/iptables-alerter-8zxbn"
Sep 17 17:33:47.097500 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095893    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-host-kubelet\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.097500 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095920    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-run-openvswitch\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.097500 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095965    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9b120543-c1f5-49a0-9b46-8494758bc055-env-overrides\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.097500 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.095994    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34ac981d-6735-4aa4-a6f2-73945a8e1c22-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xhzjj\" (UID: \"34ac981d-6735-4aa4-a6f2-73945a8e1c22\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj"
Sep 17 17:33:47.097500 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.096036    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9smf\" (UniqueName: \"kubernetes.io/projected/34ac981d-6735-4aa4-a6f2-73945a8e1c22-kube-api-access-v9smf\") pod \"aws-ebs-csi-driver-node-xhzjj\" (UID: \"34ac981d-6735-4aa4-a6f2-73945a8e1c22\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj"
Sep 17 17:33:47.103847 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.103821    2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 17:33:47.108638 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.108619    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whxpb\" (UniqueName: \"kubernetes.io/projected/0774cdca-c8f7-4cae-9037-633a199e0ff3-kube-api-access-whxpb\") pod \"multus-additional-cni-plugins-qxvb8\" (UID: \"0774cdca-c8f7-4cae-9037-633a199e0ff3\") " pod="openshift-multus/multus-additional-cni-plugins-qxvb8"
Sep 17 17:33:47.108742 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.108673    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzmtz\" (UniqueName: \"kubernetes.io/projected/479fedb1-cbab-419a-a1b0-9c2a4c5eda27-kube-api-access-lzmtz\") pod \"multus-96ncc\" (UID: \"479fedb1-cbab-419a-a1b0-9c2a4c5eda27\") " pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.108742 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.108675    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d5db\" (UniqueName: \"kubernetes.io/projected/a811ebbe-f6a5-4bac-a036-e3d2f439f394-kube-api-access-5d5db\") pod \"node-ca-cbnqr\" (UID: \"a811ebbe-f6a5-4bac-a036-e3d2f439f394\") " pod="openshift-image-registry/node-ca-cbnqr"
Sep 17 17:33:47.108820 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.108756    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbn7j\" (UniqueName: \"kubernetes.io/projected/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-kube-api-access-fbn7j\") pod \"network-metrics-daemon-kcglc\" (UID: \"c08424bd-0295-4dcb-9c57-3d5a5a6ba21d\") " pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:33:47.197109 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197051    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b120543-c1f5-49a0-9b46-8494758bc055-ovn-node-metrics-cert\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.197109 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197105    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34ac981d-6735-4aa4-a6f2-73945a8e1c22-etc-selinux\") pod \"aws-ebs-csi-driver-node-xhzjj\" (UID: \"34ac981d-6735-4aa4-a6f2-73945a8e1c22\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj"
Sep 17 17:33:47.197390 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197123    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss2vl\" (UniqueName: \"kubernetes.io/projected/6916046b-8c0d-4394-8786-14b9ee9a6bee-kube-api-access-ss2vl\") pod \"network-check-target-sfmkn\" (UID: \"6916046b-8c0d-4394-8786-14b9ee9a6bee\") " pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:33:47.197390 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197144    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-log-socket\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.197390 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197172    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/709c4e71-19d2-48bf-b0ae-65cd818ba096-iptables-alerter-script\") pod \"iptables-alerter-8zxbn\" (UID: \"709c4e71-19d2-48bf-b0ae-65cd818ba096\") " pod="openshift-network-operator/iptables-alerter-8zxbn"
Sep 17 17:33:47.197390 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197191    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-host-kubelet\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.197390 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197217    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-run-openvswitch\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.197390 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197245    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9b120543-c1f5-49a0-9b46-8494758bc055-env-overrides\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.197390 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197259    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-log-socket\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.197390 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197274    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34ac981d-6735-4aa4-a6f2-73945a8e1c22-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xhzjj\" (UID: \"34ac981d-6735-4aa4-a6f2-73945a8e1c22\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj"
Sep 17 17:33:47.197390 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197265    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34ac981d-6735-4aa4-a6f2-73945a8e1c22-etc-selinux\") pod \"aws-ebs-csi-driver-node-xhzjj\" (UID: \"34ac981d-6735-4aa4-a6f2-73945a8e1c22\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj"
Sep 17 17:33:47.197390 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197305    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9smf\" (UniqueName: \"kubernetes.io/projected/34ac981d-6735-4aa4-a6f2-73945a8e1c22-kube-api-access-v9smf\") pod \"aws-ebs-csi-driver-node-xhzjj\" (UID: \"34ac981d-6735-4aa4-a6f2-73945a8e1c22\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj"
Sep 17 17:33:47.197390 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197374    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-run-openvswitch\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.197390 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197383    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-host-kubelet\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.198010 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197445    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34ac981d-6735-4aa4-a6f2-73945a8e1c22-socket-dir\") pod \"aws-ebs-csi-driver-node-xhzjj\" (UID: \"34ac981d-6735-4aa4-a6f2-73945a8e1c22\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj"
Sep 17 17:33:47.198010 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197495    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7f97\" (UniqueName: \"kubernetes.io/projected/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-kube-api-access-d7f97\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.198010 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197524    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9b120543-c1f5-49a0-9b46-8494758bc055-ovnkube-script-lib\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.198010 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197552    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34ac981d-6735-4aa4-a6f2-73945a8e1c22-sys-fs\") pod \"aws-ebs-csi-driver-node-xhzjj\" (UID: \"34ac981d-6735-4aa4-a6f2-73945a8e1c22\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj"
Sep 17 17:33:47.198010 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197464    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34ac981d-6735-4aa4-a6f2-73945a8e1c22-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xhzjj\" (UID: \"34ac981d-6735-4aa4-a6f2-73945a8e1c22\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj"
Sep 17 17:33:47.198010 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197633    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34ac981d-6735-4aa4-a6f2-73945a8e1c22-socket-dir\") pod \"aws-ebs-csi-driver-node-xhzjj\" (UID: \"34ac981d-6735-4aa4-a6f2-73945a8e1c22\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj"
Sep 17 17:33:47.198010 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197711    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34ac981d-6735-4aa4-a6f2-73945a8e1c22-sys-fs\") pod \"aws-ebs-csi-driver-node-xhzjj\" (UID: \"34ac981d-6735-4aa4-a6f2-73945a8e1c22\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj"
Sep 17 17:33:47.198010 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197805    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-etc-sysctl-d\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.198010 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197838    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-var-lib-kubelet\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.198010 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197885    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c58d2a73-a185-4e79-bbba-0b547390e9c7-kubelet-config\") pod \"global-pull-secret-syncer-n8p54\" (UID: \"c58d2a73-a185-4e79-bbba-0b547390e9c7\") " pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:33:47.198010 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197917    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-etc-sysctl-d\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.198010 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197970    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-var-lib-kubelet\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.198010 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197986    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9b120543-c1f5-49a0-9b46-8494758bc055-env-overrides\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.198010 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198012    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34ac981d-6735-4aa4-a6f2-73945a8e1c22-device-dir\") pod \"aws-ebs-csi-driver-node-xhzjj\" (UID: \"34ac981d-6735-4aa4-a6f2-73945a8e1c22\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj"
Sep 17 17:33:47.198737 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.197985    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c58d2a73-a185-4e79-bbba-0b547390e9c7-kubelet-config\") pod \"global-pull-secret-syncer-n8p54\" (UID: \"c58d2a73-a185-4e79-bbba-0b547390e9c7\") " pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:33:47.198737 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198041    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8146ff25-6272-4f4f-bc26-f165405e2610-konnectivity-ca\") pod \"konnectivity-agent-89m26\" (UID: \"8146ff25-6272-4f4f-bc26-f165405e2610\") " pod="kube-system/konnectivity-agent-89m26"
Sep 17 17:33:47.198737 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198084    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-run\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.198737 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198094    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34ac981d-6735-4aa4-a6f2-73945a8e1c22-device-dir\") pod \"aws-ebs-csi-driver-node-xhzjj\" (UID: \"34ac981d-6735-4aa4-a6f2-73945a8e1c22\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj"
Sep 17 17:33:47.198737 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198109    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-lib-modules\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.198737 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198133    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kx2w\" (UniqueName: \"kubernetes.io/projected/b5dd23d0-8b89-453c-99ee-dfd8a18c683c-kube-api-access-6kx2w\") pod \"node-resolver-jvdsb\" (UID: \"b5dd23d0-8b89-453c-99ee-dfd8a18c683c\") " pod="openshift-dns/node-resolver-jvdsb"
Sep 17 17:33:47.198737 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198149    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-run\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.198737 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198157    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/709c4e71-19d2-48bf-b0ae-65cd818ba096-iptables-alerter-script\") pod \"iptables-alerter-8zxbn\" (UID: \"709c4e71-19d2-48bf-b0ae-65cd818ba096\") " pod="openshift-network-operator/iptables-alerter-8zxbn"
Sep 17 17:33:47.198737 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198151    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-etc-sysconfig\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.198737 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198205    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-etc-sysconfig\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.198737 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198223    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-tmp\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.198737 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198243    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9b120543-c1f5-49a0-9b46-8494758bc055-ovnkube-script-lib\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.198737 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198265    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c58d2a73-a185-4e79-bbba-0b547390e9c7-dbus\") pod \"global-pull-secret-syncer-n8p54\" (UID: \"c58d2a73-a185-4e79-bbba-0b547390e9c7\") " pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:33:47.198737 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198293    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/709c4e71-19d2-48bf-b0ae-65cd818ba096-host-slash\") pod \"iptables-alerter-8zxbn\" (UID: \"709c4e71-19d2-48bf-b0ae-65cd818ba096\") " pod="openshift-network-operator/iptables-alerter-8zxbn"
Sep 17 17:33:47.198737 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198321    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4p7g\" (UniqueName: \"kubernetes.io/projected/709c4e71-19d2-48bf-b0ae-65cd818ba096-kube-api-access-k4p7g\") pod \"iptables-alerter-8zxbn\" (UID: \"709c4e71-19d2-48bf-b0ae-65cd818ba096\") " pod="openshift-network-operator/iptables-alerter-8zxbn"
Sep 17 17:33:47.198737 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198339    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/709c4e71-19d2-48bf-b0ae-65cd818ba096-host-slash\") pod \"iptables-alerter-8zxbn\" (UID: \"709c4e71-19d2-48bf-b0ae-65cd818ba096\") " pod="openshift-network-operator/iptables-alerter-8zxbn"
Sep 17 17:33:47.198737 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198347    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-systemd-units\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.199660 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198373    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-etc-sysctl-conf\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.199660 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198383    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-lib-modules\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.199660 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198400    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b5dd23d0-8b89-453c-99ee-dfd8a18c683c-hosts-file\") pod \"node-resolver-jvdsb\" (UID: \"b5dd23d0-8b89-453c-99ee-dfd8a18c683c\") " pod="openshift-dns/node-resolver-jvdsb"
Sep 17 17:33:47.199660 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198427    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-var-lib-openvswitch\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.199660 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198444    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-systemd-units\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.199660 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198470    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-etc-openvswitch\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.199660 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198497    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-run-ovn\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.199660 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198508    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b5dd23d0-8b89-453c-99ee-dfd8a18c683c-hosts-file\") pod \"node-resolver-jvdsb\" (UID: \"b5dd23d0-8b89-453c-99ee-dfd8a18c683c\") " pod="openshift-dns/node-resolver-jvdsb"
Sep 17 17:33:47.199660 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198526    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-etc-modprobe-d\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.199660 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198551    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-sys\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.199660 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198498    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c58d2a73-a185-4e79-bbba-0b547390e9c7-dbus\") pod \"global-pull-secret-syncer-n8p54\" (UID: \"c58d2a73-a185-4e79-bbba-0b547390e9c7\") " pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:33:47.199660 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198580    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5dd23d0-8b89-453c-99ee-dfd8a18c683c-tmp-dir\") pod \"node-resolver-jvdsb\" (UID: \"b5dd23d0-8b89-453c-99ee-dfd8a18c683c\") " pod="openshift-dns/node-resolver-jvdsb"
Sep 17 17:33:47.199660 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198587    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-var-lib-openvswitch\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.199660 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198604    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-node-log\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.199660 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198635    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8146ff25-6272-4f4f-bc26-f165405e2610-konnectivity-ca\") pod \"konnectivity-agent-89m26\" (UID: \"8146ff25-6272-4f4f-bc26-f165405e2610\") " pod="kube-system/konnectivity-agent-89m26"
Sep 17 17:33:47.199660 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198650    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-node-log\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.199660 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198651    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-etc-tuned\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.199660 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198552    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-etc-openvswitch\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.200638 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198662    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-etc-sysctl-conf\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.200638 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198607    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-run-ovn\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.200638 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198699    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-etc-modprobe-d\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.200638 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198703    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-sys\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.200638 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198801    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-host-cni-bin\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.200638 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198836    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.200638 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198867    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8146ff25-6272-4f4f-bc26-f165405e2610-agent-certs\") pod \"konnectivity-agent-89m26\" (UID: \"8146ff25-6272-4f4f-bc26-f165405e2610\") " pod="kube-system/konnectivity-agent-89m26"
Sep 17 17:33:47.200638 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198883    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-host-cni-bin\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.200638 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198900    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c58d2a73-a185-4e79-bbba-0b547390e9c7-original-pull-secret\") pod \"global-pull-secret-syncer-n8p54\" (UID: \"c58d2a73-a185-4e79-bbba-0b547390e9c7\") " pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:33:47.200638 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198931    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-host\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.200638 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198902    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.200638 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198959    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-run-systemd\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.200638 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.198987    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-host-cni-netd\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.200638 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.199013    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-etc-kubernetes\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.200638 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.199018    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-host\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.200638 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.199039    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-host-slash\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.200638 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:47.199048    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:47.200638 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.199066    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-host-run-netns\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.201556 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.199083    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-host-cni-netd\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.201556 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.199111    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-host-run-ovn-kubernetes\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.201556 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.199129    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-run-systemd\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.201556 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.199139    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9b120543-c1f5-49a0-9b46-8494758bc055-ovnkube-config\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.201556 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.199168    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cx5p\" (UniqueName: \"kubernetes.io/projected/9b120543-c1f5-49a0-9b46-8494758bc055-kube-api-access-8cx5p\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.201556 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.199199    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34ac981d-6735-4aa4-a6f2-73945a8e1c22-registration-dir\") pod \"aws-ebs-csi-driver-node-xhzjj\" (UID: \"34ac981d-6735-4aa4-a6f2-73945a8e1c22\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj"
Sep 17 17:33:47.201556 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.199211    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-etc-kubernetes\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.201556 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.199229    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-etc-systemd\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.201556 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.199251    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-host-slash\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.201556 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:47.199304    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c58d2a73-a185-4e79-bbba-0b547390e9c7-original-pull-secret podName:c58d2a73-a185-4e79-bbba-0b547390e9c7 nodeName:}" failed. No retries permitted until 2026-09-17 17:33:47.699283872 +0000 UTC m=+3.028383475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c58d2a73-a185-4e79-bbba-0b547390e9c7-original-pull-secret") pod "global-pull-secret-syncer-n8p54" (UID: "c58d2a73-a185-4e79-bbba-0b547390e9c7") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:47.201556 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.199333    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34ac981d-6735-4aa4-a6f2-73945a8e1c22-registration-dir\") pod \"aws-ebs-csi-driver-node-xhzjj\" (UID: \"34ac981d-6735-4aa4-a6f2-73945a8e1c22\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj"
Sep 17 17:33:47.201556 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.199348    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-etc-systemd\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.201556 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.199170    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-host-run-netns\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.201556 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.199466    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b120543-c1f5-49a0-9b46-8494758bc055-host-run-ovn-kubernetes\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.201556 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.199521    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5dd23d0-8b89-453c-99ee-dfd8a18c683c-tmp-dir\") pod \"node-resolver-jvdsb\" (UID: \"b5dd23d0-8b89-453c-99ee-dfd8a18c683c\") " pod="openshift-dns/node-resolver-jvdsb"
Sep 17 17:33:47.201556 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.199750    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9b120543-c1f5-49a0-9b46-8494758bc055-ovnkube-config\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.201556 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.200930    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-tmp\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.202301 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.201235    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b120543-c1f5-49a0-9b46-8494758bc055-ovn-node-metrics-cert\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.202301 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.201792    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-etc-tuned\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.202301 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.201889    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8146ff25-6272-4f4f-bc26-f165405e2610-agent-certs\") pod \"konnectivity-agent-89m26\" (UID: \"8146ff25-6272-4f4f-bc26-f165405e2610\") " pod="kube-system/konnectivity-agent-89m26"
Sep 17 17:33:47.208827 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:47.208786    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:33:47.208827 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:47.208810    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:33:47.208827 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:47.208824    2574 projected.go:196] Error preparing data for projected volume kube-api-access-ss2vl for pod openshift-network-diagnostics/network-check-target-sfmkn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:47.208993 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:47.208884    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6916046b-8c0d-4394-8786-14b9ee9a6bee-kube-api-access-ss2vl podName:6916046b-8c0d-4394-8786-14b9ee9a6bee nodeName:}" failed. No retries permitted until 2026-09-17 17:33:47.708866484 +0000 UTC m=+3.037966130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ss2vl" (UniqueName: "kubernetes.io/projected/6916046b-8c0d-4394-8786-14b9ee9a6bee-kube-api-access-ss2vl") pod "network-check-target-sfmkn" (UID: "6916046b-8c0d-4394-8786-14b9ee9a6bee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:47.210831 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.210807    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4p7g\" (UniqueName: \"kubernetes.io/projected/709c4e71-19d2-48bf-b0ae-65cd818ba096-kube-api-access-k4p7g\") pod \"iptables-alerter-8zxbn\" (UID: \"709c4e71-19d2-48bf-b0ae-65cd818ba096\") " pod="openshift-network-operator/iptables-alerter-8zxbn"
Sep 17 17:33:47.210967 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.210943    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7f97\" (UniqueName: \"kubernetes.io/projected/5a8bf0a5-e82f-4706-828a-2f285f36b3f5-kube-api-access-d7f97\") pod \"tuned-snzd9\" (UID: \"5a8bf0a5-e82f-4706-828a-2f285f36b3f5\") " pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.211386 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.211319    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9smf\" (UniqueName: \"kubernetes.io/projected/34ac981d-6735-4aa4-a6f2-73945a8e1c22-kube-api-access-v9smf\") pod \"aws-ebs-csi-driver-node-xhzjj\" (UID: \"34ac981d-6735-4aa4-a6f2-73945a8e1c22\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj"
Sep 17 17:33:47.212904 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.212842    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kx2w\" (UniqueName: \"kubernetes.io/projected/b5dd23d0-8b89-453c-99ee-dfd8a18c683c-kube-api-access-6kx2w\") pod \"node-resolver-jvdsb\" (UID: \"b5dd23d0-8b89-453c-99ee-dfd8a18c683c\") " pod="openshift-dns/node-resolver-jvdsb"
Sep 17 17:33:47.212994 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.212966    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cx5p\" (UniqueName: \"kubernetes.io/projected/9b120543-c1f5-49a0-9b46-8494758bc055-kube-api-access-8cx5p\") pod \"ovnkube-node-c528h\" (UID: \"9b120543-c1f5-49a0-9b46-8494758bc055\") " pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.225575 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.225552    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:33:47.288157 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.288128    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cbnqr"
Sep 17 17:33:47.297130 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.297110    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-96ncc"
Sep 17 17:33:47.307093 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.307056    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qxvb8"
Sep 17 17:33:47.316123 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.316103    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jvdsb"
Sep 17 17:33:47.323016 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.322996    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8zxbn"
Sep 17 17:33:47.330093 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.330076    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:33:47.338064 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.338045    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-89m26"
Sep 17 17:33:47.345977 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.345954    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj"
Sep 17 17:33:47.351700 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.351683    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-snzd9"
Sep 17 17:33:47.602086 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.601989    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs\") pod \"network-metrics-daemon-kcglc\" (UID: \"c08424bd-0295-4dcb-9c57-3d5a5a6ba21d\") " pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:33:47.602242 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:47.602152    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:47.602242 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:47.602216    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs podName:c08424bd-0295-4dcb-9c57-3d5a5a6ba21d nodeName:}" failed. No retries permitted until 2026-09-17 17:33:48.602200895 +0000 UTC m=+3.931300498 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs") pod "network-metrics-daemon-kcglc" (UID: "c08424bd-0295-4dcb-9c57-3d5a5a6ba21d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:47.703170 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.703136    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c58d2a73-a185-4e79-bbba-0b547390e9c7-original-pull-secret\") pod \"global-pull-secret-syncer-n8p54\" (UID: \"c58d2a73-a185-4e79-bbba-0b547390e9c7\") " pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:33:47.703324 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:47.703251    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:47.703324 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:47.703316    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c58d2a73-a185-4e79-bbba-0b547390e9c7-original-pull-secret podName:c58d2a73-a185-4e79-bbba-0b547390e9c7 nodeName:}" failed. No retries permitted until 2026-09-17 17:33:48.703298982 +0000 UTC m=+4.032398585 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c58d2a73-a185-4e79-bbba-0b547390e9c7-original-pull-secret") pod "global-pull-secret-syncer-n8p54" (UID: "c58d2a73-a185-4e79-bbba-0b547390e9c7") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:47.804039 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:47.804009    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss2vl\" (UniqueName: \"kubernetes.io/projected/6916046b-8c0d-4394-8786-14b9ee9a6bee-kube-api-access-ss2vl\") pod \"network-check-target-sfmkn\" (UID: \"6916046b-8c0d-4394-8786-14b9ee9a6bee\") " pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:33:47.804220 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:47.804174    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:33:47.804220 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:47.804194    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:33:47.804220 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:47.804209    2574 projected.go:196] Error preparing data for projected volume kube-api-access-ss2vl for pod openshift-network-diagnostics/network-check-target-sfmkn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:47.804385 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:47.804278    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6916046b-8c0d-4394-8786-14b9ee9a6bee-kube-api-access-ss2vl podName:6916046b-8c0d-4394-8786-14b9ee9a6bee nodeName:}" failed. No retries permitted until 2026-09-17 17:33:48.804258068 +0000 UTC m=+4.133357673 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ss2vl" (UniqueName: "kubernetes.io/projected/6916046b-8c0d-4394-8786-14b9ee9a6bee-kube-api-access-ss2vl") pod "network-check-target-sfmkn" (UID: "6916046b-8c0d-4394-8786-14b9ee9a6bee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:48.046997 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:48.046914    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:28:47 +0000 UTC" deadline="2028-04-01 17:05:50.431682531 +0000 UTC"
Sep 17 17:33:48.046997 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:48.046947    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13487h32m2.38473927s"
Sep 17 17:33:48.071908 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:48.071774    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod479fedb1_cbab_419a_a1b0_9c2a4c5eda27.slice/crio-6af14222dea3b1d238c60e00b038254a6ec79eb10dadad81503ca4a53392ad62 WatchSource:0}: Error finding container 6af14222dea3b1d238c60e00b038254a6ec79eb10dadad81503ca4a53392ad62: Status 404 returned error can't find the container with id 6af14222dea3b1d238c60e00b038254a6ec79eb10dadad81503ca4a53392ad62
Sep 17 17:33:48.077519 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:48.077493    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda811ebbe_f6a5_4bac_a036_e3d2f439f394.slice/crio-0876f595e66d8f803e42f9d7d740cade7c76e6f2e8cf84a790c1ba6c7f7fd525 WatchSource:0}: Error finding container 0876f595e66d8f803e42f9d7d740cade7c76e6f2e8cf84a790c1ba6c7f7fd525: Status 404 returned error can't find the container with id 0876f595e66d8f803e42f9d7d740cade7c76e6f2e8cf84a790c1ba6c7f7fd525
Sep 17 17:33:48.079463 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:48.079436    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod709c4e71_19d2_48bf_b0ae_65cd818ba096.slice/crio-454979f2e39a56fbd570a0bdb3b0b3d64b3081a3f7029cd9b963f4f40f66bcef WatchSource:0}: Error finding container 454979f2e39a56fbd570a0bdb3b0b3d64b3081a3f7029cd9b963f4f40f66bcef: Status 404 returned error can't find the container with id 454979f2e39a56fbd570a0bdb3b0b3d64b3081a3f7029cd9b963f4f40f66bcef
Sep 17 17:33:48.080144 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:48.080118    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0774cdca_c8f7_4cae_9037_633a199e0ff3.slice/crio-2736a33331efe96e4f04b884754fa19ca584b91c402afd525ad74867ff46d5bc WatchSource:0}: Error finding container 2736a33331efe96e4f04b884754fa19ca584b91c402afd525ad74867ff46d5bc: Status 404 returned error can't find the container with id 2736a33331efe96e4f04b884754fa19ca584b91c402afd525ad74867ff46d5bc
Sep 17 17:33:48.081571 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:48.081542    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5dd23d0_8b89_453c_99ee_dfd8a18c683c.slice/crio-e7f931761f86687b7b375be48d4dee862fef5ba9edaf5b3fed71a6bd090271eb WatchSource:0}: Error finding container e7f931761f86687b7b375be48d4dee862fef5ba9edaf5b3fed71a6bd090271eb: Status 404 returned error can't find the container with id e7f931761f86687b7b375be48d4dee862fef5ba9edaf5b3fed71a6bd090271eb
Sep 17 17:33:48.083014 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:48.082625    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b120543_c1f5_49a0_9b46_8494758bc055.slice/crio-3ce5567eec091c585177b0faab4a02a56e7a3b251224d432ddf156798844297f WatchSource:0}: Error finding container 3ce5567eec091c585177b0faab4a02a56e7a3b251224d432ddf156798844297f: Status 404 returned error can't find the container with id 3ce5567eec091c585177b0faab4a02a56e7a3b251224d432ddf156798844297f
Sep 17 17:33:48.083986 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:48.083943    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ac981d_6735_4aa4_a6f2_73945a8e1c22.slice/crio-49d4e95fbeb79fc0c0828f1ef2c8d5b527be2c4b7f0b068a362ed354de175d04 WatchSource:0}: Error finding container 49d4e95fbeb79fc0c0828f1ef2c8d5b527be2c4b7f0b068a362ed354de175d04: Status 404 returned error can't find the container with id 49d4e95fbeb79fc0c0828f1ef2c8d5b527be2c4b7f0b068a362ed354de175d04
Sep 17 17:33:48.107247 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:48.107224    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8146ff25_6272_4f4f_bc26_f165405e2610.slice/crio-3f13ab8a6abc29850b2ca16da3dda22b1e18884cf54863defc368f9adb7183a0 WatchSource:0}: Error finding container 3f13ab8a6abc29850b2ca16da3dda22b1e18884cf54863defc368f9adb7183a0: Status 404 returned error can't find the container with id 3f13ab8a6abc29850b2ca16da3dda22b1e18884cf54863defc368f9adb7183a0
Sep 17 17:33:48.109068 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:33:48.109041    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8bf0a5_e82f_4706_828a_2f285f36b3f5.slice/crio-894a8075554f561cf822a9ce6058bd4a24c33f2335610d4c6f721c055f5dfe2a WatchSource:0}: Error finding container 894a8075554f561cf822a9ce6058bd4a24c33f2335610d4c6f721c055f5dfe2a: Status 404 returned error can't find the container with id 894a8075554f561cf822a9ce6058bd4a24c33f2335610d4c6f721c055f5dfe2a
Sep 17 17:33:48.110166 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:48.110146    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:33:48.111430 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:48.110279    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfmkn" podUID="6916046b-8c0d-4394-8786-14b9ee9a6bee"
Sep 17 17:33:48.121279 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:48.121244    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj" event={"ID":"34ac981d-6735-4aa4-a6f2-73945a8e1c22","Type":"ContainerStarted","Data":"49d4e95fbeb79fc0c0828f1ef2c8d5b527be2c4b7f0b068a362ed354de175d04"}
Sep 17 17:33:48.123824 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:48.123797    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qxvb8" event={"ID":"0774cdca-c8f7-4cae-9037-633a199e0ff3","Type":"ContainerStarted","Data":"2736a33331efe96e4f04b884754fa19ca584b91c402afd525ad74867ff46d5bc"}
Sep 17 17:33:48.125176 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:48.125143    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jvdsb" event={"ID":"b5dd23d0-8b89-453c-99ee-dfd8a18c683c","Type":"ContainerStarted","Data":"e7f931761f86687b7b375be48d4dee862fef5ba9edaf5b3fed71a6bd090271eb"}
Sep 17 17:33:48.126570 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:48.126547    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8zxbn" event={"ID":"709c4e71-19d2-48bf-b0ae-65cd818ba096","Type":"ContainerStarted","Data":"454979f2e39a56fbd570a0bdb3b0b3d64b3081a3f7029cd9b963f4f40f66bcef"}
Sep 17 17:33:48.127905 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:48.127849    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-snzd9" event={"ID":"5a8bf0a5-e82f-4706-828a-2f285f36b3f5","Type":"ContainerStarted","Data":"894a8075554f561cf822a9ce6058bd4a24c33f2335610d4c6f721c055f5dfe2a"}
Sep 17 17:33:48.129318 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:48.129280    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-89m26" event={"ID":"8146ff25-6272-4f4f-bc26-f165405e2610","Type":"ContainerStarted","Data":"3f13ab8a6abc29850b2ca16da3dda22b1e18884cf54863defc368f9adb7183a0"}
Sep 17 17:33:48.130680 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:48.130648    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c528h" event={"ID":"9b120543-c1f5-49a0-9b46-8494758bc055","Type":"ContainerStarted","Data":"3ce5567eec091c585177b0faab4a02a56e7a3b251224d432ddf156798844297f"}
Sep 17 17:33:48.132079 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:48.132043    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cbnqr" event={"ID":"a811ebbe-f6a5-4bac-a036-e3d2f439f394","Type":"ContainerStarted","Data":"0876f595e66d8f803e42f9d7d740cade7c76e6f2e8cf84a790c1ba6c7f7fd525"}
Sep 17 17:33:48.133474 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:48.133451    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-96ncc" event={"ID":"479fedb1-cbab-419a-a1b0-9c2a4c5eda27","Type":"ContainerStarted","Data":"6af14222dea3b1d238c60e00b038254a6ec79eb10dadad81503ca4a53392ad62"}
Sep 17 17:33:48.609597 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:48.609438    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs\") pod \"network-metrics-daemon-kcglc\" (UID: \"c08424bd-0295-4dcb-9c57-3d5a5a6ba21d\") " pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:33:48.609597 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:48.609590    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:48.609829 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:48.609667    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs podName:c08424bd-0295-4dcb-9c57-3d5a5a6ba21d nodeName:}" failed. No retries permitted until 2026-09-17 17:33:50.609644338 +0000 UTC m=+5.938743945 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs") pod "network-metrics-daemon-kcglc" (UID: "c08424bd-0295-4dcb-9c57-3d5a5a6ba21d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:48.710822 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:48.710691    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c58d2a73-a185-4e79-bbba-0b547390e9c7-original-pull-secret\") pod \"global-pull-secret-syncer-n8p54\" (UID: \"c58d2a73-a185-4e79-bbba-0b547390e9c7\") " pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:33:48.711046 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:48.710988    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:48.711151 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:48.711085    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c58d2a73-a185-4e79-bbba-0b547390e9c7-original-pull-secret podName:c58d2a73-a185-4e79-bbba-0b547390e9c7 nodeName:}" failed. No retries permitted until 2026-09-17 17:33:50.711048339 +0000 UTC m=+6.040147959 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c58d2a73-a185-4e79-bbba-0b547390e9c7-original-pull-secret") pod "global-pull-secret-syncer-n8p54" (UID: "c58d2a73-a185-4e79-bbba-0b547390e9c7") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:48.811842 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:48.811804    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss2vl\" (UniqueName: \"kubernetes.io/projected/6916046b-8c0d-4394-8786-14b9ee9a6bee-kube-api-access-ss2vl\") pod \"network-check-target-sfmkn\" (UID: \"6916046b-8c0d-4394-8786-14b9ee9a6bee\") " pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:33:48.812042 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:48.811977    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:33:48.812042 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:48.812003    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:33:48.812042 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:48.812016    2574 projected.go:196] Error preparing data for projected volume kube-api-access-ss2vl for pod openshift-network-diagnostics/network-check-target-sfmkn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:48.812291 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:48.812076    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6916046b-8c0d-4394-8786-14b9ee9a6bee-kube-api-access-ss2vl podName:6916046b-8c0d-4394-8786-14b9ee9a6bee nodeName:}" failed. No retries permitted until 2026-09-17 17:33:50.81205748 +0000 UTC m=+6.141157090 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ss2vl" (UniqueName: "kubernetes.io/projected/6916046b-8c0d-4394-8786-14b9ee9a6bee-kube-api-access-ss2vl") pod "network-check-target-sfmkn" (UID: "6916046b-8c0d-4394-8786-14b9ee9a6bee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:49.047271 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:49.047154    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:28:47 +0000 UTC" deadline="2028-03-04 15:05:30.04307904 +0000 UTC"
Sep 17 17:33:49.047271 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:49.047220    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12813h31m40.995863375s"
Sep 17 17:33:49.112952 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:49.112273    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:33:49.112952 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:49.112425    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcglc" podUID="c08424bd-0295-4dcb-9c57-3d5a5a6ba21d"
Sep 17 17:33:49.113455 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:49.113297    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:33:49.113455 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:49.113414    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8p54" podUID="c58d2a73-a185-4e79-bbba-0b547390e9c7"
Sep 17 17:33:49.155840 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:49.155809    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-4.ec2.internal" event={"ID":"3771e4c220b5a223d0f484f25cdc29e8","Type":"ContainerStarted","Data":"c59c2efcca0e469e6622b16dbb1439ff2220579fe79ff80ae9977df837757e96"}
Sep 17 17:33:49.171753 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:49.171724    2574 generic.go:356] "Generic (PLEG): container finished" podID="68edadcfa29cc8be7e22dcaea45c88a8" containerID="b653adc90821a81ce7848de64de231044ffd86ca233f1a2c61091e158b31b0c4" exitCode=0
Sep 17 17:33:49.171867 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:49.171776    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal" event={"ID":"68edadcfa29cc8be7e22dcaea45c88a8","Type":"ContainerDied","Data":"b653adc90821a81ce7848de64de231044ffd86ca233f1a2c61091e158b31b0c4"}
Sep 17 17:33:49.201174 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:49.201100    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-4.ec2.internal" podStartSLOduration=3.201080889 podStartE2EDuration="3.201080889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:33:49.180306994 +0000 UTC m=+4.509406614" watchObservedRunningTime="2026-09-17 17:33:49.201080889 +0000 UTC m=+4.530180516"
Sep 17 17:33:50.110868 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:50.110829    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:33:50.111416 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:50.110962    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfmkn" podUID="6916046b-8c0d-4394-8786-14b9ee9a6bee"
Sep 17 17:33:50.193665 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:50.193520    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal" event={"ID":"68edadcfa29cc8be7e22dcaea45c88a8","Type":"ContainerStarted","Data":"16fd3681808576589238c944c74d44ab76409a9b6d666554aeb18d264facada3"}
Sep 17 17:33:50.210087 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:50.209226    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-4.ec2.internal" podStartSLOduration=4.20920703 podStartE2EDuration="4.20920703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:33:50.208709998 +0000 UTC m=+5.537809626" watchObservedRunningTime="2026-09-17 17:33:50.20920703 +0000 UTC m=+5.538306661"
Sep 17 17:33:50.628962 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:50.628347    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs\") pod \"network-metrics-daemon-kcglc\" (UID: \"c08424bd-0295-4dcb-9c57-3d5a5a6ba21d\") " pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:33:50.628962 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:50.628535    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:50.628962 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:50.628602    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs podName:c08424bd-0295-4dcb-9c57-3d5a5a6ba21d nodeName:}" failed. No retries permitted until 2026-09-17 17:33:54.628583328 +0000 UTC m=+9.957682952 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs") pod "network-metrics-daemon-kcglc" (UID: "c08424bd-0295-4dcb-9c57-3d5a5a6ba21d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:50.734149 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:50.729917    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c58d2a73-a185-4e79-bbba-0b547390e9c7-original-pull-secret\") pod \"global-pull-secret-syncer-n8p54\" (UID: \"c58d2a73-a185-4e79-bbba-0b547390e9c7\") " pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:33:50.738090 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:50.734627    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:50.738090 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:50.734720    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c58d2a73-a185-4e79-bbba-0b547390e9c7-original-pull-secret podName:c58d2a73-a185-4e79-bbba-0b547390e9c7 nodeName:}" failed. No retries permitted until 2026-09-17 17:33:54.734698435 +0000 UTC m=+10.063798060 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c58d2a73-a185-4e79-bbba-0b547390e9c7-original-pull-secret") pod "global-pull-secret-syncer-n8p54" (UID: "c58d2a73-a185-4e79-bbba-0b547390e9c7") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:50.832545 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:50.830870    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss2vl\" (UniqueName: \"kubernetes.io/projected/6916046b-8c0d-4394-8786-14b9ee9a6bee-kube-api-access-ss2vl\") pod \"network-check-target-sfmkn\" (UID: \"6916046b-8c0d-4394-8786-14b9ee9a6bee\") " pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:33:50.832545 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:50.831051    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:33:50.832545 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:50.831078    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:33:50.832545 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:50.831109    2574 projected.go:196] Error preparing data for projected volume kube-api-access-ss2vl for pod openshift-network-diagnostics/network-check-target-sfmkn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:50.832545 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:50.831179    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6916046b-8c0d-4394-8786-14b9ee9a6bee-kube-api-access-ss2vl podName:6916046b-8c0d-4394-8786-14b9ee9a6bee nodeName:}" failed. No retries permitted until 2026-09-17 17:33:54.831157296 +0000 UTC m=+10.160256903 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ss2vl" (UniqueName: "kubernetes.io/projected/6916046b-8c0d-4394-8786-14b9ee9a6bee-kube-api-access-ss2vl") pod "network-check-target-sfmkn" (UID: "6916046b-8c0d-4394-8786-14b9ee9a6bee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:51.111433 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:51.111369    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:33:51.111904 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:51.111533    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcglc" podUID="c08424bd-0295-4dcb-9c57-3d5a5a6ba21d"
Sep 17 17:33:51.111904 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:51.111583    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:33:51.111904 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:51.111710    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8p54" podUID="c58d2a73-a185-4e79-bbba-0b547390e9c7"
Sep 17 17:33:52.110961 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:52.110462    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:33:52.110961 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:52.110588    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfmkn" podUID="6916046b-8c0d-4394-8786-14b9ee9a6bee"
Sep 17 17:33:53.115169 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:53.115138    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:33:53.115658 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:53.115137    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:33:53.115658 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:53.115289    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8p54" podUID="c58d2a73-a185-4e79-bbba-0b547390e9c7"
Sep 17 17:33:53.115658 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:53.115337    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcglc" podUID="c08424bd-0295-4dcb-9c57-3d5a5a6ba21d"
Sep 17 17:33:54.110910 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:54.110877    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:33:54.111122 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:54.111023    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfmkn" podUID="6916046b-8c0d-4394-8786-14b9ee9a6bee"
Sep 17 17:33:54.664936 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:54.664405    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs\") pod \"network-metrics-daemon-kcglc\" (UID: \"c08424bd-0295-4dcb-9c57-3d5a5a6ba21d\") " pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:33:54.664936 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:54.664545    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:54.664936 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:54.664614    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs podName:c08424bd-0295-4dcb-9c57-3d5a5a6ba21d nodeName:}" failed. No retries permitted until 2026-09-17 17:34:02.664595685 +0000 UTC m=+17.993695288 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs") pod "network-metrics-daemon-kcglc" (UID: "c08424bd-0295-4dcb-9c57-3d5a5a6ba21d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:54.765721 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:54.765679    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c58d2a73-a185-4e79-bbba-0b547390e9c7-original-pull-secret\") pod \"global-pull-secret-syncer-n8p54\" (UID: \"c58d2a73-a185-4e79-bbba-0b547390e9c7\") " pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:33:54.765974 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:54.765829    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:54.765974 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:54.765896    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c58d2a73-a185-4e79-bbba-0b547390e9c7-original-pull-secret podName:c58d2a73-a185-4e79-bbba-0b547390e9c7 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:02.765877359 +0000 UTC m=+18.094976976 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c58d2a73-a185-4e79-bbba-0b547390e9c7-original-pull-secret") pod "global-pull-secret-syncer-n8p54" (UID: "c58d2a73-a185-4e79-bbba-0b547390e9c7") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:54.866642 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:54.866546    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss2vl\" (UniqueName: \"kubernetes.io/projected/6916046b-8c0d-4394-8786-14b9ee9a6bee-kube-api-access-ss2vl\") pod \"network-check-target-sfmkn\" (UID: \"6916046b-8c0d-4394-8786-14b9ee9a6bee\") " pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:33:54.866813 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:54.866765    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:33:54.866813 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:54.866790    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:33:54.866813 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:54.866802    2574 projected.go:196] Error preparing data for projected volume kube-api-access-ss2vl for pod openshift-network-diagnostics/network-check-target-sfmkn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:54.866973 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:54.866863    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6916046b-8c0d-4394-8786-14b9ee9a6bee-kube-api-access-ss2vl podName:6916046b-8c0d-4394-8786-14b9ee9a6bee nodeName:}" failed. No retries permitted until 2026-09-17 17:34:02.866844222 +0000 UTC m=+18.195943838 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ss2vl" (UniqueName: "kubernetes.io/projected/6916046b-8c0d-4394-8786-14b9ee9a6bee-kube-api-access-ss2vl") pod "network-check-target-sfmkn" (UID: "6916046b-8c0d-4394-8786-14b9ee9a6bee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:55.114868 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:55.114681    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:33:55.114868 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:55.114792    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8p54" podUID="c58d2a73-a185-4e79-bbba-0b547390e9c7"
Sep 17 17:33:55.114868 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:55.114838    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:33:55.115178 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:55.114958    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcglc" podUID="c08424bd-0295-4dcb-9c57-3d5a5a6ba21d"
Sep 17 17:33:56.110832 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:56.110794    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:33:56.111326 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:56.110915    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfmkn" podUID="6916046b-8c0d-4394-8786-14b9ee9a6bee"
Sep 17 17:33:57.110853 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:57.110821    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:33:57.111326 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:57.110871    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:33:57.111326 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:57.110946    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcglc" podUID="c08424bd-0295-4dcb-9c57-3d5a5a6ba21d"
Sep 17 17:33:57.111326 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:57.111091    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8p54" podUID="c58d2a73-a185-4e79-bbba-0b547390e9c7"
Sep 17 17:33:58.110205 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:58.110176    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:33:58.110373 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:58.110292    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfmkn" podUID="6916046b-8c0d-4394-8786-14b9ee9a6bee"
Sep 17 17:33:59.110710 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:59.110675    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:33:59.111158 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:59.110814    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcglc" podUID="c08424bd-0295-4dcb-9c57-3d5a5a6ba21d"
Sep 17 17:33:59.111158 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:33:59.110866    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:33:59.111158 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:33:59.110972    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8p54" podUID="c58d2a73-a185-4e79-bbba-0b547390e9c7"
Sep 17 17:34:00.110409 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:00.110378    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:34:00.110583 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:00.110480    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfmkn" podUID="6916046b-8c0d-4394-8786-14b9ee9a6bee"
Sep 17 17:34:01.110125 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:01.110087    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:34:01.110125 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:01.110099    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:34:01.110668 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:01.110222    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcglc" podUID="c08424bd-0295-4dcb-9c57-3d5a5a6ba21d"
Sep 17 17:34:01.110668 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:01.110368    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8p54" podUID="c58d2a73-a185-4e79-bbba-0b547390e9c7"
Sep 17 17:34:02.111113 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:02.111078    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:34:02.111559 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:02.111178    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfmkn" podUID="6916046b-8c0d-4394-8786-14b9ee9a6bee"
Sep 17 17:34:02.720621 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:02.720587    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs\") pod \"network-metrics-daemon-kcglc\" (UID: \"c08424bd-0295-4dcb-9c57-3d5a5a6ba21d\") " pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:34:02.720814 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:02.720752    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:02.720875 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:02.720826    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs podName:c08424bd-0295-4dcb-9c57-3d5a5a6ba21d nodeName:}" failed. No retries permitted until 2026-09-17 17:34:18.72080564 +0000 UTC m=+34.049905242 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs") pod "network-metrics-daemon-kcglc" (UID: "c08424bd-0295-4dcb-9c57-3d5a5a6ba21d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:02.820882 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:02.820849    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c58d2a73-a185-4e79-bbba-0b547390e9c7-original-pull-secret\") pod \"global-pull-secret-syncer-n8p54\" (UID: \"c58d2a73-a185-4e79-bbba-0b547390e9c7\") " pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:34:02.821042 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:02.820997    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:02.821110 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:02.821082    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c58d2a73-a185-4e79-bbba-0b547390e9c7-original-pull-secret podName:c58d2a73-a185-4e79-bbba-0b547390e9c7 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:18.821062846 +0000 UTC m=+34.150162461 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c58d2a73-a185-4e79-bbba-0b547390e9c7-original-pull-secret") pod "global-pull-secret-syncer-n8p54" (UID: "c58d2a73-a185-4e79-bbba-0b547390e9c7") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:02.921761 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:02.921732    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss2vl\" (UniqueName: \"kubernetes.io/projected/6916046b-8c0d-4394-8786-14b9ee9a6bee-kube-api-access-ss2vl\") pod \"network-check-target-sfmkn\" (UID: \"6916046b-8c0d-4394-8786-14b9ee9a6bee\") " pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:34:02.921932 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:02.921917    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:02.921990 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:02.921937    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:02.921990 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:02.921947    2574 projected.go:196] Error preparing data for projected volume kube-api-access-ss2vl for pod openshift-network-diagnostics/network-check-target-sfmkn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:02.922091 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:02.922003    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6916046b-8c0d-4394-8786-14b9ee9a6bee-kube-api-access-ss2vl podName:6916046b-8c0d-4394-8786-14b9ee9a6bee nodeName:}" failed. No retries permitted until 2026-09-17 17:34:18.921984588 +0000 UTC m=+34.251084201 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ss2vl" (UniqueName: "kubernetes.io/projected/6916046b-8c0d-4394-8786-14b9ee9a6bee-kube-api-access-ss2vl") pod "network-check-target-sfmkn" (UID: "6916046b-8c0d-4394-8786-14b9ee9a6bee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:03.111151 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:03.111121    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:34:03.111552 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:03.111120    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:34:03.111552 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:03.111256    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8p54" podUID="c58d2a73-a185-4e79-bbba-0b547390e9c7"
Sep 17 17:34:03.111552 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:03.111320    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcglc" podUID="c08424bd-0295-4dcb-9c57-3d5a5a6ba21d"
Sep 17 17:34:04.110688 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:04.110656    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:34:04.110846 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:04.110760    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfmkn" podUID="6916046b-8c0d-4394-8786-14b9ee9a6bee"
Sep 17 17:34:05.111315 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:05.111279    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:34:05.111762 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:05.111407    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcglc" podUID="c08424bd-0295-4dcb-9c57-3d5a5a6ba21d"
Sep 17 17:34:05.111762 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:05.111463    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:34:05.111762 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:05.111592    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8p54" podUID="c58d2a73-a185-4e79-bbba-0b547390e9c7"
Sep 17 17:34:06.110956 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:06.110923    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:34:06.111118 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:06.111033    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfmkn" podUID="6916046b-8c0d-4394-8786-14b9ee9a6bee"
Sep 17 17:34:07.110761 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:07.110592    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:34:07.111453 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:07.110612    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:34:07.111453 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:07.110841    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcglc" podUID="c08424bd-0295-4dcb-9c57-3d5a5a6ba21d"
Sep 17 17:34:07.111453 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:07.110948    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8p54" podUID="c58d2a73-a185-4e79-bbba-0b547390e9c7"
Sep 17 17:34:07.230356 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:07.230086    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c528h" event={"ID":"9b120543-c1f5-49a0-9b46-8494758bc055","Type":"ContainerStarted","Data":"dc0b8fced6479cb40ad95d4707258a495049bea8d64538814d6879ab629ed139"}
Sep 17 17:34:07.230570 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:07.230368    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c528h" event={"ID":"9b120543-c1f5-49a0-9b46-8494758bc055","Type":"ContainerStarted","Data":"872d67116e5265a881ae8f66ba6735eab4e3b88df4b0a23a34ce4eefb1bdc7ab"}
Sep 17 17:34:07.230570 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:07.230417    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c528h" event={"ID":"9b120543-c1f5-49a0-9b46-8494758bc055","Type":"ContainerStarted","Data":"e84dc14fc347503937e1ca2832a0d311b2f1b933c3bf04888b5b00b70f009de7"}
Sep 17 17:34:07.230570 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:07.230468    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c528h" event={"ID":"9b120543-c1f5-49a0-9b46-8494758bc055","Type":"ContainerStarted","Data":"8bb960dcd6ed3a10a4bc566d872c682c3f981cc66e634bf51e50731c8cbfbb5e"}
Sep 17 17:34:07.230570 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:07.230516    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c528h" event={"ID":"9b120543-c1f5-49a0-9b46-8494758bc055","Type":"ContainerStarted","Data":"ea2ca59cbfa69e0642c7ae2d0f4aa5bef4b7e98965d060bd03756edfbdaef5d8"}
Sep 17 17:34:07.230570 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:07.230561    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c528h" event={"ID":"9b120543-c1f5-49a0-9b46-8494758bc055","Type":"ContainerStarted","Data":"d3d6411c0dbfe9691ea689ffce9ffaba17657949d71b3b3acc4efbc0f753c508"}
Sep 17 17:34:07.231878 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:07.231847    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cbnqr" event={"ID":"a811ebbe-f6a5-4bac-a036-e3d2f439f394","Type":"ContainerStarted","Data":"c766de4b7caa601ee4d7466c61fde8a97e8de17c3ee0363e558db6b524434e72"}
Sep 17 17:34:07.233661 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:07.233634    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-96ncc" event={"ID":"479fedb1-cbab-419a-a1b0-9c2a4c5eda27","Type":"ContainerStarted","Data":"70b66c9a371c399b70f4d5352f15171711e40d48c329b3da3dd250329c171dc9"}
Sep 17 17:34:07.235406 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:07.235381    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj" event={"ID":"34ac981d-6735-4aa4-a6f2-73945a8e1c22","Type":"ContainerStarted","Data":"b27e538541c0bf720ae11ff10e69c9a0d185e7b2986ab3e8ef551625bd5b117a"}
Sep 17 17:34:07.237206 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:07.237176    2574 generic.go:356] "Generic (PLEG): container finished" podID="0774cdca-c8f7-4cae-9037-633a199e0ff3" containerID="c660ea2bd1a9eae0ff997bf8c59fd1341605049dd97eda03056032a95b1537aa" exitCode=0
Sep 17 17:34:07.237308 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:07.237257    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qxvb8" event={"ID":"0774cdca-c8f7-4cae-9037-633a199e0ff3","Type":"ContainerDied","Data":"c660ea2bd1a9eae0ff997bf8c59fd1341605049dd97eda03056032a95b1537aa"}
Sep 17 17:34:07.238950 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:07.238926    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jvdsb" event={"ID":"b5dd23d0-8b89-453c-99ee-dfd8a18c683c","Type":"ContainerStarted","Data":"0abbace342c9a2c89d8055136fa5a7fbf0b5bf8e0c1d0752438d73ac695a2f0b"}
Sep 17 17:34:07.240675 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:07.240653    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-snzd9" event={"ID":"5a8bf0a5-e82f-4706-828a-2f285f36b3f5","Type":"ContainerStarted","Data":"4c13c528a6dfabb1051b3e75d74bc9a7be473c08b305370d9b1880b3d043b65d"}
Sep 17 17:34:07.242241 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:07.242221    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-89m26" event={"ID":"8146ff25-6272-4f4f-bc26-f165405e2610","Type":"ContainerStarted","Data":"5b66efb96aabc8768aab04900d80ad0da38d202a2b36b6c31f1eb840ea035952"}
Sep 17 17:34:07.247766 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:07.247707    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cbnqr" podStartSLOduration=4.016398877 podStartE2EDuration="22.247689664s" totalImagesPullingTime="18.231290787s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:07.247601064 +0000 UTC m=+22.576700689" watchObservedRunningTime="2026-09-17 17:34:07.247689664 +0000 UTC m=+22.576789291"
Sep 17 17:34:07.269453 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:07.268664    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-96ncc" podStartSLOduration=3.9149476659999998 podStartE2EDuration="22.26864052s" totalImagesPullingTime="18.353692854s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:07.267311211 +0000 UTC m=+22.596410836" watchObservedRunningTime="2026-09-17 17:34:07.26864052 +0000 UTC m=+22.597740147"
Sep 17 17:34:07.302048 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:07.302011    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jvdsb" podStartSLOduration=3.09740998 podStartE2EDuration="21.302000141s" totalImagesPullingTime="18.204590161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:07.301555215 +0000 UTC m=+22.630655024" watchObservedRunningTime="2026-09-17 17:34:07.302000141 +0000 UTC m=+22.631099764"
Sep 17 17:34:07.316492 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:07.316459    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-89m26" podStartSLOduration=10.54147877 podStartE2EDuration="21.316446437s" totalImagesPullingTime="10.774967667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:07.316366844 +0000 UTC m=+22.645466468" watchObservedRunningTime="2026-09-17 17:34:07.316446437 +0000 UTC m=+22.645546072"
Sep 17 17:34:07.337947 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:07.337910    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-snzd9" podStartSLOduration=3.129540115 podStartE2EDuration="21.337899374s" totalImagesPullingTime="18.208359259s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:07.337508605 +0000 UTC m=+22.666608232" watchObservedRunningTime="2026-09-17 17:34:07.337899374 +0000 UTC m=+22.666998996"
Sep 17 17:34:07.534742 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:07.534720    2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 17:34:08.040686 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:08.040574    2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:34:07.534736729Z","UUID":"da18ca95-073d-4b7d-b4fb-2990f4ec65e2","Handler":null,"Name":"","Endpoint":""}
Sep 17 17:34:08.042340 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:08.042316    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 17:34:08.042475 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:08.042352    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 17:34:08.110481 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:08.110457    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:34:08.110635 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:08.110580    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfmkn" podUID="6916046b-8c0d-4394-8786-14b9ee9a6bee"
Sep 17 17:34:08.247041 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:08.247004    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj" event={"ID":"34ac981d-6735-4aa4-a6f2-73945a8e1c22","Type":"ContainerStarted","Data":"e86cd8025354677dc995eb97e78ec8e8a371bb132122c05132243dd4b7eb5be0"}
Sep 17 17:34:08.249006 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:08.248965    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8zxbn" event={"ID":"709c4e71-19d2-48bf-b0ae-65cd818ba096","Type":"ContainerStarted","Data":"10bd48de29013d2eea6fd366427b0ede249ad1b543a9f9f6c69da7b6343a95ec"}
Sep 17 17:34:09.111016 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:09.110991    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:34:09.111163 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:09.111142    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcglc" podUID="c08424bd-0295-4dcb-9c57-3d5a5a6ba21d"
Sep 17 17:34:09.111223 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:09.111203    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:34:09.111339 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:09.111318    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8p54" podUID="c58d2a73-a185-4e79-bbba-0b547390e9c7"
Sep 17 17:34:09.256386 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:09.256353    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c528h" event={"ID":"9b120543-c1f5-49a0-9b46-8494758bc055","Type":"ContainerStarted","Data":"f3f701eba9844e9725dcc19c24d004173431c3d4ab828b3aecf528b4a3e4e81d"}
Sep 17 17:34:09.259123 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:09.259067    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj" event={"ID":"34ac981d-6735-4aa4-a6f2-73945a8e1c22","Type":"ContainerStarted","Data":"978b81c4c3764884e8e193c64a9a850558b2e0cc3371848fefef40b3faa58eba"}
Sep 17 17:34:09.284686 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:09.284623    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhzjj" podStartSLOduration=3.223330463 podStartE2EDuration="23.284607891s" totalImagesPullingTime="20.061277428s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:09.284308974 +0000 UTC m=+24.613408598" watchObservedRunningTime="2026-09-17 17:34:09.284607891 +0000 UTC m=+24.613707515"
Sep 17 17:34:09.284995 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:09.284965    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8zxbn" podStartSLOduration=5.055704425 podStartE2EDuration="23.284956907s" totalImagesPullingTime="18.229252482s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:08.266078934 +0000 UTC m=+23.595178551" watchObservedRunningTime="2026-09-17 17:34:09.284956907 +0000 UTC m=+24.614056581"
Sep 17 17:34:10.110637 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:10.110608    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:34:10.110805 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:10.110727    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfmkn" podUID="6916046b-8c0d-4394-8786-14b9ee9a6bee"
Sep 17 17:34:10.215982 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:10.215957    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-89m26"
Sep 17 17:34:10.216580 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:10.216558    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-89m26"
Sep 17 17:34:11.110075 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:11.110028    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:34:11.110075 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:11.110048    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:34:11.110536 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:11.110182    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcglc" podUID="c08424bd-0295-4dcb-9c57-3d5a5a6ba21d"
Sep 17 17:34:11.110536 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:11.110280    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8p54" podUID="c58d2a73-a185-4e79-bbba-0b547390e9c7"
Sep 17 17:34:12.111150 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:12.110969    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:34:12.111905 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:12.111236    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfmkn" podUID="6916046b-8c0d-4394-8786-14b9ee9a6bee"
Sep 17 17:34:12.267610 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:12.267578    2574 generic.go:356] "Generic (PLEG): container finished" podID="0774cdca-c8f7-4cae-9037-633a199e0ff3" containerID="1e4c20ee587135c4b085f0d08928a0cadfbab06318bbaafca7ce732beb163d3e" exitCode=0
Sep 17 17:34:12.267760 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:12.267661    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qxvb8" event={"ID":"0774cdca-c8f7-4cae-9037-633a199e0ff3","Type":"ContainerDied","Data":"1e4c20ee587135c4b085f0d08928a0cadfbab06318bbaafca7ce732beb163d3e"}
Sep 17 17:34:12.274589 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:12.274563    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c528h" event={"ID":"9b120543-c1f5-49a0-9b46-8494758bc055","Type":"ContainerStarted","Data":"6a2529bb86e15b8821f0ef9325c5761bc787746a1a0e211f1cd6a490aee3dcea"}
Sep 17 17:34:12.274838 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:12.274822    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:34:12.289380 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:12.289358    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:34:13.111039 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:13.110976    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:34:13.111166 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:13.111119    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcglc" podUID="c08424bd-0295-4dcb-9c57-3d5a5a6ba21d"
Sep 17 17:34:13.111426 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:13.111174    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:34:13.111426 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:13.111262    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8p54" podUID="c58d2a73-a185-4e79-bbba-0b547390e9c7"
Sep 17 17:34:13.271374 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:13.271315    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-c528h" podStartSLOduration=9.032290223 podStartE2EDuration="27.27129769s" totalImagesPullingTime="18.239007467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:12.33954797 +0000 UTC m=+27.668647595" watchObservedRunningTime="2026-09-17 17:34:13.27129769 +0000 UTC m=+28.600397329"
Sep 17 17:34:13.271555 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:13.271521    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n8p54"]
Sep 17 17:34:13.274602 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:13.274564    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sfmkn"]
Sep 17 17:34:13.274755 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:13.274710    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:34:13.274902 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:13.274827    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfmkn" podUID="6916046b-8c0d-4394-8786-14b9ee9a6bee"
Sep 17 17:34:13.275353 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:13.275328    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kcglc"]
Sep 17 17:34:13.279143 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:13.279123    2574 generic.go:356] "Generic (PLEG): container finished" podID="0774cdca-c8f7-4cae-9037-633a199e0ff3" containerID="81afe8f478f7f9458cfe46d53469267c5d3dc91a20bafaff1de4b3603c3a886c" exitCode=0
Sep 17 17:34:13.279255 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:13.279237    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qxvb8" event={"ID":"0774cdca-c8f7-4cae-9037-633a199e0ff3","Type":"ContainerDied","Data":"81afe8f478f7f9458cfe46d53469267c5d3dc91a20bafaff1de4b3603c3a886c"}
Sep 17 17:34:13.279792 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:13.279617    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:34:13.279792 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:13.279707    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8p54" podUID="c58d2a73-a185-4e79-bbba-0b547390e9c7"
Sep 17 17:34:13.279792 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:13.279722    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:34:13.279985 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:13.279797    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcglc" podUID="c08424bd-0295-4dcb-9c57-3d5a5a6ba21d"
Sep 17 17:34:13.279985 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:13.279736    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:34:13.279985 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:13.279874    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:34:13.295717 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:13.295701    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:34:13.679971 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:13.679936    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-89m26"
Sep 17 17:34:13.680172 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:13.680122    2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:34:13.680619 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:13.680596    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-89m26"
Sep 17 17:34:14.284042 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:14.283968    2574 generic.go:356] "Generic (PLEG): container finished" podID="0774cdca-c8f7-4cae-9037-633a199e0ff3" containerID="04c25980f306cedcbf4e1f66ca65c3950cd3678ce60ac7a4bb7a7bc9ca7d24b6" exitCode=0
Sep 17 17:34:14.284458 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:14.284046    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qxvb8" event={"ID":"0774cdca-c8f7-4cae-9037-633a199e0ff3","Type":"ContainerDied","Data":"04c25980f306cedcbf4e1f66ca65c3950cd3678ce60ac7a4bb7a7bc9ca7d24b6"}
Sep 17 17:34:15.111163 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:15.111131    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:34:15.111364 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:15.111238    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:34:15.111364 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:15.111244    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcglc" podUID="c08424bd-0295-4dcb-9c57-3d5a5a6ba21d"
Sep 17 17:34:15.111364 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:15.111271    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:34:15.111364 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:15.111344    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8p54" podUID="c58d2a73-a185-4e79-bbba-0b547390e9c7"
Sep 17 17:34:15.111564 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:15.111417    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfmkn" podUID="6916046b-8c0d-4394-8786-14b9ee9a6bee"
Sep 17 17:34:17.110548 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:17.110315    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:34:17.110985 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:17.110329    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:34:17.110985 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:17.110382    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:34:17.110985 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:17.110709    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8p54" podUID="c58d2a73-a185-4e79-bbba-0b547390e9c7"
Sep 17 17:34:17.110985 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:17.110800    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kcglc" podUID="c08424bd-0295-4dcb-9c57-3d5a5a6ba21d"
Sep 17 17:34:17.110985 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:17.110914    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sfmkn" podUID="6916046b-8c0d-4394-8786-14b9ee9a6bee"
Sep 17 17:34:18.570039 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:18.570012    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:18.681909 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:18.681849    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:18.736635 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:18.736605    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs\") pod \"network-metrics-daemon-kcglc\" (UID: \"c08424bd-0295-4dcb-9c57-3d5a5a6ba21d\") " pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:34:18.736761 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:18.736719    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:18.736820 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:18.736779    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs podName:c08424bd-0295-4dcb-9c57-3d5a5a6ba21d nodeName:}" failed. No retries permitted until 2026-09-17 17:34:50.736762343 +0000 UTC m=+66.065861954 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs") pod "network-metrics-daemon-kcglc" (UID: "c08424bd-0295-4dcb-9c57-3d5a5a6ba21d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:18.837196 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:18.837170    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c58d2a73-a185-4e79-bbba-0b547390e9c7-original-pull-secret\") pod \"global-pull-secret-syncer-n8p54\" (UID: \"c58d2a73-a185-4e79-bbba-0b547390e9c7\") " pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:34:18.837317 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:18.837275    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:18.837380 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:18.837334    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c58d2a73-a185-4e79-bbba-0b547390e9c7-original-pull-secret podName:c58d2a73-a185-4e79-bbba-0b547390e9c7 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:50.837318023 +0000 UTC m=+66.166417626 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c58d2a73-a185-4e79-bbba-0b547390e9c7-original-pull-secret") pod "global-pull-secret-syncer-n8p54" (UID: "c58d2a73-a185-4e79-bbba-0b547390e9c7") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:18.921970 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:18.921950    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-4.ec2.internal" event="NodeReady"
Sep 17 17:34:18.922124 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:18.922111    2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 17:34:18.938509 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:18.938489    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss2vl\" (UniqueName: \"kubernetes.io/projected/6916046b-8c0d-4394-8786-14b9ee9a6bee-kube-api-access-ss2vl\") pod \"network-check-target-sfmkn\" (UID: \"6916046b-8c0d-4394-8786-14b9ee9a6bee\") " pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:34:18.938682 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:18.938662    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:18.938731 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:18.938689    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:18.938731 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:18.938704    2574 projected.go:196] Error preparing data for projected volume kube-api-access-ss2vl for pod openshift-network-diagnostics/network-check-target-sfmkn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:18.938819 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:18.938763    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6916046b-8c0d-4394-8786-14b9ee9a6bee-kube-api-access-ss2vl podName:6916046b-8c0d-4394-8786-14b9ee9a6bee nodeName:}" failed. No retries permitted until 2026-09-17 17:34:50.938744681 +0000 UTC m=+66.267844302 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ss2vl" (UniqueName: "kubernetes.io/projected/6916046b-8c0d-4394-8786-14b9ee9a6bee-kube-api-access-ss2vl") pod "network-check-target-sfmkn" (UID: "6916046b-8c0d-4394-8786-14b9ee9a6bee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:18.986347 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:18.986326    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qpdzb"]
Sep 17 17:34:18.998396 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:18.998379    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lqq6s"]
Sep 17 17:34:18.998517 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:18.998500    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qpdzb"
Sep 17 17:34:19.001308 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.001290    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vdp8v\""
Sep 17 17:34:19.001397 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.001312    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 17:34:19.001397 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.001321    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 17:34:19.009606 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.009583    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lqq6s"]
Sep 17 17:34:19.009716 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.009705    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lqq6s"
Sep 17 17:34:19.013233 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.011495    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qpdzb"]
Sep 17 17:34:19.013233 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.012289    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 17:34:19.014021 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.013576    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8mqcm\""
Sep 17 17:34:19.014021 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.013590    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 17:34:19.014194 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.014143    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 17:34:19.111264 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.111240    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:34:19.111358 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.111240    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:34:19.111609 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.111249    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:34:19.114169 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.114123    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-745tx\""
Sep 17 17:34:19.114410 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.114280    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:34:19.114560 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.114527    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:34:19.114655 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.114564    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:34:19.114655 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.114639    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7zxgv\""
Sep 17 17:34:19.114962 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.114946    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:34:19.140326 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.140306    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7656e584-bfe6-4f58-bdac-fe20192d5a44-config-volume\") pod \"dns-default-qpdzb\" (UID: \"7656e584-bfe6-4f58-bdac-fe20192d5a44\") " pod="openshift-dns/dns-default-qpdzb"
Sep 17 17:34:19.140426 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.140346    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls\") pod \"dns-default-qpdzb\" (UID: \"7656e584-bfe6-4f58-bdac-fe20192d5a44\") " pod="openshift-dns/dns-default-qpdzb"
Sep 17 17:34:19.140426 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.140366    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac78b330-46cb-49bd-970a-f7eaf7c4afb4-cert\") pod \"ingress-canary-lqq6s\" (UID: \"ac78b330-46cb-49bd-970a-f7eaf7c4afb4\") " pod="openshift-ingress-canary/ingress-canary-lqq6s"
Sep 17 17:34:19.140426 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.140380    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7656e584-bfe6-4f58-bdac-fe20192d5a44-tmp-dir\") pod \"dns-default-qpdzb\" (UID: \"7656e584-bfe6-4f58-bdac-fe20192d5a44\") " pod="openshift-dns/dns-default-qpdzb"
Sep 17 17:34:19.140426 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.140414    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwb6c\" (UniqueName: \"kubernetes.io/projected/7656e584-bfe6-4f58-bdac-fe20192d5a44-kube-api-access-nwb6c\") pod \"dns-default-qpdzb\" (UID: \"7656e584-bfe6-4f58-bdac-fe20192d5a44\") " pod="openshift-dns/dns-default-qpdzb"
Sep 17 17:34:19.140569 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.140552    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xvm9\" (UniqueName: \"kubernetes.io/projected/ac78b330-46cb-49bd-970a-f7eaf7c4afb4-kube-api-access-4xvm9\") pod \"ingress-canary-lqq6s\" (UID: \"ac78b330-46cb-49bd-970a-f7eaf7c4afb4\") " pod="openshift-ingress-canary/ingress-canary-lqq6s"
Sep 17 17:34:19.241755 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.241736    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7656e584-bfe6-4f58-bdac-fe20192d5a44-config-volume\") pod \"dns-default-qpdzb\" (UID: \"7656e584-bfe6-4f58-bdac-fe20192d5a44\") " pod="openshift-dns/dns-default-qpdzb"
Sep 17 17:34:19.241841 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.241774    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls\") pod \"dns-default-qpdzb\" (UID: \"7656e584-bfe6-4f58-bdac-fe20192d5a44\") " pod="openshift-dns/dns-default-qpdzb"
Sep 17 17:34:19.241841 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.241795    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac78b330-46cb-49bd-970a-f7eaf7c4afb4-cert\") pod \"ingress-canary-lqq6s\" (UID: \"ac78b330-46cb-49bd-970a-f7eaf7c4afb4\") " pod="openshift-ingress-canary/ingress-canary-lqq6s"
Sep 17 17:34:19.241911 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.241833    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7656e584-bfe6-4f58-bdac-fe20192d5a44-tmp-dir\") pod \"dns-default-qpdzb\" (UID: \"7656e584-bfe6-4f58-bdac-fe20192d5a44\") " pod="openshift-dns/dns-default-qpdzb"
Sep 17 17:34:19.241911 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.241872    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwb6c\" (UniqueName: \"kubernetes.io/projected/7656e584-bfe6-4f58-bdac-fe20192d5a44-kube-api-access-nwb6c\") pod \"dns-default-qpdzb\" (UID: \"7656e584-bfe6-4f58-bdac-fe20192d5a44\") " pod="openshift-dns/dns-default-qpdzb"
Sep 17 17:34:19.241911 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:19.241904    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:19.242045 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.241927    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xvm9\" (UniqueName: \"kubernetes.io/projected/ac78b330-46cb-49bd-970a-f7eaf7c4afb4-kube-api-access-4xvm9\") pod \"ingress-canary-lqq6s\" (UID: \"ac78b330-46cb-49bd-970a-f7eaf7c4afb4\") " pod="openshift-ingress-canary/ingress-canary-lqq6s"
Sep 17 17:34:19.242045 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:19.241965    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls podName:7656e584-bfe6-4f58-bdac-fe20192d5a44 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:19.741944216 +0000 UTC m=+35.071043835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls") pod "dns-default-qpdzb" (UID: "7656e584-bfe6-4f58-bdac-fe20192d5a44") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:19.242251 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.242233    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7656e584-bfe6-4f58-bdac-fe20192d5a44-tmp-dir\") pod \"dns-default-qpdzb\" (UID: \"7656e584-bfe6-4f58-bdac-fe20192d5a44\") " pod="openshift-dns/dns-default-qpdzb"
Sep 17 17:34:19.242400 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.242383    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7656e584-bfe6-4f58-bdac-fe20192d5a44-config-volume\") pod \"dns-default-qpdzb\" (UID: \"7656e584-bfe6-4f58-bdac-fe20192d5a44\") " pod="openshift-dns/dns-default-qpdzb"
Sep 17 17:34:19.245920 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.245902    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac78b330-46cb-49bd-970a-f7eaf7c4afb4-cert\") pod \"ingress-canary-lqq6s\" (UID: \"ac78b330-46cb-49bd-970a-f7eaf7c4afb4\") " pod="openshift-ingress-canary/ingress-canary-lqq6s"
Sep 17 17:34:19.259741 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.259717    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwb6c\" (UniqueName: \"kubernetes.io/projected/7656e584-bfe6-4f58-bdac-fe20192d5a44-kube-api-access-nwb6c\") pod \"dns-default-qpdzb\" (UID: \"7656e584-bfe6-4f58-bdac-fe20192d5a44\") " pod="openshift-dns/dns-default-qpdzb"
Sep 17 17:34:19.259955 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.259938    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xvm9\" (UniqueName: \"kubernetes.io/projected/ac78b330-46cb-49bd-970a-f7eaf7c4afb4-kube-api-access-4xvm9\") pod \"ingress-canary-lqq6s\" (UID: \"ac78b330-46cb-49bd-970a-f7eaf7c4afb4\") " pod="openshift-ingress-canary/ingress-canary-lqq6s"
Sep 17 17:34:19.285652 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.285631    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:19.298793 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.298773    2574 generic.go:356] "Generic (PLEG): container finished" podID="0774cdca-c8f7-4cae-9037-633a199e0ff3" containerID="b92906d4f1e48fd7752ad62d19f5582e82c66581d334a793ec67b3c43584e9af" exitCode=0
Sep 17 17:34:19.298875 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.298818    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qxvb8" event={"ID":"0774cdca-c8f7-4cae-9037-633a199e0ff3","Type":"ContainerDied","Data":"b92906d4f1e48fd7752ad62d19f5582e82c66581d334a793ec67b3c43584e9af"}
Sep 17 17:34:19.321410 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.321378    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lqq6s"
Sep 17 17:34:19.356196 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.356140    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:19.654500 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.654263    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lqq6s"]
Sep 17 17:34:19.657103 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:34:19.657079    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac78b330_46cb_49bd_970a_f7eaf7c4afb4.slice/crio-68599e336acf55c08a7604bb62627e7618c030daa7b73fdb2f3fbe5ccdc52076 WatchSource:0}: Error finding container 68599e336acf55c08a7604bb62627e7618c030daa7b73fdb2f3fbe5ccdc52076: Status 404 returned error can't find the container with id 68599e336acf55c08a7604bb62627e7618c030daa7b73fdb2f3fbe5ccdc52076
Sep 17 17:34:19.747613 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:19.747589    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls\") pod \"dns-default-qpdzb\" (UID: \"7656e584-bfe6-4f58-bdac-fe20192d5a44\") " pod="openshift-dns/dns-default-qpdzb"
Sep 17 17:34:19.747741 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:19.747723    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:19.747788 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:19.747779    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls podName:7656e584-bfe6-4f58-bdac-fe20192d5a44 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:20.747765903 +0000 UTC m=+36.076865505 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls") pod "dns-default-qpdzb" (UID: "7656e584-bfe6-4f58-bdac-fe20192d5a44") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:20.305306 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:20.305271    2574 generic.go:356] "Generic (PLEG): container finished" podID="0774cdca-c8f7-4cae-9037-633a199e0ff3" containerID="33b56712ae5a656639f457f8242ae3a711d2972d3efd3dfe99efbfa165948749" exitCode=0
Sep 17 17:34:20.305471 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:20.305347    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qxvb8" event={"ID":"0774cdca-c8f7-4cae-9037-633a199e0ff3","Type":"ContainerDied","Data":"33b56712ae5a656639f457f8242ae3a711d2972d3efd3dfe99efbfa165948749"}
Sep 17 17:34:20.306714 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:20.306688    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lqq6s" event={"ID":"ac78b330-46cb-49bd-970a-f7eaf7c4afb4","Type":"ContainerStarted","Data":"68599e336acf55c08a7604bb62627e7618c030daa7b73fdb2f3fbe5ccdc52076"}
Sep 17 17:34:20.756341 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:20.756312    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls\") pod \"dns-default-qpdzb\" (UID: \"7656e584-bfe6-4f58-bdac-fe20192d5a44\") " pod="openshift-dns/dns-default-qpdzb"
Sep 17 17:34:20.756808 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:20.756425    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:20.756808 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:20.756484    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls podName:7656e584-bfe6-4f58-bdac-fe20192d5a44 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:22.756465348 +0000 UTC m=+38.085564957 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls") pod "dns-default-qpdzb" (UID: "7656e584-bfe6-4f58-bdac-fe20192d5a44") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:21.312022 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:21.311985    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qxvb8" event={"ID":"0774cdca-c8f7-4cae-9037-633a199e0ff3","Type":"ContainerStarted","Data":"a70bb282ceea222b5ee5906ef3d3529bfebdb120880ab32c728725f4d3797a04"}
Sep 17 17:34:21.336417 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:21.336368    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qxvb8" podStartSLOduration=8.253869138 podStartE2EDuration="36.336354698s" totalImagesPullingTime="28.08248556s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:21.335220022 +0000 UTC m=+36.664319645" watchObservedRunningTime="2026-09-17 17:34:21.336354698 +0000 UTC m=+36.665454322"
Sep 17 17:34:21.462832 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:21.462797    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:21.563392 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:21.563318    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:22.154903 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:22.154872    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:22.248986 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:22.248953    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:22.350084 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:22.350031    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:22.770607 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:22.770421    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls\") pod \"dns-default-qpdzb\" (UID: \"7656e584-bfe6-4f58-bdac-fe20192d5a44\") " pod="openshift-dns/dns-default-qpdzb"
Sep 17 17:34:22.770785 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:22.770580    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:22.770785 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:22.770739    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls podName:7656e584-bfe6-4f58-bdac-fe20192d5a44 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:26.770717294 +0000 UTC m=+42.099816899 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls") pod "dns-default-qpdzb" (UID: "7656e584-bfe6-4f58-bdac-fe20192d5a44") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:23.076328 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:23.076255    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:23.167138 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:23.167108    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:23.318886 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:23.318860    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lqq6s" event={"ID":"ac78b330-46cb-49bd-970a-f7eaf7c4afb4","Type":"ContainerStarted","Data":"97dcb040f366a3df3cda0a103b45605dd1510fa772fd54caf82d66e7dbe1b000"}
Sep 17 17:34:23.336882 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:23.336842    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lqq6s" podStartSLOduration=2.7686132839999997 podStartE2EDuration="5.336825832s" totalImagesPullingTime="2.568212548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:23.335991621 +0000 UTC m=+38.665091244" watchObservedRunningTime="2026-09-17 17:34:23.336825832 +0000 UTC m=+38.665925506"
Sep 17 17:34:23.609471 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:23.609387    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:23.719637 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:23.719602    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:24.045490 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:24.045424    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:24.265767 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:24.265733    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:24.348625 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:24.348603    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:26.795266 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:26.795231    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls\") pod \"dns-default-qpdzb\" (UID: \"7656e584-bfe6-4f58-bdac-fe20192d5a44\") " pod="openshift-dns/dns-default-qpdzb"
Sep 17 17:34:26.795609 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:26.795379    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:26.795609 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:26.795475    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls podName:7656e584-bfe6-4f58-bdac-fe20192d5a44 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:34.795459225 +0000 UTC m=+50.124558826 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls") pod "dns-default-qpdzb" (UID: "7656e584-bfe6-4f58-bdac-fe20192d5a44") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:27.034286 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:27.034254    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:27.130038 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:27.130004    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:27.231129 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:27.231082    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:27.330237 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:27.330210    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:27.457754 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:27.457697    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:33.968046 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:33.968015    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:34.067599 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:34.067572    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:34.851036 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:34.851006    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls\") pod \"dns-default-qpdzb\" (UID: \"7656e584-bfe6-4f58-bdac-fe20192d5a44\") " pod="openshift-dns/dns-default-qpdzb"
Sep 17 17:34:34.851219 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:34.851120    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:34.851219 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:34.851182    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls podName:7656e584-bfe6-4f58-bdac-fe20192d5a44 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:50.851167656 +0000 UTC m=+66.180267258 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls") pod "dns-default-qpdzb" (UID: "7656e584-bfe6-4f58-bdac-fe20192d5a44") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:37.200001 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:37.199964    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:37.277725 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:37.277690    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:38.161231 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:38.161194    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:38.231934 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:38.231911    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:44.947871 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:44.947826    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:45.034434 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:45.034403    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:45.296723 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:45.296647    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c528h"
Sep 17 17:34:45.469463 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:45.469430    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:45.549101 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:45.549017    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:50.754331 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:50.754288    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs\") pod \"network-metrics-daemon-kcglc\" (UID: \"c08424bd-0295-4dcb-9c57-3d5a5a6ba21d\") " pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:34:50.757332 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:50.757312    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:34:50.764837 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:50.764816    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:34:50.764919 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:50.764870    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs podName:c08424bd-0295-4dcb-9c57-3d5a5a6ba21d nodeName:}" failed. No retries permitted until 2026-09-17 17:35:54.764853858 +0000 UTC m=+130.093953460 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs") pod "network-metrics-daemon-kcglc" (UID: "c08424bd-0295-4dcb-9c57-3d5a5a6ba21d") : secret "metrics-daemon-secret" not found
Sep 17 17:34:50.855395 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:50.855373    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c58d2a73-a185-4e79-bbba-0b547390e9c7-original-pull-secret\") pod \"global-pull-secret-syncer-n8p54\" (UID: \"c58d2a73-a185-4e79-bbba-0b547390e9c7\") " pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:34:50.855478 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:50.855432    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls\") pod \"dns-default-qpdzb\" (UID: \"7656e584-bfe6-4f58-bdac-fe20192d5a44\") " pod="openshift-dns/dns-default-qpdzb"
Sep 17 17:34:50.855543 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:50.855500    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:50.855582 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:34:50.855545    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls podName:7656e584-bfe6-4f58-bdac-fe20192d5a44 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:22.855534779 +0000 UTC m=+98.184634381 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls") pod "dns-default-qpdzb" (UID: "7656e584-bfe6-4f58-bdac-fe20192d5a44") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:50.857995 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:50.857979    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:34:50.869336 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:50.869318    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c58d2a73-a185-4e79-bbba-0b547390e9c7-original-pull-secret\") pod \"global-pull-secret-syncer-n8p54\" (UID: \"c58d2a73-a185-4e79-bbba-0b547390e9c7\") " pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:34:50.956642 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:50.956615    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss2vl\" (UniqueName: \"kubernetes.io/projected/6916046b-8c0d-4394-8786-14b9ee9a6bee-kube-api-access-ss2vl\") pod \"network-check-target-sfmkn\" (UID: \"6916046b-8c0d-4394-8786-14b9ee9a6bee\") " pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:34:50.959450 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:50.959433    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:34:50.969254 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:50.969235    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:34:50.979912 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:50.979882    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss2vl\" (UniqueName: \"kubernetes.io/projected/6916046b-8c0d-4394-8786-14b9ee9a6bee-kube-api-access-ss2vl\") pod \"network-check-target-sfmkn\" (UID: \"6916046b-8c0d-4394-8786-14b9ee9a6bee\") " pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:34:50.987639 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:50.987621    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8p54"
Sep 17 17:34:51.259204 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:51.259176    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n8p54"]
Sep 17 17:34:51.262413 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:34:51.262385    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc58d2a73_a185_4e79_bbba_0b547390e9c7.slice/crio-d804eca7294df8db86d0bda1ec48d9885252060a846809ce66f033f6da51d828 WatchSource:0}: Error finding container d804eca7294df8db86d0bda1ec48d9885252060a846809ce66f033f6da51d828: Status 404 returned error can't find the container with id d804eca7294df8db86d0bda1ec48d9885252060a846809ce66f033f6da51d828
Sep 17 17:34:51.280797 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:51.280776    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7zxgv\""
Sep 17 17:34:51.289145 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:51.289102    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:34:51.379672 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:51.379642    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n8p54" event={"ID":"c58d2a73-a185-4e79-bbba-0b547390e9c7","Type":"ContainerStarted","Data":"d804eca7294df8db86d0bda1ec48d9885252060a846809ce66f033f6da51d828"}
Sep 17 17:34:51.562766 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:51.562705    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sfmkn"]
Sep 17 17:34:51.565355 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:34:51.565331    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6916046b_8c0d_4394_8786_14b9ee9a6bee.slice/crio-f0263d96acb70f290da0127509c2332a3d4d18c10e06e605a0c988e8cc4aeddc WatchSource:0}: Error finding container f0263d96acb70f290da0127509c2332a3d4d18c10e06e605a0c988e8cc4aeddc: Status 404 returned error can't find the container with id f0263d96acb70f290da0127509c2332a3d4d18c10e06e605a0c988e8cc4aeddc
Sep 17 17:34:52.384373 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:52.384336    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sfmkn" event={"ID":"6916046b-8c0d-4394-8786-14b9ee9a6bee","Type":"ContainerStarted","Data":"f0263d96acb70f290da0127509c2332a3d4d18c10e06e605a0c988e8cc4aeddc"}
Sep 17 17:34:56.396481 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:56.396408    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n8p54" event={"ID":"c58d2a73-a185-4e79-bbba-0b547390e9c7","Type":"ContainerStarted","Data":"960e9e930f708edfd4408867aa34649ccfbe2276a48445e4208c95db0b488db1"}
Sep 17 17:34:56.397983 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:56.397958    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sfmkn" event={"ID":"6916046b-8c0d-4394-8786-14b9ee9a6bee","Type":"ContainerStarted","Data":"5fd1788f2792026585f00d5c0daf742cb30c80bd0c1f58abaa52140370c7936e"}
Sep 17 17:34:56.398111 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:56.398091    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:34:56.416385 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:56.416318    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-n8p54" podStartSLOduration=65.919567521 podStartE2EDuration="1m10.416300343s" totalImagesPullingTime="4.496732822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:56.416205262 +0000 UTC m=+71.745304897" watchObservedRunningTime="2026-09-17 17:34:56.416300343 +0000 UTC m=+71.745399967"
Sep 17 17:34:56.441353 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:34:56.441306    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sfmkn" podStartSLOduration=66.252731505 podStartE2EDuration="1m10.441293703s" totalImagesPullingTime="4.188562198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:56.44087175 +0000 UTC m=+71.769971387" watchObservedRunningTime="2026-09-17 17:34:56.441293703 +0000 UTC m=+71.770393328"
Sep 17 17:35:00.167784 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:35:00.167751    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:00.263759 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:35:00.263735    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:00.329529 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:35:00.329502    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:22.856799 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:35:22.856758    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls\") pod \"dns-default-qpdzb\" (UID: \"7656e584-bfe6-4f58-bdac-fe20192d5a44\") " pod="openshift-dns/dns-default-qpdzb"
Sep 17 17:35:22.857160 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:35:22.856891    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:22.857160 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:35:22.856949    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls podName:7656e584-bfe6-4f58-bdac-fe20192d5a44 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:26.856932207 +0000 UTC m=+162.186031814 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls") pod "dns-default-qpdzb" (UID: "7656e584-bfe6-4f58-bdac-fe20192d5a44") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:27.402504 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:35:27.402465    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sfmkn"
Sep 17 17:35:35.382355 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:35:35.382320    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:36.868688 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:35:36.868657    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:54.854145 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:35:54.854109    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs\") pod \"network-metrics-daemon-kcglc\" (UID: \"c08424bd-0295-4dcb-9c57-3d5a5a6ba21d\") " pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:35:54.854617 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:35:54.854229    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:35:54.854617 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:35:54.854287    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs podName:c08424bd-0295-4dcb-9c57-3d5a5a6ba21d nodeName:}" failed. No retries permitted until 2026-09-17 17:37:56.854271783 +0000 UTC m=+252.183371390 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs") pod "network-metrics-daemon-kcglc" (UID: "c08424bd-0295-4dcb-9c57-3d5a5a6ba21d") : secret "metrics-daemon-secret" not found
Sep 17 17:36:18.577558 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.575927    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-67f77bb47b-mlz49"]
Sep 17 17:36:18.578677 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.578650    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:18.582845 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.582825    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 17:36:18.583244 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.583226    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 17:36:18.583328 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.583308    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 17:36:18.583611 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.583589    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-l82lc\""
Sep 17 17:36:18.583800 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.583783    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 17:36:18.583887 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.583803    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 17:36:18.583887 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.583838    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 17:36:18.605313 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.605288    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-67f77bb47b-mlz49"]
Sep 17 17:36:18.661087 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.661049    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nxp84"]
Sep 17 17:36:18.662516 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.662501    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nxp84"
Sep 17 17:36:18.664982 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.664958    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 17:36:18.665496 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.665477    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 17:36:18.667040 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.667015    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 17:36:18.667160 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.667075    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 17:36:18.667220 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.667166    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-z8c8m\""
Sep 17 17:36:18.680437 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.680416    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nxp84"]
Sep 17 17:36:18.704114 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.704094    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/999e96ab-504d-4201-9988-df670a74ee7e-metrics-certs\") pod \"router-default-67f77bb47b-mlz49\" (UID: \"999e96ab-504d-4201-9988-df670a74ee7e\") " pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:18.704210 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.704135    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7027d734-65db-4433-8b0e-387cf90fa1b8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nxp84\" (UID: \"7027d734-65db-4433-8b0e-387cf90fa1b8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nxp84"
Sep 17 17:36:18.704210 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.704155    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnglh\" (UniqueName: \"kubernetes.io/projected/999e96ab-504d-4201-9988-df670a74ee7e-kube-api-access-tnglh\") pod \"router-default-67f77bb47b-mlz49\" (UID: \"999e96ab-504d-4201-9988-df670a74ee7e\") " pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:18.704324 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.704257    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/999e96ab-504d-4201-9988-df670a74ee7e-default-certificate\") pod \"router-default-67f77bb47b-mlz49\" (UID: \"999e96ab-504d-4201-9988-df670a74ee7e\") " pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:18.704324 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.704298    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999e96ab-504d-4201-9988-df670a74ee7e-service-ca-bundle\") pod \"router-default-67f77bb47b-mlz49\" (UID: \"999e96ab-504d-4201-9988-df670a74ee7e\") " pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:18.704427 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.704325    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnnfz\" (UniqueName: \"kubernetes.io/projected/7027d734-65db-4433-8b0e-387cf90fa1b8-kube-api-access-xnnfz\") pod \"cluster-monitoring-operator-847b8bd8d7-nxp84\" (UID: \"7027d734-65db-4433-8b0e-387cf90fa1b8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nxp84"
Sep 17 17:36:18.704427 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.704356    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/999e96ab-504d-4201-9988-df670a74ee7e-stats-auth\") pod \"router-default-67f77bb47b-mlz49\" (UID: \"999e96ab-504d-4201-9988-df670a74ee7e\") " pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:18.704427 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.704386    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7027d734-65db-4433-8b0e-387cf90fa1b8-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-nxp84\" (UID: \"7027d734-65db-4433-8b0e-387cf90fa1b8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nxp84"
Sep 17 17:36:18.755316 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.755293    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-hlzfx"]
Sep 17 17:36:18.756834 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.756814    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hlzfx"
Sep 17 17:36:18.759871 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.759851    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 17:36:18.760034 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.760009    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-wsl7t\""
Sep 17 17:36:18.760143 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.760061    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 17:36:18.760324 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.760311    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 17:36:18.760372 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.760325    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:36:18.771393 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.771374    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-hlzfx"]
Sep 17 17:36:18.804662 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.804641    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb25ecfe-4234-42bb-9148-caf3f6d6fb29-serving-cert\") pod \"service-ca-operator-57cdc4b768-hlzfx\" (UID: \"fb25ecfe-4234-42bb-9148-caf3f6d6fb29\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hlzfx"
Sep 17 17:36:18.804748 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.804670    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/999e96ab-504d-4201-9988-df670a74ee7e-metrics-certs\") pod \"router-default-67f77bb47b-mlz49\" (UID: \"999e96ab-504d-4201-9988-df670a74ee7e\") " pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:18.804748 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.804724    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7027d734-65db-4433-8b0e-387cf90fa1b8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nxp84\" (UID: \"7027d734-65db-4433-8b0e-387cf90fa1b8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nxp84"
Sep 17 17:36:18.804848 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.804754    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnglh\" (UniqueName: \"kubernetes.io/projected/999e96ab-504d-4201-9988-df670a74ee7e-kube-api-access-tnglh\") pod \"router-default-67f77bb47b-mlz49\" (UID: \"999e96ab-504d-4201-9988-df670a74ee7e\") " pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:18.804848 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:18.804776    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:36:18.804848 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.804785    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb25ecfe-4234-42bb-9148-caf3f6d6fb29-config\") pod \"service-ca-operator-57cdc4b768-hlzfx\" (UID: \"fb25ecfe-4234-42bb-9148-caf3f6d6fb29\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hlzfx"
Sep 17 17:36:18.804848 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.804815    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6mc4\" (UniqueName: \"kubernetes.io/projected/fb25ecfe-4234-42bb-9148-caf3f6d6fb29-kube-api-access-g6mc4\") pod \"service-ca-operator-57cdc4b768-hlzfx\" (UID: \"fb25ecfe-4234-42bb-9148-caf3f6d6fb29\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hlzfx"
Sep 17 17:36:18.804848 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:18.804836    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/999e96ab-504d-4201-9988-df670a74ee7e-metrics-certs podName:999e96ab-504d-4201-9988-df670a74ee7e nodeName:}" failed. No retries permitted until 2026-09-17 17:36:19.304819768 +0000 UTC m=+154.633919373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/999e96ab-504d-4201-9988-df670a74ee7e-metrics-certs") pod "router-default-67f77bb47b-mlz49" (UID: "999e96ab-504d-4201-9988-df670a74ee7e") : secret "router-metrics-certs-default" not found
Sep 17 17:36:18.804848 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:18.804839    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:18.805169 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.804868    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/999e96ab-504d-4201-9988-df670a74ee7e-default-certificate\") pod \"router-default-67f77bb47b-mlz49\" (UID: \"999e96ab-504d-4201-9988-df670a74ee7e\") " pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:18.805169 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.804890    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999e96ab-504d-4201-9988-df670a74ee7e-service-ca-bundle\") pod \"router-default-67f77bb47b-mlz49\" (UID: \"999e96ab-504d-4201-9988-df670a74ee7e\") " pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:18.805169 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.804913    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnnfz\" (UniqueName: \"kubernetes.io/projected/7027d734-65db-4433-8b0e-387cf90fa1b8-kube-api-access-xnnfz\") pod \"cluster-monitoring-operator-847b8bd8d7-nxp84\" (UID: \"7027d734-65db-4433-8b0e-387cf90fa1b8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nxp84"
Sep 17 17:36:18.805169 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.804939    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/999e96ab-504d-4201-9988-df670a74ee7e-stats-auth\") pod \"router-default-67f77bb47b-mlz49\" (UID: \"999e96ab-504d-4201-9988-df670a74ee7e\") " pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:18.805169 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:18.804954    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7027d734-65db-4433-8b0e-387cf90fa1b8-cluster-monitoring-operator-tls podName:7027d734-65db-4433-8b0e-387cf90fa1b8 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:19.304934473 +0000 UTC m=+154.634034092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7027d734-65db-4433-8b0e-387cf90fa1b8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-nxp84" (UID: "7027d734-65db-4433-8b0e-387cf90fa1b8") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:18.805169 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:18.804982    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/999e96ab-504d-4201-9988-df670a74ee7e-service-ca-bundle podName:999e96ab-504d-4201-9988-df670a74ee7e nodeName:}" failed. No retries permitted until 2026-09-17 17:36:19.304970792 +0000 UTC m=+154.634070395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/999e96ab-504d-4201-9988-df670a74ee7e-service-ca-bundle") pod "router-default-67f77bb47b-mlz49" (UID: "999e96ab-504d-4201-9988-df670a74ee7e") : configmap references non-existent config key: service-ca.crt
Sep 17 17:36:18.805169 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.805009    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7027d734-65db-4433-8b0e-387cf90fa1b8-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-nxp84\" (UID: \"7027d734-65db-4433-8b0e-387cf90fa1b8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nxp84"
Sep 17 17:36:18.805745 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.805724    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7027d734-65db-4433-8b0e-387cf90fa1b8-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-nxp84\" (UID: \"7027d734-65db-4433-8b0e-387cf90fa1b8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nxp84"
Sep 17 17:36:18.807282 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.807267    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/999e96ab-504d-4201-9988-df670a74ee7e-default-certificate\") pod \"router-default-67f77bb47b-mlz49\" (UID: \"999e96ab-504d-4201-9988-df670a74ee7e\") " pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:18.807345 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.807304    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/999e96ab-504d-4201-9988-df670a74ee7e-stats-auth\") pod \"router-default-67f77bb47b-mlz49\" (UID: \"999e96ab-504d-4201-9988-df670a74ee7e\") " pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:18.813805 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.813782    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnglh\" (UniqueName: \"kubernetes.io/projected/999e96ab-504d-4201-9988-df670a74ee7e-kube-api-access-tnglh\") pod \"router-default-67f77bb47b-mlz49\" (UID: \"999e96ab-504d-4201-9988-df670a74ee7e\") " pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:18.815254 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.815233    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:18.815383 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.815367    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnnfz\" (UniqueName: \"kubernetes.io/projected/7027d734-65db-4433-8b0e-387cf90fa1b8-kube-api-access-xnnfz\") pod \"cluster-monitoring-operator-847b8bd8d7-nxp84\" (UID: \"7027d734-65db-4433-8b0e-387cf90fa1b8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nxp84"
Sep 17 17:36:18.856198 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.856172    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-dqbxl"]
Sep 17 17:36:18.857912 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.857895    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-dqbxl"
Sep 17 17:36:18.860165 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.860140    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 17:36:18.860165 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.860145    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-7nrvs\""
Sep 17 17:36:18.860743 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.860727    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 17:36:18.860835 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.860820    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 17:36:18.860901 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.860854    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 17:36:18.866592 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.866576    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 17:36:18.872257 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.872237    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-dqbxl"]
Sep 17 17:36:18.905449 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.905422    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb25ecfe-4234-42bb-9148-caf3f6d6fb29-serving-cert\") pod \"service-ca-operator-57cdc4b768-hlzfx\" (UID: \"fb25ecfe-4234-42bb-9148-caf3f6d6fb29\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hlzfx"
Sep 17 17:36:18.905549 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.905461    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/496b2135-6189-4a33-b138-112cfd1a57d2-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-dqbxl\" (UID: \"496b2135-6189-4a33-b138-112cfd1a57d2\") " pod="openshift-insights/insights-operator-6f9c644dd9-dqbxl"
Sep 17 17:36:18.905549 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.905509    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/496b2135-6189-4a33-b138-112cfd1a57d2-tmp\") pod \"insights-operator-6f9c644dd9-dqbxl\" (UID: \"496b2135-6189-4a33-b138-112cfd1a57d2\") " pod="openshift-insights/insights-operator-6f9c644dd9-dqbxl"
Sep 17 17:36:18.905549 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.905538    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/496b2135-6189-4a33-b138-112cfd1a57d2-serving-cert\") pod \"insights-operator-6f9c644dd9-dqbxl\" (UID: \"496b2135-6189-4a33-b138-112cfd1a57d2\") " pod="openshift-insights/insights-operator-6f9c644dd9-dqbxl"
Sep 17 17:36:18.905710 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.905606    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/496b2135-6189-4a33-b138-112cfd1a57d2-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-dqbxl\" (UID: \"496b2135-6189-4a33-b138-112cfd1a57d2\") " pod="openshift-insights/insights-operator-6f9c644dd9-dqbxl"
Sep 17 17:36:18.905710 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.905655    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb25ecfe-4234-42bb-9148-caf3f6d6fb29-config\") pod \"service-ca-operator-57cdc4b768-hlzfx\" (UID: \"fb25ecfe-4234-42bb-9148-caf3f6d6fb29\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hlzfx"
Sep 17 17:36:18.905710 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.905686    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6mc4\" (UniqueName: \"kubernetes.io/projected/fb25ecfe-4234-42bb-9148-caf3f6d6fb29-kube-api-access-g6mc4\") pod \"service-ca-operator-57cdc4b768-hlzfx\" (UID: \"fb25ecfe-4234-42bb-9148-caf3f6d6fb29\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hlzfx"
Sep 17 17:36:18.905853 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.905745    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/496b2135-6189-4a33-b138-112cfd1a57d2-snapshots\") pod \"insights-operator-6f9c644dd9-dqbxl\" (UID: \"496b2135-6189-4a33-b138-112cfd1a57d2\") " pod="openshift-insights/insights-operator-6f9c644dd9-dqbxl"
Sep 17 17:36:18.905853 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.905779    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b5gh\" (UniqueName: \"kubernetes.io/projected/496b2135-6189-4a33-b138-112cfd1a57d2-kube-api-access-4b5gh\") pod \"insights-operator-6f9c644dd9-dqbxl\" (UID: \"496b2135-6189-4a33-b138-112cfd1a57d2\") " pod="openshift-insights/insights-operator-6f9c644dd9-dqbxl"
Sep 17 17:36:18.906151 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.906134    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb25ecfe-4234-42bb-9148-caf3f6d6fb29-config\") pod \"service-ca-operator-57cdc4b768-hlzfx\" (UID: \"fb25ecfe-4234-42bb-9148-caf3f6d6fb29\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hlzfx"
Sep 17 17:36:18.907458 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.907441    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb25ecfe-4234-42bb-9148-caf3f6d6fb29-serving-cert\") pod \"service-ca-operator-57cdc4b768-hlzfx\" (UID: \"fb25ecfe-4234-42bb-9148-caf3f6d6fb29\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hlzfx"
Sep 17 17:36:18.907862 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.907830    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:18.914646 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:18.914630    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6mc4\" (UniqueName: \"kubernetes.io/projected/fb25ecfe-4234-42bb-9148-caf3f6d6fb29-kube-api-access-g6mc4\") pod \"service-ca-operator-57cdc4b768-hlzfx\" (UID: \"fb25ecfe-4234-42bb-9148-caf3f6d6fb29\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hlzfx"
Sep 17 17:36:19.007026 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:19.006995    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/496b2135-6189-4a33-b138-112cfd1a57d2-snapshots\") pod \"insights-operator-6f9c644dd9-dqbxl\" (UID: \"496b2135-6189-4a33-b138-112cfd1a57d2\") " pod="openshift-insights/insights-operator-6f9c644dd9-dqbxl"
Sep 17 17:36:19.007116 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:19.007034    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b5gh\" (UniqueName: \"kubernetes.io/projected/496b2135-6189-4a33-b138-112cfd1a57d2-kube-api-access-4b5gh\") pod \"insights-operator-6f9c644dd9-dqbxl\" (UID: \"496b2135-6189-4a33-b138-112cfd1a57d2\") " pod="openshift-insights/insights-operator-6f9c644dd9-dqbxl"
Sep 17 17:36:19.007116 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:19.007094    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/496b2135-6189-4a33-b138-112cfd1a57d2-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-dqbxl\" (UID: \"496b2135-6189-4a33-b138-112cfd1a57d2\") " pod="openshift-insights/insights-operator-6f9c644dd9-dqbxl"
Sep 17 17:36:19.007224 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:19.007132    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/496b2135-6189-4a33-b138-112cfd1a57d2-tmp\") pod \"insights-operator-6f9c644dd9-dqbxl\" (UID: \"496b2135-6189-4a33-b138-112cfd1a57d2\") " pod="openshift-insights/insights-operator-6f9c644dd9-dqbxl"
Sep 17 17:36:19.007224 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:19.007151    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/496b2135-6189-4a33-b138-112cfd1a57d2-serving-cert\") pod \"insights-operator-6f9c644dd9-dqbxl\" (UID: \"496b2135-6189-4a33-b138-112cfd1a57d2\") " pod="openshift-insights/insights-operator-6f9c644dd9-dqbxl"
Sep 17 17:36:19.007224 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:19.007199    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/496b2135-6189-4a33-b138-112cfd1a57d2-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-dqbxl\" (UID: \"496b2135-6189-4a33-b138-112cfd1a57d2\") " pod="openshift-insights/insights-operator-6f9c644dd9-dqbxl"
Sep 17 17:36:19.007589 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:19.007572    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/496b2135-6189-4a33-b138-112cfd1a57d2-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-dqbxl\" (UID: \"496b2135-6189-4a33-b138-112cfd1a57d2\") " pod="openshift-insights/insights-operator-6f9c644dd9-dqbxl"
Sep 17 17:36:19.007657 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:19.007633    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/496b2135-6189-4a33-b138-112cfd1a57d2-tmp\") pod \"insights-operator-6f9c644dd9-dqbxl\" (UID: \"496b2135-6189-4a33-b138-112cfd1a57d2\") " pod="openshift-insights/insights-operator-6f9c644dd9-dqbxl"
Sep 17 17:36:19.007709 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:19.007692    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/496b2135-6189-4a33-b138-112cfd1a57d2-snapshots\") pod \"insights-operator-6f9c644dd9-dqbxl\" (UID: \"496b2135-6189-4a33-b138-112cfd1a57d2\") " pod="openshift-insights/insights-operator-6f9c644dd9-dqbxl"
Sep 17 17:36:19.008007 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:19.007989    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/496b2135-6189-4a33-b138-112cfd1a57d2-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-dqbxl\" (UID: \"496b2135-6189-4a33-b138-112cfd1a57d2\") " pod="openshift-insights/insights-operator-6f9c644dd9-dqbxl"
Sep 17 17:36:19.009169 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:19.009152    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/496b2135-6189-4a33-b138-112cfd1a57d2-serving-cert\") pod \"insights-operator-6f9c644dd9-dqbxl\" (UID: \"496b2135-6189-4a33-b138-112cfd1a57d2\") " pod="openshift-insights/insights-operator-6f9c644dd9-dqbxl"
Sep 17 17:36:19.017085 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:19.017069    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b5gh\" (UniqueName: \"kubernetes.io/projected/496b2135-6189-4a33-b138-112cfd1a57d2-kube-api-access-4b5gh\") pod \"insights-operator-6f9c644dd9-dqbxl\" (UID: \"496b2135-6189-4a33-b138-112cfd1a57d2\") " pod="openshift-insights/insights-operator-6f9c644dd9-dqbxl"
Sep 17 17:36:19.064921 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:19.064883    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hlzfx"
Sep 17 17:36:19.167164 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:19.167139    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-dqbxl"
Sep 17 17:36:19.309882 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:19.309854    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/999e96ab-504d-4201-9988-df670a74ee7e-metrics-certs\") pod \"router-default-67f77bb47b-mlz49\" (UID: \"999e96ab-504d-4201-9988-df670a74ee7e\") " pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:19.310015 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:19.309908    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7027d734-65db-4433-8b0e-387cf90fa1b8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nxp84\" (UID: \"7027d734-65db-4433-8b0e-387cf90fa1b8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nxp84"
Sep 17 17:36:19.310015 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:19.310005    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:19.310136 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:19.310011    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:36:19.310136 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:19.310039    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999e96ab-504d-4201-9988-df670a74ee7e-service-ca-bundle\") pod \"router-default-67f77bb47b-mlz49\" (UID: \"999e96ab-504d-4201-9988-df670a74ee7e\") " pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:19.310136 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:19.310080    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7027d734-65db-4433-8b0e-387cf90fa1b8-cluster-monitoring-operator-tls podName:7027d734-65db-4433-8b0e-387cf90fa1b8 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:20.310046849 +0000 UTC m=+155.639146452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7027d734-65db-4433-8b0e-387cf90fa1b8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-nxp84" (UID: "7027d734-65db-4433-8b0e-387cf90fa1b8") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:19.310136 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:19.310122    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/999e96ab-504d-4201-9988-df670a74ee7e-metrics-certs podName:999e96ab-504d-4201-9988-df670a74ee7e nodeName:}" failed. No retries permitted until 2026-09-17 17:36:20.310107486 +0000 UTC m=+155.639207091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/999e96ab-504d-4201-9988-df670a74ee7e-metrics-certs") pod "router-default-67f77bb47b-mlz49" (UID: "999e96ab-504d-4201-9988-df670a74ee7e") : secret "router-metrics-certs-default" not found
Sep 17 17:36:19.310309 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:19.310146    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/999e96ab-504d-4201-9988-df670a74ee7e-service-ca-bundle podName:999e96ab-504d-4201-9988-df670a74ee7e nodeName:}" failed. No retries permitted until 2026-09-17 17:36:20.310136757 +0000 UTC m=+155.639236359 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/999e96ab-504d-4201-9988-df670a74ee7e-service-ca-bundle") pod "router-default-67f77bb47b-mlz49" (UID: "999e96ab-504d-4201-9988-df670a74ee7e") : configmap references non-existent config key: service-ca.crt
Sep 17 17:36:19.333299 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:19.333271    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-hlzfx"]
Sep 17 17:36:19.335983 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:36:19.335960    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb25ecfe_4234_42bb_9148_caf3f6d6fb29.slice/crio-43e6282ee341287cd45bce1fbc2a6da4bd403edb631dbcafcd1201738587bdbc WatchSource:0}: Error finding container 43e6282ee341287cd45bce1fbc2a6da4bd403edb631dbcafcd1201738587bdbc: Status 404 returned error can't find the container with id 43e6282ee341287cd45bce1fbc2a6da4bd403edb631dbcafcd1201738587bdbc
Sep 17 17:36:19.434598 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:19.434525    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-dqbxl"]
Sep 17 17:36:19.437158 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:36:19.437131    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496b2135_6189_4a33_b138_112cfd1a57d2.slice/crio-163654bfb8122651f42992a066514c39a99b1893a91275ae0164718cf830e00f WatchSource:0}: Error finding container 163654bfb8122651f42992a066514c39a99b1893a91275ae0164718cf830e00f: Status 404 returned error can't find the container with id 163654bfb8122651f42992a066514c39a99b1893a91275ae0164718cf830e00f
Sep 17 17:36:19.602361 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:19.602300    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-dqbxl" event={"ID":"496b2135-6189-4a33-b138-112cfd1a57d2","Type":"ContainerStarted","Data":"163654bfb8122651f42992a066514c39a99b1893a91275ae0164718cf830e00f"}
Sep 17 17:36:19.603503 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:19.603478    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hlzfx" event={"ID":"fb25ecfe-4234-42bb-9148-caf3f6d6fb29","Type":"ContainerStarted","Data":"43e6282ee341287cd45bce1fbc2a6da4bd403edb631dbcafcd1201738587bdbc"}
Sep 17 17:36:20.317568 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:20.317534    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/999e96ab-504d-4201-9988-df670a74ee7e-metrics-certs\") pod \"router-default-67f77bb47b-mlz49\" (UID: \"999e96ab-504d-4201-9988-df670a74ee7e\") " pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:20.317746 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:20.317599    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7027d734-65db-4433-8b0e-387cf90fa1b8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nxp84\" (UID: \"7027d734-65db-4433-8b0e-387cf90fa1b8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nxp84"
Sep 17 17:36:20.317746 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:20.317638    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999e96ab-504d-4201-9988-df670a74ee7e-service-ca-bundle\") pod \"router-default-67f77bb47b-mlz49\" (UID: \"999e96ab-504d-4201-9988-df670a74ee7e\") " pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:20.317921 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:20.317796    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:20.317921 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:20.317795    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:36:20.317921 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:20.317878    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/999e96ab-504d-4201-9988-df670a74ee7e-service-ca-bundle podName:999e96ab-504d-4201-9988-df670a74ee7e nodeName:}" failed. No retries permitted until 2026-09-17 17:36:22.317863163 +0000 UTC m=+157.646962770 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/999e96ab-504d-4201-9988-df670a74ee7e-service-ca-bundle") pod "router-default-67f77bb47b-mlz49" (UID: "999e96ab-504d-4201-9988-df670a74ee7e") : configmap references non-existent config key: service-ca.crt
Sep 17 17:36:20.317921 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:20.317907    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/999e96ab-504d-4201-9988-df670a74ee7e-metrics-certs podName:999e96ab-504d-4201-9988-df670a74ee7e nodeName:}" failed. No retries permitted until 2026-09-17 17:36:22.317895581 +0000 UTC m=+157.646995184 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/999e96ab-504d-4201-9988-df670a74ee7e-metrics-certs") pod "router-default-67f77bb47b-mlz49" (UID: "999e96ab-504d-4201-9988-df670a74ee7e") : secret "router-metrics-certs-default" not found
Sep 17 17:36:20.318150 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:20.317931    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7027d734-65db-4433-8b0e-387cf90fa1b8-cluster-monitoring-operator-tls podName:7027d734-65db-4433-8b0e-387cf90fa1b8 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:22.317921774 +0000 UTC m=+157.647021377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7027d734-65db-4433-8b0e-387cf90fa1b8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-nxp84" (UID: "7027d734-65db-4433-8b0e-387cf90fa1b8") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:21.421722 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:21.421679    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:21.511803 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:21.511767    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:22.007294 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:22.007252    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-qpdzb" podUID="7656e584-bfe6-4f58-bdac-fe20192d5a44"
Sep 17 17:36:22.182813 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:22.182778    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-kcglc" podUID="c08424bd-0295-4dcb-9c57-3d5a5a6ba21d"
Sep 17 17:36:22.333833 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:22.333794    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/999e96ab-504d-4201-9988-df670a74ee7e-metrics-certs\") pod \"router-default-67f77bb47b-mlz49\" (UID: \"999e96ab-504d-4201-9988-df670a74ee7e\") " pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:22.333994 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:22.333868    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7027d734-65db-4433-8b0e-387cf90fa1b8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nxp84\" (UID: \"7027d734-65db-4433-8b0e-387cf90fa1b8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nxp84"
Sep 17 17:36:22.333994 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:22.333910    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999e96ab-504d-4201-9988-df670a74ee7e-service-ca-bundle\") pod \"router-default-67f77bb47b-mlz49\" (UID: \"999e96ab-504d-4201-9988-df670a74ee7e\") " pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:22.333994 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:22.333943    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:36:22.334128 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:22.334012    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/999e96ab-504d-4201-9988-df670a74ee7e-metrics-certs podName:999e96ab-504d-4201-9988-df670a74ee7e nodeName:}" failed. No retries permitted until 2026-09-17 17:36:26.333993172 +0000 UTC m=+161.663092777 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/999e96ab-504d-4201-9988-df670a74ee7e-metrics-certs") pod "router-default-67f77bb47b-mlz49" (UID: "999e96ab-504d-4201-9988-df670a74ee7e") : secret "router-metrics-certs-default" not found
Sep 17 17:36:22.334128 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:22.334068    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/999e96ab-504d-4201-9988-df670a74ee7e-service-ca-bundle podName:999e96ab-504d-4201-9988-df670a74ee7e nodeName:}" failed. No retries permitted until 2026-09-17 17:36:26.334038519 +0000 UTC m=+161.663138127 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/999e96ab-504d-4201-9988-df670a74ee7e-service-ca-bundle") pod "router-default-67f77bb47b-mlz49" (UID: "999e96ab-504d-4201-9988-df670a74ee7e") : configmap references non-existent config key: service-ca.crt
Sep 17 17:36:22.334205 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:22.334131    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:22.334205 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:22.334181    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7027d734-65db-4433-8b0e-387cf90fa1b8-cluster-monitoring-operator-tls podName:7027d734-65db-4433-8b0e-387cf90fa1b8 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:26.334169391 +0000 UTC m=+161.663269007 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7027d734-65db-4433-8b0e-387cf90fa1b8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-nxp84" (UID: "7027d734-65db-4433-8b0e-387cf90fa1b8") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:22.615011 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:22.614974    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-dqbxl" event={"ID":"496b2135-6189-4a33-b138-112cfd1a57d2","Type":"ContainerStarted","Data":"bac0a623dd0823c2d552f6ae02e89b22137b1a647017d5d92ab96c4150bb51fd"}
Sep 17 17:36:22.616474 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:22.616441    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hlzfx" event={"ID":"fb25ecfe-4234-42bb-9148-caf3f6d6fb29","Type":"ContainerStarted","Data":"339618ef531af15c262fd19a6a2fac61dd04963b618f58a2f6b2fd31bd7ae910"}
Sep 17 17:36:22.616588 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:22.616452    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qpdzb"
Sep 17 17:36:22.640962 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:22.640914    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-dqbxl" podStartSLOduration=2.2689016410000002 podStartE2EDuration="4.640901504s" totalImagesPullingTime="2.371999863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:22.639680464 +0000 UTC m=+157.968780088" watchObservedRunningTime="2026-09-17 17:36:22.640901504 +0000 UTC m=+157.970001146"
Sep 17 17:36:22.660041 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:22.659989    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hlzfx" podStartSLOduration=2.182609572 podStartE2EDuration="4.659969893s" totalImagesPullingTime="2.477360321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:22.658882063 +0000 UTC m=+157.987981688" watchObservedRunningTime="2026-09-17 17:36:22.659969893 +0000 UTC m=+157.989069518"
Sep 17 17:36:22.913870 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:22.913790    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:23.019478 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.019446    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:23.026716 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.026692    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9jwbf"]
Sep 17 17:36:23.029921 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.029888    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9jwbf"
Sep 17 17:36:23.032378 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.032347    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 17:36:23.032649 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.032631    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kp8x9\""
Sep 17 17:36:23.033252 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.033234    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 17:36:23.040954 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.040936    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9jwbf"]
Sep 17 17:36:23.095197 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.095177    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:23.142039 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.142016    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7137d714-8bad-4b65-ad47-49d4101a4237-data-volume\") pod \"insights-runtime-extractor-9jwbf\" (UID: \"7137d714-8bad-4b65-ad47-49d4101a4237\") " pod="openshift-insights/insights-runtime-extractor-9jwbf"
Sep 17 17:36:23.142142 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.142072    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7137d714-8bad-4b65-ad47-49d4101a4237-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9jwbf\" (UID: \"7137d714-8bad-4b65-ad47-49d4101a4237\") " pod="openshift-insights/insights-runtime-extractor-9jwbf"
Sep 17 17:36:23.142142 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.142129    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7137d714-8bad-4b65-ad47-49d4101a4237-crio-socket\") pod \"insights-runtime-extractor-9jwbf\" (UID: \"7137d714-8bad-4b65-ad47-49d4101a4237\") " pod="openshift-insights/insights-runtime-extractor-9jwbf"
Sep 17 17:36:23.142231 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.142183    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7137d714-8bad-4b65-ad47-49d4101a4237-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9jwbf\" (UID: \"7137d714-8bad-4b65-ad47-49d4101a4237\") " pod="openshift-insights/insights-runtime-extractor-9jwbf"
Sep 17 17:36:23.142231 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.142221    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt9vh\" (UniqueName: \"kubernetes.io/projected/7137d714-8bad-4b65-ad47-49d4101a4237-kube-api-access-nt9vh\") pod \"insights-runtime-extractor-9jwbf\" (UID: \"7137d714-8bad-4b65-ad47-49d4101a4237\") " pod="openshift-insights/insights-runtime-extractor-9jwbf"
Sep 17 17:36:23.243616 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.243542    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7137d714-8bad-4b65-ad47-49d4101a4237-data-volume\") pod \"insights-runtime-extractor-9jwbf\" (UID: \"7137d714-8bad-4b65-ad47-49d4101a4237\") " pod="openshift-insights/insights-runtime-extractor-9jwbf"
Sep 17 17:36:23.243616 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.243575    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7137d714-8bad-4b65-ad47-49d4101a4237-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9jwbf\" (UID: \"7137d714-8bad-4b65-ad47-49d4101a4237\") " pod="openshift-insights/insights-runtime-extractor-9jwbf"
Sep 17 17:36:23.243616 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.243595    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7137d714-8bad-4b65-ad47-49d4101a4237-crio-socket\") pod \"insights-runtime-extractor-9jwbf\" (UID: \"7137d714-8bad-4b65-ad47-49d4101a4237\") " pod="openshift-insights/insights-runtime-extractor-9jwbf"
Sep 17 17:36:23.243852 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.243676    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7137d714-8bad-4b65-ad47-49d4101a4237-crio-socket\") pod \"insights-runtime-extractor-9jwbf\" (UID: \"7137d714-8bad-4b65-ad47-49d4101a4237\") " pod="openshift-insights/insights-runtime-extractor-9jwbf"
Sep 17 17:36:23.243852 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:23.243679    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:36:23.243852 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:23.243739    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7137d714-8bad-4b65-ad47-49d4101a4237-kube-rbac-proxy-cm podName:7137d714-8bad-4b65-ad47-49d4101a4237 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:23.743722875 +0000 UTC m=+159.072822477 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7137d714-8bad-4b65-ad47-49d4101a4237-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9jwbf" (UID: "7137d714-8bad-4b65-ad47-49d4101a4237") : configmap "kube-rbac-proxy" not found
Sep 17 17:36:23.243852 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.243763    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7137d714-8bad-4b65-ad47-49d4101a4237-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9jwbf\" (UID: \"7137d714-8bad-4b65-ad47-49d4101a4237\") " pod="openshift-insights/insights-runtime-extractor-9jwbf"
Sep 17 17:36:23.243852 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.243784    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt9vh\" (UniqueName: \"kubernetes.io/projected/7137d714-8bad-4b65-ad47-49d4101a4237-kube-api-access-nt9vh\") pod \"insights-runtime-extractor-9jwbf\" (UID: \"7137d714-8bad-4b65-ad47-49d4101a4237\") " pod="openshift-insights/insights-runtime-extractor-9jwbf"
Sep 17 17:36:23.244091 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:23.243879    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:23.244091 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:23.243942    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7137d714-8bad-4b65-ad47-49d4101a4237-insights-runtime-extractor-tls podName:7137d714-8bad-4b65-ad47-49d4101a4237 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:23.743925131 +0000 UTC m=+159.073024755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7137d714-8bad-4b65-ad47-49d4101a4237-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9jwbf" (UID: "7137d714-8bad-4b65-ad47-49d4101a4237") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:23.244091 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.243944    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7137d714-8bad-4b65-ad47-49d4101a4237-data-volume\") pod \"insights-runtime-extractor-9jwbf\" (UID: \"7137d714-8bad-4b65-ad47-49d4101a4237\") " pod="openshift-insights/insights-runtime-extractor-9jwbf"
Sep 17 17:36:23.268152 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.268130    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt9vh\" (UniqueName: \"kubernetes.io/projected/7137d714-8bad-4b65-ad47-49d4101a4237-kube-api-access-nt9vh\") pod \"insights-runtime-extractor-9jwbf\" (UID: \"7137d714-8bad-4b65-ad47-49d4101a4237\") " pod="openshift-insights/insights-runtime-extractor-9jwbf"
Sep 17 17:36:23.414006 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.413974    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:23.525718 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.525655    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:23.533820 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.533798    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-7f8r6"]
Sep 17 17:36:23.536563 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.536547    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7f8r6"
Sep 17 17:36:23.539160 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.539141    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-q2f72\""
Sep 17 17:36:23.539320 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.539302    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 17:36:23.544932 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.544915    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 17:36:23.570016 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.569991    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-7f8r6"]
Sep 17 17:36:23.644392 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.644349    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:23.646969 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.646947    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4cj2\" (UniqueName: \"kubernetes.io/projected/c2cebd92-d4c4-49d3-9fae-dadee9638035-kube-api-access-g4cj2\") pod \"migrator-76794cbd6c-7f8r6\" (UID: \"c2cebd92-d4c4-49d3-9fae-dadee9638035\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7f8r6"
Sep 17 17:36:23.738429 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.738398    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:23.747539 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.747511    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7137d714-8bad-4b65-ad47-49d4101a4237-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9jwbf\" (UID: \"7137d714-8bad-4b65-ad47-49d4101a4237\") " pod="openshift-insights/insights-runtime-extractor-9jwbf"
Sep 17 17:36:23.747651 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:23.747606    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:23.747722 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:23.747665    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7137d714-8bad-4b65-ad47-49d4101a4237-insights-runtime-extractor-tls podName:7137d714-8bad-4b65-ad47-49d4101a4237 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:24.747645274 +0000 UTC m=+160.076744877 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7137d714-8bad-4b65-ad47-49d4101a4237-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9jwbf" (UID: "7137d714-8bad-4b65-ad47-49d4101a4237") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:23.747722 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.747666    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4cj2\" (UniqueName: \"kubernetes.io/projected/c2cebd92-d4c4-49d3-9fae-dadee9638035-kube-api-access-g4cj2\") pod \"migrator-76794cbd6c-7f8r6\" (UID: \"c2cebd92-d4c4-49d3-9fae-dadee9638035\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7f8r6"
Sep 17 17:36:23.747818 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.747734    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7137d714-8bad-4b65-ad47-49d4101a4237-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9jwbf\" (UID: \"7137d714-8bad-4b65-ad47-49d4101a4237\") " pod="openshift-insights/insights-runtime-extractor-9jwbf"
Sep 17 17:36:23.747818 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:23.747790    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:36:23.747894 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:23.747840    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7137d714-8bad-4b65-ad47-49d4101a4237-kube-rbac-proxy-cm podName:7137d714-8bad-4b65-ad47-49d4101a4237 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:24.747826188 +0000 UTC m=+160.076925794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7137d714-8bad-4b65-ad47-49d4101a4237-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9jwbf" (UID: "7137d714-8bad-4b65-ad47-49d4101a4237") : configmap "kube-rbac-proxy" not found
Sep 17 17:36:23.757763 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.757738    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4cj2\" (UniqueName: \"kubernetes.io/projected/c2cebd92-d4c4-49d3-9fae-dadee9638035-kube-api-access-g4cj2\") pod \"migrator-76794cbd6c-7f8r6\" (UID: \"c2cebd92-d4c4-49d3-9fae-dadee9638035\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7f8r6"
Sep 17 17:36:23.845794 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:23.845776    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7f8r6"
Sep 17 17:36:24.116881 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:24.116803    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-7f8r6"]
Sep 17 17:36:24.120487 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:36:24.120459    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2cebd92_d4c4_49d3_9fae_dadee9638035.slice/crio-30845ed1a6d55c649bad476159c5c9e586431b65f06a6e60784574bee3373e1c WatchSource:0}: Error finding container 30845ed1a6d55c649bad476159c5c9e586431b65f06a6e60784574bee3373e1c: Status 404 returned error can't find the container with id 30845ed1a6d55c649bad476159c5c9e586431b65f06a6e60784574bee3373e1c
Sep 17 17:36:24.622115 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:24.622072    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7f8r6" event={"ID":"c2cebd92-d4c4-49d3-9fae-dadee9638035","Type":"ContainerStarted","Data":"30845ed1a6d55c649bad476159c5c9e586431b65f06a6e60784574bee3373e1c"}
Sep 17 17:36:24.755191 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:24.755157    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7137d714-8bad-4b65-ad47-49d4101a4237-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9jwbf\" (UID: \"7137d714-8bad-4b65-ad47-49d4101a4237\") " pod="openshift-insights/insights-runtime-extractor-9jwbf"
Sep 17 17:36:24.755694 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:24.755256    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:36:24.755694 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:24.755380    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:24.755694 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:24.755394    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7137d714-8bad-4b65-ad47-49d4101a4237-kube-rbac-proxy-cm podName:7137d714-8bad-4b65-ad47-49d4101a4237 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:26.755371743 +0000 UTC m=+162.084471358 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7137d714-8bad-4b65-ad47-49d4101a4237-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9jwbf" (UID: "7137d714-8bad-4b65-ad47-49d4101a4237") : configmap "kube-rbac-proxy" not found
Sep 17 17:36:24.755694 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:24.755295    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7137d714-8bad-4b65-ad47-49d4101a4237-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9jwbf\" (UID: \"7137d714-8bad-4b65-ad47-49d4101a4237\") " pod="openshift-insights/insights-runtime-extractor-9jwbf"
Sep 17 17:36:24.755694 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:24.755434    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7137d714-8bad-4b65-ad47-49d4101a4237-insights-runtime-extractor-tls podName:7137d714-8bad-4b65-ad47-49d4101a4237 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:26.755418098 +0000 UTC m=+162.084517705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7137d714-8bad-4b65-ad47-49d4101a4237-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9jwbf" (UID: "7137d714-8bad-4b65-ad47-49d4101a4237") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:24.828037 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:24.828003    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:25.148633 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:25.148611    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jvdsb_b5dd23d0-8b89-453c-99ee-dfd8a18c683c/dns-node-resolver/0.log"
Sep 17 17:36:25.256334 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:25.256308    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:25.328039 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:25.327972    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:25.627652 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:25.627617    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7f8r6" event={"ID":"c2cebd92-d4c4-49d3-9fae-dadee9638035","Type":"ContainerStarted","Data":"3e14ed98f45fa3e08a39d5674c0cd066911584bfbe31eee21b3c636aab19c27a"}
Sep 17 17:36:25.627817 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:25.627668    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7f8r6" event={"ID":"c2cebd92-d4c4-49d3-9fae-dadee9638035","Type":"ContainerStarted","Data":"31531ab42eeeaa336dfcf3a24533e8aed64b4d0a356478bd332dcebda6b40d23"}
Sep 17 17:36:25.645684 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:25.645631    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7f8r6" podStartSLOduration=1.719610373 podStartE2EDuration="2.645616581s" totalImagesPullingTime="926.006208ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:25.644085809 +0000 UTC m=+160.973185430" watchObservedRunningTime="2026-09-17 17:36:25.645616581 +0000 UTC m=+160.974716206"
Sep 17 17:36:25.741730 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:25.741702    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:25.849004 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:25.848982    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:25.949324 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:25.949266    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cbnqr_a811ebbe-f6a5-4bac-a036-e3d2f439f394/node-ca/0.log"
Sep 17 17:36:26.365606 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:26.365578    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999e96ab-504d-4201-9988-df670a74ee7e-service-ca-bundle\") pod \"router-default-67f77bb47b-mlz49\" (UID: \"999e96ab-504d-4201-9988-df670a74ee7e\") " pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:26.365769 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:26.365617    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/999e96ab-504d-4201-9988-df670a74ee7e-metrics-certs\") pod \"router-default-67f77bb47b-mlz49\" (UID: \"999e96ab-504d-4201-9988-df670a74ee7e\") " pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:26.365769 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:26.365636    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7027d734-65db-4433-8b0e-387cf90fa1b8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nxp84\" (UID: \"7027d734-65db-4433-8b0e-387cf90fa1b8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nxp84"
Sep 17 17:36:26.365769 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:26.365721    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/999e96ab-504d-4201-9988-df670a74ee7e-service-ca-bundle podName:999e96ab-504d-4201-9988-df670a74ee7e nodeName:}" failed. No retries permitted until 2026-09-17 17:36:34.36570428 +0000 UTC m=+169.694803881 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/999e96ab-504d-4201-9988-df670a74ee7e-service-ca-bundle") pod "router-default-67f77bb47b-mlz49" (UID: "999e96ab-504d-4201-9988-df670a74ee7e") : configmap references non-existent config key: service-ca.crt
Sep 17 17:36:26.365769 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:26.365728    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:26.365769 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:26.365738    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:36:26.365769 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:26.365766    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7027d734-65db-4433-8b0e-387cf90fa1b8-cluster-monitoring-operator-tls podName:7027d734-65db-4433-8b0e-387cf90fa1b8 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:34.365755558 +0000 UTC m=+169.694855160 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7027d734-65db-4433-8b0e-387cf90fa1b8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-nxp84" (UID: "7027d734-65db-4433-8b0e-387cf90fa1b8") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:26.366010 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:26.365795    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/999e96ab-504d-4201-9988-df670a74ee7e-metrics-certs podName:999e96ab-504d-4201-9988-df670a74ee7e nodeName:}" failed. No retries permitted until 2026-09-17 17:36:34.365780911 +0000 UTC m=+169.694880518 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/999e96ab-504d-4201-9988-df670a74ee7e-metrics-certs") pod "router-default-67f77bb47b-mlz49" (UID: "999e96ab-504d-4201-9988-df670a74ee7e") : secret "router-metrics-certs-default" not found
Sep 17 17:36:26.437516 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:26.437489    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:26.768442 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:26.768362    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7137d714-8bad-4b65-ad47-49d4101a4237-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9jwbf\" (UID: \"7137d714-8bad-4b65-ad47-49d4101a4237\") " pod="openshift-insights/insights-runtime-extractor-9jwbf"
Sep 17 17:36:26.768442 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:26.768417    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7137d714-8bad-4b65-ad47-49d4101a4237-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9jwbf\" (UID: \"7137d714-8bad-4b65-ad47-49d4101a4237\") " pod="openshift-insights/insights-runtime-extractor-9jwbf"
Sep 17 17:36:26.768651 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:26.768487    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:36:26.768651 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:26.768526    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:26.768651 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:26.768541    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7137d714-8bad-4b65-ad47-49d4101a4237-kube-rbac-proxy-cm podName:7137d714-8bad-4b65-ad47-49d4101a4237 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:30.768522681 +0000 UTC m=+166.097622297 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7137d714-8bad-4b65-ad47-49d4101a4237-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9jwbf" (UID: "7137d714-8bad-4b65-ad47-49d4101a4237") : configmap "kube-rbac-proxy" not found
Sep 17 17:36:26.768651 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:26.768583    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7137d714-8bad-4b65-ad47-49d4101a4237-insights-runtime-extractor-tls podName:7137d714-8bad-4b65-ad47-49d4101a4237 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:30.768566803 +0000 UTC m=+166.097666404 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7137d714-8bad-4b65-ad47-49d4101a4237-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9jwbf" (UID: "7137d714-8bad-4b65-ad47-49d4101a4237") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:26.869121 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:26.869091    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls\") pod \"dns-default-qpdzb\" (UID: \"7656e584-bfe6-4f58-bdac-fe20192d5a44\") " pod="openshift-dns/dns-default-qpdzb"
Sep 17 17:36:26.869435 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:26.869196    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:36:26.869435 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:26.869242    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls podName:7656e584-bfe6-4f58-bdac-fe20192d5a44 nodeName:}" failed. No retries permitted until 2026-09-17 17:38:28.869227689 +0000 UTC m=+284.198327297 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls") pod "dns-default-qpdzb" (UID: "7656e584-bfe6-4f58-bdac-fe20192d5a44") : secret "dns-default-metrics-tls" not found
Sep 17 17:36:26.948733 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:26.948713    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lqq6s_ac78b330-46cb-49bd-970a-f7eaf7c4afb4/serve-healthcheck-canary/0.log"
Sep 17 17:36:28.744141 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:28.744106    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:28.875755 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:28.875720    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:28.982579 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:28.982548    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:29.081829 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:29.081805    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:29.159913 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:29.159884    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:29.240566 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:29.240543    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:29.446168 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:29.446099    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:29.535121 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:29.535098    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:30.240502 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:30.240474    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:30.636947 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:30.636914    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:30.793300 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:30.793278    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7137d714-8bad-4b65-ad47-49d4101a4237-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9jwbf\" (UID: \"7137d714-8bad-4b65-ad47-49d4101a4237\") " pod="openshift-insights/insights-runtime-extractor-9jwbf"
Sep 17 17:36:30.793477 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:30.793317    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7137d714-8bad-4b65-ad47-49d4101a4237-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9jwbf\" (UID: \"7137d714-8bad-4b65-ad47-49d4101a4237\") " pod="openshift-insights/insights-runtime-extractor-9jwbf"
Sep 17 17:36:30.793477 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:30.793422    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:30.793549 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:30.793483    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7137d714-8bad-4b65-ad47-49d4101a4237-insights-runtime-extractor-tls podName:7137d714-8bad-4b65-ad47-49d4101a4237 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:38.793468356 +0000 UTC m=+174.122567958 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7137d714-8bad-4b65-ad47-49d4101a4237-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9jwbf" (UID: "7137d714-8bad-4b65-ad47-49d4101a4237") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:30.793549 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:30.793423    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:36:30.793618 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:30.793557    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7137d714-8bad-4b65-ad47-49d4101a4237-kube-rbac-proxy-cm podName:7137d714-8bad-4b65-ad47-49d4101a4237 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:38.793546069 +0000 UTC m=+174.122645671 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7137d714-8bad-4b65-ad47-49d4101a4237-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9jwbf" (UID: "7137d714-8bad-4b65-ad47-49d4101a4237") : configmap "kube-rbac-proxy" not found
Sep 17 17:36:31.045934 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:31.045865    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:31.239039 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:31.239011    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:31.853569 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:31.853532    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:32.032517 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:32.032489    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:32.448543 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:32.448513    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:32.680064 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:32.680034    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:32.751153 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:32.751085    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:32.835068 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:32.835029    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:32.936814 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:32.936786    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:33.444218 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:33.444190    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:33.858925 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:33.858900    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:34.250333 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:34.250258    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:34.421564 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:34.421535    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999e96ab-504d-4201-9988-df670a74ee7e-service-ca-bundle\") pod \"router-default-67f77bb47b-mlz49\" (UID: \"999e96ab-504d-4201-9988-df670a74ee7e\") " pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:34.421724 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:34.421578    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/999e96ab-504d-4201-9988-df670a74ee7e-metrics-certs\") pod \"router-default-67f77bb47b-mlz49\" (UID: \"999e96ab-504d-4201-9988-df670a74ee7e\") " pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:34.421724 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:34.421603    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7027d734-65db-4433-8b0e-387cf90fa1b8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nxp84\" (UID: \"7027d734-65db-4433-8b0e-387cf90fa1b8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nxp84"
Sep 17 17:36:34.421724 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:34.421689    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:34.421724 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:34.421690    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/999e96ab-504d-4201-9988-df670a74ee7e-service-ca-bundle podName:999e96ab-504d-4201-9988-df670a74ee7e nodeName:}" failed. No retries permitted until 2026-09-17 17:36:50.421668417 +0000 UTC m=+185.750768019 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/999e96ab-504d-4201-9988-df670a74ee7e-service-ca-bundle") pod "router-default-67f77bb47b-mlz49" (UID: "999e96ab-504d-4201-9988-df670a74ee7e") : configmap references non-existent config key: service-ca.crt
Sep 17 17:36:34.421945 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:36:34.421742    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7027d734-65db-4433-8b0e-387cf90fa1b8-cluster-monitoring-operator-tls podName:7027d734-65db-4433-8b0e-387cf90fa1b8 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:50.421725505 +0000 UTC m=+185.750825107 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7027d734-65db-4433-8b0e-387cf90fa1b8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-nxp84" (UID: "7027d734-65db-4433-8b0e-387cf90fa1b8") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:34.423874 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:34.423848    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/999e96ab-504d-4201-9988-df670a74ee7e-metrics-certs\") pod \"router-default-67f77bb47b-mlz49\" (UID: \"999e96ab-504d-4201-9988-df670a74ee7e\") " pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:34.440884 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:34.440854    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:35.055214 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:35.055176    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:35.244043 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:35.244010    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:35.445168 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:35.445135    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:35.851781 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:35.851751    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:36.098773 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:36.098745    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:36.264219 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:36.264151    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:36.448851 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:36.448819    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:37.066973 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:37.066940    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:37.110505 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:37.110484    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:36:37.446258 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:37.446226    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:37.847287 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:37.847258    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:38.046188 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:38.046159    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:38.650480 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:38.650450    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:38.854389 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:38.854361    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7137d714-8bad-4b65-ad47-49d4101a4237-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9jwbf\" (UID: \"7137d714-8bad-4b65-ad47-49d4101a4237\") " pod="openshift-insights/insights-runtime-extractor-9jwbf"
Sep 17 17:36:38.854732 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:38.854409    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7137d714-8bad-4b65-ad47-49d4101a4237-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9jwbf\" (UID: \"7137d714-8bad-4b65-ad47-49d4101a4237\") " pod="openshift-insights/insights-runtime-extractor-9jwbf"
Sep 17 17:36:38.854880 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:38.854859    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7137d714-8bad-4b65-ad47-49d4101a4237-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9jwbf\" (UID: \"7137d714-8bad-4b65-ad47-49d4101a4237\") " pod="openshift-insights/insights-runtime-extractor-9jwbf"
Sep 17 17:36:38.856578 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:38.856562    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7137d714-8bad-4b65-ad47-49d4101a4237-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9jwbf\" (UID: \"7137d714-8bad-4b65-ad47-49d4101a4237\") " pod="openshift-insights/insights-runtime-extractor-9jwbf"
Sep 17 17:36:38.939488 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:38.939430    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9jwbf"
Sep 17 17:36:39.060074 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:39.058073    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:39.229415 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:39.229343    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9jwbf"]
Sep 17 17:36:39.232568 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:36:39.232545    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7137d714_8bad_4b65_ad47_49d4101a4237.slice/crio-10366a4285dc79eeafe5e9fee18e29153e53c948681046105858629c5ef6f2ff WatchSource:0}: Error finding container 10366a4285dc79eeafe5e9fee18e29153e53c948681046105858629c5ef6f2ff: Status 404 returned error can't find the container with id 10366a4285dc79eeafe5e9fee18e29153e53c948681046105858629c5ef6f2ff
Sep 17 17:36:39.266188 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:39.266161    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:39.451030 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:39.451001    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:39.670725 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:39.670692    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9jwbf" event={"ID":"7137d714-8bad-4b65-ad47-49d4101a4237","Type":"ContainerStarted","Data":"d770ba3619575b26538c9dfaa94c59c27d286e9960a48412c634b18ba1902862"}
Sep 17 17:36:39.670876 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:39.670748    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9jwbf" event={"ID":"7137d714-8bad-4b65-ad47-49d4101a4237","Type":"ContainerStarted","Data":"10366a4285dc79eeafe5e9fee18e29153e53c948681046105858629c5ef6f2ff"}
Sep 17 17:36:40.676274 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:40.676243    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9jwbf" event={"ID":"7137d714-8bad-4b65-ad47-49d4101a4237","Type":"ContainerStarted","Data":"d126e6144a137a46e59026ea656b21536dcdc424f7690def7df94e7586c8e0f5"}
Sep 17 17:36:41.681035 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:41.681006    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9jwbf" event={"ID":"7137d714-8bad-4b65-ad47-49d4101a4237","Type":"ContainerStarted","Data":"449b1a20537a9d78dd49a1c939d6a2ffc591d19be03d671cf2bccc68a146283a"}
Sep 17 17:36:41.704080 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:41.704025    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9jwbf" podStartSLOduration=16.760498322 podStartE2EDuration="18.704011864s" totalImagesPullingTime="1.943513542s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:36:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:41.703175708 +0000 UTC m=+177.032275332" watchObservedRunningTime="2026-09-17 17:36:41.704011864 +0000 UTC m=+177.033111485"
Sep 17 17:36:42.288592 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:42.288561    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:42.365987 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:42.365952    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:43.433240 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:43.433209    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:43.527308 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:43.527281    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:43.602590 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:43.602563    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:43.717960 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:43.717904    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:43.896680 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:43.896652    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:43.977627 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:43.977569    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:44.107087 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:44.107047    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:44.299972 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:44.299914    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:44.402625 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:44.402602    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:45.306748 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:45.306717    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:45.521414 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:45.521384    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:45.624535 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:45.624508    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:45.746447 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:45.746419    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:45.873121 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:45.873096    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:47.596354 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:47.596323    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-wt58c"]
Sep 17 17:36:47.599442 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:47.599423    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-wt58c"
Sep 17 17:36:47.601915 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:47.601891    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 17:36:47.602022 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:47.602000    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 17:36:47.602229 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:47.602215    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-pkhkv\""
Sep 17 17:36:47.611227 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:47.611205    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-wt58c"]
Sep 17 17:36:47.694805 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:47.694782    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:47.715483 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:47.715462    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr6kr\" (UniqueName: \"kubernetes.io/projected/efe4cd38-a36c-445f-aeb9-0f7167b306ec-kube-api-access-dr6kr\") pod \"downloads-6594f4c75-wt58c\" (UID: \"efe4cd38-a36c-445f-aeb9-0f7167b306ec\") " pod="openshift-console/downloads-6594f4c75-wt58c"
Sep 17 17:36:47.786494 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:47.786468    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:47.816625 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:47.816603    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr6kr\" (UniqueName: \"kubernetes.io/projected/efe4cd38-a36c-445f-aeb9-0f7167b306ec-kube-api-access-dr6kr\") pod \"downloads-6594f4c75-wt58c\" (UID: \"efe4cd38-a36c-445f-aeb9-0f7167b306ec\") " pod="openshift-console/downloads-6594f4c75-wt58c"
Sep 17 17:36:47.827154 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:47.827128    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr6kr\" (UniqueName: \"kubernetes.io/projected/efe4cd38-a36c-445f-aeb9-0f7167b306ec-kube-api-access-dr6kr\") pod \"downloads-6594f4c75-wt58c\" (UID: \"efe4cd38-a36c-445f-aeb9-0f7167b306ec\") " pod="openshift-console/downloads-6594f4c75-wt58c"
Sep 17 17:36:47.907830 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:47.907781    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-wt58c"
Sep 17 17:36:48.181921 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:48.181849    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-wt58c"]
Sep 17 17:36:48.185491 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:36:48.185463    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe4cd38_a36c_445f_aeb9_0f7167b306ec.slice/crio-a3c4a8ee3b5b0a1b29067e7a1aa80b1fcc42bf7a69a45861be1837f5fb2df0a1 WatchSource:0}: Error finding container a3c4a8ee3b5b0a1b29067e7a1aa80b1fcc42bf7a69a45861be1837f5fb2df0a1: Status 404 returned error can't find the container with id a3c4a8ee3b5b0a1b29067e7a1aa80b1fcc42bf7a69a45861be1837f5fb2df0a1
Sep 17 17:36:48.701805 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:48.701771    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-wt58c" event={"ID":"efe4cd38-a36c-445f-aeb9-0f7167b306ec","Type":"ContainerStarted","Data":"a3c4a8ee3b5b0a1b29067e7a1aa80b1fcc42bf7a69a45861be1837f5fb2df0a1"}
Sep 17 17:36:50.436022 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:50.435987    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999e96ab-504d-4201-9988-df670a74ee7e-service-ca-bundle\") pod \"router-default-67f77bb47b-mlz49\" (UID: \"999e96ab-504d-4201-9988-df670a74ee7e\") " pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:50.436508 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:50.436038    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7027d734-65db-4433-8b0e-387cf90fa1b8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nxp84\" (UID: \"7027d734-65db-4433-8b0e-387cf90fa1b8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nxp84"
Sep 17 17:36:50.436770 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:50.436744    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/999e96ab-504d-4201-9988-df670a74ee7e-service-ca-bundle\") pod \"router-default-67f77bb47b-mlz49\" (UID: \"999e96ab-504d-4201-9988-df670a74ee7e\") " pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:50.438730 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:50.438702    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7027d734-65db-4433-8b0e-387cf90fa1b8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nxp84\" (UID: \"7027d734-65db-4433-8b0e-387cf90fa1b8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nxp84"
Sep 17 17:36:50.473749 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:50.473728    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-z8c8m\""
Sep 17 17:36:50.481020 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:50.481002    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nxp84"
Sep 17 17:36:50.689947 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:50.689870    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-l82lc\""
Sep 17 17:36:50.697957 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:50.697927    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:50.760851 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:50.760774    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nxp84"]
Sep 17 17:36:50.764570 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:36:50.764540    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7027d734_65db_4433_8b0e_387cf90fa1b8.slice/crio-45b7cf5faedb9144a733353786b7b5bdd8b993d03d506af6ef0ab236b67c1d88 WatchSource:0}: Error finding container 45b7cf5faedb9144a733353786b7b5bdd8b993d03d506af6ef0ab236b67c1d88: Status 404 returned error can't find the container with id 45b7cf5faedb9144a733353786b7b5bdd8b993d03d506af6ef0ab236b67c1d88
Sep 17 17:36:50.928758 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:50.928726    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:50.984406 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:50.984338    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-67f77bb47b-mlz49"]
Sep 17 17:36:50.987234 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:36:50.987204    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod999e96ab_504d_4201_9988_df670a74ee7e.slice/crio-c256394a01611913b2f9a6da6c998f85cc82c07ffc90f051603aad646369c1bd WatchSource:0}: Error finding container c256394a01611913b2f9a6da6c998f85cc82c07ffc90f051603aad646369c1bd: Status 404 returned error can't find the container with id c256394a01611913b2f9a6da6c998f85cc82c07ffc90f051603aad646369c1bd
Sep 17 17:36:51.029375 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:51.029351    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:51.713828 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:51.713792    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-67f77bb47b-mlz49" event={"ID":"999e96ab-504d-4201-9988-df670a74ee7e","Type":"ContainerStarted","Data":"a94a90f939b87c19b6d45826f1b35156d5f02fb5ade04ac15896cc10b1e85645"}
Sep 17 17:36:51.714329 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:51.713849    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-67f77bb47b-mlz49" event={"ID":"999e96ab-504d-4201-9988-df670a74ee7e","Type":"ContainerStarted","Data":"c256394a01611913b2f9a6da6c998f85cc82c07ffc90f051603aad646369c1bd"}
Sep 17 17:36:51.715584 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:51.715551    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nxp84" event={"ID":"7027d734-65db-4433-8b0e-387cf90fa1b8","Type":"ContainerStarted","Data":"45b7cf5faedb9144a733353786b7b5bdd8b993d03d506af6ef0ab236b67c1d88"}
Sep 17 17:36:51.740844 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:51.740801    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-67f77bb47b-mlz49" podStartSLOduration=33.740782823 podStartE2EDuration="33.740782823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:51.740007908 +0000 UTC m=+187.069107533" watchObservedRunningTime="2026-09-17 17:36:51.740782823 +0000 UTC m=+187.069882448"
Sep 17 17:36:51.937912 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:51.937875    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86b4799dc9-k6whf"]
Sep 17 17:36:51.941003 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:51.940983    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86b4799dc9-k6whf"
Sep 17 17:36:51.943588 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:51.943522    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 17:36:51.943588 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:51.943553    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 17:36:51.943778 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:51.943588    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 17:36:51.944206 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:51.943832    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xdhkz\""
Sep 17 17:36:51.944206 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:51.943948    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 17:36:51.944206 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:51.944018    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 17:36:51.950964 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:51.950930    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86b4799dc9-k6whf"]
Sep 17 17:36:52.051932 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.051858    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fc0acd7-708f-44ab-83c5-bb6fa7891010-console-config\") pod \"console-86b4799dc9-k6whf\" (UID: \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\") " pod="openshift-console/console-86b4799dc9-k6whf"
Sep 17 17:36:52.051932 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.051915    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fc0acd7-708f-44ab-83c5-bb6fa7891010-console-oauth-config\") pod \"console-86b4799dc9-k6whf\" (UID: \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\") " pod="openshift-console/console-86b4799dc9-k6whf"
Sep 17 17:36:52.052145 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.052032    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc0acd7-708f-44ab-83c5-bb6fa7891010-service-ca\") pod \"console-86b4799dc9-k6whf\" (UID: \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\") " pod="openshift-console/console-86b4799dc9-k6whf"
Sep 17 17:36:52.052199 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.052147    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl7dn\" (UniqueName: \"kubernetes.io/projected/2fc0acd7-708f-44ab-83c5-bb6fa7891010-kube-api-access-gl7dn\") pod \"console-86b4799dc9-k6whf\" (UID: \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\") " pod="openshift-console/console-86b4799dc9-k6whf"
Sep 17 17:36:52.052199 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.052179    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fc0acd7-708f-44ab-83c5-bb6fa7891010-console-serving-cert\") pod \"console-86b4799dc9-k6whf\" (UID: \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\") " pod="openshift-console/console-86b4799dc9-k6whf"
Sep 17 17:36:52.052298 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.052205    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fc0acd7-708f-44ab-83c5-bb6fa7891010-oauth-serving-cert\") pod \"console-86b4799dc9-k6whf\" (UID: \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\") " pod="openshift-console/console-86b4799dc9-k6whf"
Sep 17 17:36:52.153299 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.153261    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl7dn\" (UniqueName: \"kubernetes.io/projected/2fc0acd7-708f-44ab-83c5-bb6fa7891010-kube-api-access-gl7dn\") pod \"console-86b4799dc9-k6whf\" (UID: \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\") " pod="openshift-console/console-86b4799dc9-k6whf"
Sep 17 17:36:52.153472 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.153310    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fc0acd7-708f-44ab-83c5-bb6fa7891010-console-serving-cert\") pod \"console-86b4799dc9-k6whf\" (UID: \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\") " pod="openshift-console/console-86b4799dc9-k6whf"
Sep 17 17:36:52.153472 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.153340    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fc0acd7-708f-44ab-83c5-bb6fa7891010-oauth-serving-cert\") pod \"console-86b4799dc9-k6whf\" (UID: \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\") " pod="openshift-console/console-86b4799dc9-k6whf"
Sep 17 17:36:52.153472 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.153372    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fc0acd7-708f-44ab-83c5-bb6fa7891010-console-config\") pod \"console-86b4799dc9-k6whf\" (UID: \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\") " pod="openshift-console/console-86b4799dc9-k6whf"
Sep 17 17:36:52.153472 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.153407    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fc0acd7-708f-44ab-83c5-bb6fa7891010-console-oauth-config\") pod \"console-86b4799dc9-k6whf\" (UID: \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\") " pod="openshift-console/console-86b4799dc9-k6whf"
Sep 17 17:36:52.153472 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.153467    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc0acd7-708f-44ab-83c5-bb6fa7891010-service-ca\") pod \"console-86b4799dc9-k6whf\" (UID: \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\") " pod="openshift-console/console-86b4799dc9-k6whf"
Sep 17 17:36:52.154885 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.154228    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc0acd7-708f-44ab-83c5-bb6fa7891010-service-ca\") pod \"console-86b4799dc9-k6whf\" (UID: \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\") " pod="openshift-console/console-86b4799dc9-k6whf"
Sep 17 17:36:52.155338 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.155310    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fc0acd7-708f-44ab-83c5-bb6fa7891010-console-config\") pod \"console-86b4799dc9-k6whf\" (UID: \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\") " pod="openshift-console/console-86b4799dc9-k6whf"
Sep 17 17:36:52.155453 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.155342    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fc0acd7-708f-44ab-83c5-bb6fa7891010-oauth-serving-cert\") pod \"console-86b4799dc9-k6whf\" (UID: \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\") " pod="openshift-console/console-86b4799dc9-k6whf"
Sep 17 17:36:52.157491 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.157467    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fc0acd7-708f-44ab-83c5-bb6fa7891010-console-oauth-config\") pod \"console-86b4799dc9-k6whf\" (UID: \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\") " pod="openshift-console/console-86b4799dc9-k6whf"
Sep 17 17:36:52.157601 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.157578    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fc0acd7-708f-44ab-83c5-bb6fa7891010-console-serving-cert\") pod \"console-86b4799dc9-k6whf\" (UID: \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\") " pod="openshift-console/console-86b4799dc9-k6whf"
Sep 17 17:36:52.165431 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.165402    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl7dn\" (UniqueName: \"kubernetes.io/projected/2fc0acd7-708f-44ab-83c5-bb6fa7891010-kube-api-access-gl7dn\") pod \"console-86b4799dc9-k6whf\" (UID: \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\") " pod="openshift-console/console-86b4799dc9-k6whf"
Sep 17 17:36:52.252968 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.252929    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86b4799dc9-k6whf"
Sep 17 17:36:52.699115 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.699013    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:52.703088 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.703044    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:52.721204 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.721172    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nxp84" event={"ID":"7027d734-65db-4433-8b0e-387cf90fa1b8","Type":"ContainerStarted","Data":"e261018f5ed6cffbe3c6c1bd602e58062167e417aeecafeaead2b6d67cd6e60b"}
Sep 17 17:36:52.721611 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.721223    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:52.722662 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.722619    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-67f77bb47b-mlz49"
Sep 17 17:36:52.745395 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.745301    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nxp84" podStartSLOduration=32.979463779 podStartE2EDuration="34.745281985s" totalImagesPullingTime="1.765818206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:52.743632131 +0000 UTC m=+188.072731756" watchObservedRunningTime="2026-09-17 17:36:52.745281985 +0000 UTC m=+188.074381818"
Sep 17 17:36:52.759126 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.759102    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86b4799dc9-k6whf"]
Sep 17 17:36:52.762696 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:36:52.762669    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc0acd7_708f_44ab_83c5_bb6fa7891010.slice/crio-910874de1bffbcd57c66e15242b81c572e0eac51402d1737db3438f886aa7cd7 WatchSource:0}: Error finding container 910874de1bffbcd57c66e15242b81c572e0eac51402d1737db3438f886aa7cd7: Status 404 returned error can't find the container with id 910874de1bffbcd57c66e15242b81c572e0eac51402d1737db3438f886aa7cd7
Sep 17 17:36:52.880620 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.880581    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:52.994710 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:52.994677    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:53.067651 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:53.067581    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-r9x5k"]
Sep 17 17:36:53.070987 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:53.070966    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-r9x5k"
Sep 17 17:36:53.073263 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:53.073242    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 17:36:53.073356 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:53.073272    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-j2gmt\""
Sep 17 17:36:53.079123 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:53.079101    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-r9x5k"]
Sep 17 17:36:53.083111 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:53.083088    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:53.164894 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:53.164862    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5f71b6c1-e6be-4c65-a5de-e4a2a9e728e7-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-r9x5k\" (UID: \"5f71b6c1-e6be-4c65-a5de-e4a2a9e728e7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-r9x5k"
Sep 17 17:36:53.266140 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:53.266096    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5f71b6c1-e6be-4c65-a5de-e4a2a9e728e7-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-r9x5k\" (UID: \"5f71b6c1-e6be-4c65-a5de-e4a2a9e728e7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-r9x5k"
Sep 17 17:36:53.273147 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:53.273117    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5f71b6c1-e6be-4c65-a5de-e4a2a9e728e7-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-r9x5k\" (UID: \"5f71b6c1-e6be-4c65-a5de-e4a2a9e728e7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-r9x5k"
Sep 17 17:36:53.277502 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:53.277477    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:53.383201 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:53.383166    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-r9x5k"
Sep 17 17:36:53.701089 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:53.699366    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-r9x5k"]
Sep 17 17:36:53.728507 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:53.728462    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-r9x5k" event={"ID":"5f71b6c1-e6be-4c65-a5de-e4a2a9e728e7","Type":"ContainerStarted","Data":"dbcf0c96d224170e8ee87e0b7b49071194879899ea6138d4bf5af2b04b091b58"}
Sep 17 17:36:53.730847 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:53.730809    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86b4799dc9-k6whf" event={"ID":"2fc0acd7-708f-44ab-83c5-bb6fa7891010","Type":"ContainerStarted","Data":"910874de1bffbcd57c66e15242b81c572e0eac51402d1737db3438f886aa7cd7"}
Sep 17 17:36:54.927704 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:54.927365    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:55.023376 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:55.023343    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:58.752930 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:58.752889    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86b4799dc9-k6whf" event={"ID":"2fc0acd7-708f-44ab-83c5-bb6fa7891010","Type":"ContainerStarted","Data":"307a62092062bda243d758c019a832850513ecfba846cae57991f0daa9d5f125"}
Sep 17 17:36:58.754670 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:58.754640    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-r9x5k" event={"ID":"5f71b6c1-e6be-4c65-a5de-e4a2a9e728e7","Type":"ContainerStarted","Data":"d5398c4fb4e20ec297c64a0a613e3d3493e84b0d158aa017799def4093e67fd6"}
Sep 17 17:36:58.754856 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:58.754835    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-r9x5k"
Sep 17 17:36:58.761284 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:58.761260    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-r9x5k"
Sep 17 17:36:58.778731 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:58.778682    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86b4799dc9-k6whf" podStartSLOduration=2.713673472 podStartE2EDuration="7.778665228s" totalImagesPullingTime="5.064991756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:58.777888203 +0000 UTC m=+194.106987833" watchObservedRunningTime="2026-09-17 17:36:58.778665228 +0000 UTC m=+194.107764856"
Sep 17 17:36:58.796034 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:58.795554    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-r9x5k" podStartSLOduration=1.729555166 podStartE2EDuration="5.795538393s" totalImagesPullingTime="4.065983227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:58.794712408 +0000 UTC m=+194.123812034" watchObservedRunningTime="2026-09-17 17:36:58.795538393 +0000 UTC m=+194.124638021"
Sep 17 17:36:59.108124 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:59.108082    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:59.187180 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:59.187145    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-4q5r7"]
Sep 17 17:36:59.191459 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:59.191434    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4q5r7"
Sep 17 17:36:59.192568 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:59.192539    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:59.194206 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:59.194180    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 17:36:59.194206 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:59.194200    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 17:36:59.195071 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:59.195028    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8rklp\""
Sep 17 17:36:59.195210 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:59.195105    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 17:36:59.202837 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:59.202815    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-4q5r7"]
Sep 17 17:36:59.219728 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:59.219692    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-4q5r7\" (UID: \"c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4q5r7"
Sep 17 17:36:59.219837 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:59.219815    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgmxc\" (UniqueName: \"kubernetes.io/projected/c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9-kube-api-access-fgmxc\") pod \"prometheus-operator-7bd5d56d6-4q5r7\" (UID: \"c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4q5r7"
Sep 17 17:36:59.219900 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:59.219862    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-4q5r7\" (UID: \"c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4q5r7"
Sep 17 17:36:59.219958 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:59.219912    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-4q5r7\" (UID: \"c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4q5r7"
Sep 17 17:36:59.272926 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:59.272901    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:59.320803 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:59.320773    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgmxc\" (UniqueName: \"kubernetes.io/projected/c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9-kube-api-access-fgmxc\") pod \"prometheus-operator-7bd5d56d6-4q5r7\" (UID: \"c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4q5r7"
Sep 17 17:36:59.320908 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:59.320817    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-4q5r7\" (UID: \"c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4q5r7"
Sep 17 17:36:59.320908 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:59.320857    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-4q5r7\" (UID: \"c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4q5r7"
Sep 17 17:36:59.320908 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:59.320903    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-4q5r7\" (UID: \"c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4q5r7"
Sep 17 17:36:59.321663 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:59.321628    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-4q5r7\" (UID: \"c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4q5r7"
Sep 17 17:36:59.323627 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:59.323607    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-4q5r7\" (UID: \"c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4q5r7"
Sep 17 17:36:59.323726 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:59.323648    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-4q5r7\" (UID: \"c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4q5r7"
Sep 17 17:36:59.331508 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:59.331485    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgmxc\" (UniqueName: \"kubernetes.io/projected/c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9-kube-api-access-fgmxc\") pod \"prometheus-operator-7bd5d56d6-4q5r7\" (UID: \"c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4q5r7"
Sep 17 17:36:59.373005 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:59.372941    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:59.502791 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:36:59.502762    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4q5r7"
Sep 17 17:37:00.122093 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:00.122027    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:00.190299 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:00.190257    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:00.712108 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:00.712071    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d7dd577c7-62nv5"]
Sep 17 17:37:00.715253 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:00.715227    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:00.724567 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:00.724548    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 17:37:00.728574 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:00.728550    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d7dd577c7-62nv5"]
Sep 17 17:37:00.731560 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:00.731532    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-console-config\") pod \"console-5d7dd577c7-62nv5\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") " pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:00.731678 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:00.731578    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-service-ca\") pod \"console-5d7dd577c7-62nv5\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") " pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:00.731678 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:00.731613    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvnfn\" (UniqueName: \"kubernetes.io/projected/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-kube-api-access-vvnfn\") pod \"console-5d7dd577c7-62nv5\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") " pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:00.731797 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:00.731698    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-console-oauth-config\") pod \"console-5d7dd577c7-62nv5\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") " pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:00.731797 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:00.731736    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-trusted-ca-bundle\") pod \"console-5d7dd577c7-62nv5\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") " pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:00.731797 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:00.731791    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-oauth-serving-cert\") pod \"console-5d7dd577c7-62nv5\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") " pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:00.731910 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:00.731851    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-console-serving-cert\") pod \"console-5d7dd577c7-62nv5\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") " pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:00.832468 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:00.832435    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-console-config\") pod \"console-5d7dd577c7-62nv5\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") " pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:00.832640 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:00.832482    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-service-ca\") pod \"console-5d7dd577c7-62nv5\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") " pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:00.832703 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:00.832674    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvnfn\" (UniqueName: \"kubernetes.io/projected/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-kube-api-access-vvnfn\") pod \"console-5d7dd577c7-62nv5\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") " pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:00.832759 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:00.832721    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-console-oauth-config\") pod \"console-5d7dd577c7-62nv5\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") " pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:00.832759 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:00.832742    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-trusted-ca-bundle\") pod \"console-5d7dd577c7-62nv5\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") " pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:00.832865 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:00.832766    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-oauth-serving-cert\") pod \"console-5d7dd577c7-62nv5\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") " pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:00.832865 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:00.832793    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-console-serving-cert\") pod \"console-5d7dd577c7-62nv5\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") " pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:00.833319 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:00.833293    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-service-ca\") pod \"console-5d7dd577c7-62nv5\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") " pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:00.833457 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:00.833355    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-console-config\") pod \"console-5d7dd577c7-62nv5\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") " pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:00.833589 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:00.833558    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-oauth-serving-cert\") pod \"console-5d7dd577c7-62nv5\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") " pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:00.833760 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:00.833733    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-trusted-ca-bundle\") pod \"console-5d7dd577c7-62nv5\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") " pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:00.835566 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:00.835541    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-console-oauth-config\") pod \"console-5d7dd577c7-62nv5\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") " pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:00.835685 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:00.835665    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-console-serving-cert\") pod \"console-5d7dd577c7-62nv5\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") " pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:00.849244 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:00.849218    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvnfn\" (UniqueName: \"kubernetes.io/projected/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-kube-api-access-vvnfn\") pod \"console-5d7dd577c7-62nv5\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") " pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:01.026758 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:01.026669    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:01.415173 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:01.415130    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:01.504422 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:01.504392    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:02.254032 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:02.253985    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86b4799dc9-k6whf"
Sep 17 17:37:02.254224 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:02.254107    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86b4799dc9-k6whf"
Sep 17 17:37:02.259425 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:02.259399    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86b4799dc9-k6whf"
Sep 17 17:37:02.774699 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:02.774667    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86b4799dc9-k6whf"
Sep 17 17:37:05.035069 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:05.035020    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:05.135982 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:05.135937    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:05.237999 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:05.237957    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:05.553389 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:05.553331    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-4q5r7"]
Sep 17 17:37:05.554694 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:05.554669    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d7dd577c7-62nv5"]
Sep 17 17:37:05.556219 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:37:05.555971    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc67049c7_bfd5_4e2f_b68a_03e00ac9a5f9.slice/crio-687084b114448ce0ad82c3a0156d39fd4a75cd8b96d9f9df95fedc159a002416 WatchSource:0}: Error finding container 687084b114448ce0ad82c3a0156d39fd4a75cd8b96d9f9df95fedc159a002416: Status 404 returned error can't find the container with id 687084b114448ce0ad82c3a0156d39fd4a75cd8b96d9f9df95fedc159a002416
Sep 17 17:37:05.556771 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:37:05.556738    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2f0ba9_7b23_4965_8b35_2441ef9283eb.slice/crio-d763c292c4e970d8c26426cc90754339346e2a9be732848e19a7f1d8ce41df8b WatchSource:0}: Error finding container d763c292c4e970d8c26426cc90754339346e2a9be732848e19a7f1d8ce41df8b: Status 404 returned error can't find the container with id d763c292c4e970d8c26426cc90754339346e2a9be732848e19a7f1d8ce41df8b
Sep 17 17:37:05.592246 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:05.592211    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:05.693117 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:05.693077    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:05.783399 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:05.783363    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4q5r7" event={"ID":"c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9","Type":"ContainerStarted","Data":"687084b114448ce0ad82c3a0156d39fd4a75cd8b96d9f9df95fedc159a002416"}
Sep 17 17:37:05.785422 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:05.785390    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-wt58c" event={"ID":"efe4cd38-a36c-445f-aeb9-0f7167b306ec","Type":"ContainerStarted","Data":"9770647f2c79fe4c1483024834444df3bbd04623b1fc832e14114394cbe794b5"}
Sep 17 17:37:05.785590 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:05.785565    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-wt58c"
Sep 17 17:37:05.787869 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:05.787846    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d7dd577c7-62nv5" event={"ID":"0a2f0ba9-7b23-4965-8b35-2441ef9283eb","Type":"ContainerStarted","Data":"0b1362d62167a8063753d92e7a3211b095ad9c13892b5731786de449f09e4c95"}
Sep 17 17:37:05.787990 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:05.787895    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d7dd577c7-62nv5" event={"ID":"0a2f0ba9-7b23-4965-8b35-2441ef9283eb","Type":"ContainerStarted","Data":"d763c292c4e970d8c26426cc90754339346e2a9be732848e19a7f1d8ce41df8b"}
Sep 17 17:37:05.802286 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:05.802265    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-wt58c"
Sep 17 17:37:05.810010 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:05.809944    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-wt58c" podStartSLOduration=1.623641823 podStartE2EDuration="18.809922992s" totalImagesPullingTime="17.186281169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:05.808106431 +0000 UTC m=+201.137206056" watchObservedRunningTime="2026-09-17 17:37:05.809922992 +0000 UTC m=+201.139022616"
Sep 17 17:37:05.855530 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:05.855479    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d7dd577c7-62nv5" podStartSLOduration=5.855462936 podStartE2EDuration="5.855462936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:05.85450812 +0000 UTC m=+201.183607745" watchObservedRunningTime="2026-09-17 17:37:05.855462936 +0000 UTC m=+201.184562561"
Sep 17 17:37:07.620809 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:07.620774    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:07.799393 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:07.799338    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4q5r7" event={"ID":"c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9","Type":"ContainerStarted","Data":"fdc32eb9d30a340cbb981c25ec1db82bacb3898e63917474df06959dd7a40979"}
Sep 17 17:37:07.799557 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:07.799426    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4q5r7" event={"ID":"c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9","Type":"ContainerStarted","Data":"a020e6d26c03df645f0880c3b4b5016fb8200bc41213b9b9d8b19a59c1f5938c"}
Sep 17 17:37:07.821367 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:07.821311    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4q5r7" podStartSLOduration=7.098830667 podStartE2EDuration="8.821296194s" totalImagesPullingTime="1.722465527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:07.820653179 +0000 UTC m=+203.149752804" watchObservedRunningTime="2026-09-17 17:37:07.821296194 +0000 UTC m=+203.150395817"
Sep 17 17:37:08.329404 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:08.329361    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:08.395923 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:08.395891    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:09.138707 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.138670    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:09.246959 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.246920    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:09.345277 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.345240    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:09.593598 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.593560    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:09.634568 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.634531    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-95v8k"]
Sep 17 17:37:09.656834 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.656805    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-js28h"]
Sep 17 17:37:09.656989 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.656929    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-95v8k"
Sep 17 17:37:09.662182 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.662148    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 17:37:09.662314 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.662195    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 17:37:09.662314 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.662255    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 17:37:09.662314 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.662148    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xmmgr\""
Sep 17 17:37:09.670152 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.670132    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-95v8k"]
Sep 17 17:37:09.670259 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.670244    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.673366 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.673340    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 17:37:09.673511 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.673489    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 17:37:09.673828 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.673811    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 17:37:09.677942 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.677923    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dq864\""
Sep 17 17:37:09.689458 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.689426    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:09.704953 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.704928    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cdaea3b1-196c-426f-8823-83e199793607-node-exporter-textfile\") pod \"node-exporter-js28h\" (UID: \"cdaea3b1-196c-426f-8823-83e199793607\") " pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.705047 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.704969    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59cjq\" (UniqueName: \"kubernetes.io/projected/d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea-kube-api-access-59cjq\") pod \"kube-state-metrics-84c885658c-95v8k\" (UID: \"d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-95v8k"
Sep 17 17:37:09.705047 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.704997    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cdaea3b1-196c-426f-8823-83e199793607-node-exporter-accelerators-collector-config\") pod \"node-exporter-js28h\" (UID: \"cdaea3b1-196c-426f-8823-83e199793607\") " pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.705151 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.705066    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cdaea3b1-196c-426f-8823-83e199793607-sys\") pod \"node-exporter-js28h\" (UID: \"cdaea3b1-196c-426f-8823-83e199793607\") " pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.705151 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.705124    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-95v8k\" (UID: \"d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-95v8k"
Sep 17 17:37:09.705248 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.705156    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-95v8k\" (UID: \"d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-95v8k"
Sep 17 17:37:09.705248 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.705220    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-95v8k\" (UID: \"d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-95v8k"
Sep 17 17:37:09.705330 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.705283    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdaea3b1-196c-426f-8823-83e199793607-metrics-client-ca\") pod \"node-exporter-js28h\" (UID: \"cdaea3b1-196c-426f-8823-83e199793607\") " pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.705383 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.705326    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cdaea3b1-196c-426f-8823-83e199793607-node-exporter-tls\") pod \"node-exporter-js28h\" (UID: \"cdaea3b1-196c-426f-8823-83e199793607\") " pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.705383 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.705357    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-95v8k\" (UID: \"d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-95v8k"
Sep 17 17:37:09.705467 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.705384    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cdaea3b1-196c-426f-8823-83e199793607-root\") pod \"node-exporter-js28h\" (UID: \"cdaea3b1-196c-426f-8823-83e199793607\") " pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.705467 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.705442    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-95v8k\" (UID: \"d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-95v8k"
Sep 17 17:37:09.705553 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.705471    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cdaea3b1-196c-426f-8823-83e199793607-node-exporter-wtmp\") pod \"node-exporter-js28h\" (UID: \"cdaea3b1-196c-426f-8823-83e199793607\") " pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.705553 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.705507    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cdaea3b1-196c-426f-8823-83e199793607-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-js28h\" (UID: \"cdaea3b1-196c-426f-8823-83e199793607\") " pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.705659 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.705565    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7k9t\" (UniqueName: \"kubernetes.io/projected/cdaea3b1-196c-426f-8823-83e199793607-kube-api-access-p7k9t\") pod \"node-exporter-js28h\" (UID: \"cdaea3b1-196c-426f-8823-83e199793607\") " pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.769887 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.769858    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:09.806276 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.806244    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cdaea3b1-196c-426f-8823-83e199793607-node-exporter-textfile\") pod \"node-exporter-js28h\" (UID: \"cdaea3b1-196c-426f-8823-83e199793607\") " pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.806390 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.806293    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59cjq\" (UniqueName: \"kubernetes.io/projected/d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea-kube-api-access-59cjq\") pod \"kube-state-metrics-84c885658c-95v8k\" (UID: \"d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-95v8k"
Sep 17 17:37:09.806390 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.806327    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cdaea3b1-196c-426f-8823-83e199793607-node-exporter-accelerators-collector-config\") pod \"node-exporter-js28h\" (UID: \"cdaea3b1-196c-426f-8823-83e199793607\") " pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.806390 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.806360    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cdaea3b1-196c-426f-8823-83e199793607-sys\") pod \"node-exporter-js28h\" (UID: \"cdaea3b1-196c-426f-8823-83e199793607\") " pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.806553 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.806422    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-95v8k\" (UID: \"d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-95v8k"
Sep 17 17:37:09.806553 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.806495    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-95v8k\" (UID: \"d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-95v8k"
Sep 17 17:37:09.806659 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.806560    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cdaea3b1-196c-426f-8823-83e199793607-sys\") pod \"node-exporter-js28h\" (UID: \"cdaea3b1-196c-426f-8823-83e199793607\") " pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.806659 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.806562    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-95v8k\" (UID: \"d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-95v8k"
Sep 17 17:37:09.806659 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.806617    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cdaea3b1-196c-426f-8823-83e199793607-node-exporter-textfile\") pod \"node-exporter-js28h\" (UID: \"cdaea3b1-196c-426f-8823-83e199793607\") " pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.806659 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.806616    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdaea3b1-196c-426f-8823-83e199793607-metrics-client-ca\") pod \"node-exporter-js28h\" (UID: \"cdaea3b1-196c-426f-8823-83e199793607\") " pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.806864 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.806666    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cdaea3b1-196c-426f-8823-83e199793607-node-exporter-tls\") pod \"node-exporter-js28h\" (UID: \"cdaea3b1-196c-426f-8823-83e199793607\") " pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.806864 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.806700    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-95v8k\" (UID: \"d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-95v8k"
Sep 17 17:37:09.806864 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.806724    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cdaea3b1-196c-426f-8823-83e199793607-root\") pod \"node-exporter-js28h\" (UID: \"cdaea3b1-196c-426f-8823-83e199793607\") " pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.806864 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.806770    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-95v8k\" (UID: \"d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-95v8k"
Sep 17 17:37:09.806864 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.806800    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cdaea3b1-196c-426f-8823-83e199793607-node-exporter-wtmp\") pod \"node-exporter-js28h\" (UID: \"cdaea3b1-196c-426f-8823-83e199793607\") " pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.806864 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.806835    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cdaea3b1-196c-426f-8823-83e199793607-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-js28h\" (UID: \"cdaea3b1-196c-426f-8823-83e199793607\") " pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.806864 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.806863    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7k9t\" (UniqueName: \"kubernetes.io/projected/cdaea3b1-196c-426f-8823-83e199793607-kube-api-access-p7k9t\") pod \"node-exporter-js28h\" (UID: \"cdaea3b1-196c-426f-8823-83e199793607\") " pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.807228 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.807077    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-95v8k\" (UID: \"d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-95v8k"
Sep 17 17:37:09.807582 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.807295    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-95v8k\" (UID: \"d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-95v8k"
Sep 17 17:37:09.807582 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.807447    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cdaea3b1-196c-426f-8823-83e199793607-node-exporter-wtmp\") pod \"node-exporter-js28h\" (UID: \"cdaea3b1-196c-426f-8823-83e199793607\") " pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.810664 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.808131    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cdaea3b1-196c-426f-8823-83e199793607-root\") pod \"node-exporter-js28h\" (UID: \"cdaea3b1-196c-426f-8823-83e199793607\") " pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.810664 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.808573    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-95v8k\" (UID: \"d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-95v8k"
Sep 17 17:37:09.810664 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.808684    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cdaea3b1-196c-426f-8823-83e199793607-node-exporter-accelerators-collector-config\") pod \"node-exporter-js28h\" (UID: \"cdaea3b1-196c-426f-8823-83e199793607\") " pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.810664 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.809019    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdaea3b1-196c-426f-8823-83e199793607-metrics-client-ca\") pod \"node-exporter-js28h\" (UID: \"cdaea3b1-196c-426f-8823-83e199793607\") " pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.810664 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.809440    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-95v8k\" (UID: \"d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-95v8k"
Sep 17 17:37:09.819511 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.819482    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-95v8k\" (UID: \"d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-95v8k"
Sep 17 17:37:09.819922 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.819902    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cdaea3b1-196c-426f-8823-83e199793607-node-exporter-tls\") pod \"node-exporter-js28h\" (UID: \"cdaea3b1-196c-426f-8823-83e199793607\") " pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.820092 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.820046    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cdaea3b1-196c-426f-8823-83e199793607-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-js28h\" (UID: \"cdaea3b1-196c-426f-8823-83e199793607\") " pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.824529 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.824495    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59cjq\" (UniqueName: \"kubernetes.io/projected/d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea-kube-api-access-59cjq\") pod \"kube-state-metrics-84c885658c-95v8k\" (UID: \"d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-95v8k"
Sep 17 17:37:09.829000 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.828976    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7k9t\" (UniqueName: \"kubernetes.io/projected/cdaea3b1-196c-426f-8823-83e199793607-kube-api-access-p7k9t\") pod \"node-exporter-js28h\" (UID: \"cdaea3b1-196c-426f-8823-83e199793607\") " pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:09.910774 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.910691    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:09.968621 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.968595    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-95v8k"
Sep 17 17:37:09.979756 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:09.979730    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-js28h"
Sep 17 17:37:10.025019 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:10.024956    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:10.034695 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:37:10.034665    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdaea3b1_196c_426f_8823_83e199793607.slice/crio-656fdd03595223849a97b526f72b2fa2b5e0b26ccb3ad59f1fd852f4e0382cf7 WatchSource:0}: Error finding container 656fdd03595223849a97b526f72b2fa2b5e0b26ccb3ad59f1fd852f4e0382cf7: Status 404 returned error can't find the container with id 656fdd03595223849a97b526f72b2fa2b5e0b26ccb3ad59f1fd852f4e0382cf7
Sep 17 17:37:10.145881 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:10.145850    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:10.257455 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:10.257382    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:10.317047 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:10.317016    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-95v8k"]
Sep 17 17:37:10.320615 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:37:10.320590    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f5d75e_3ef8_4fe7_aa57_db82c43f48ea.slice/crio-46eadde3906e57af2eaba7150cf2b37f62a579692f17f937c489ab5ef117670e WatchSource:0}: Error finding container 46eadde3906e57af2eaba7150cf2b37f62a579692f17f937c489ab5ef117670e: Status 404 returned error can't find the container with id 46eadde3906e57af2eaba7150cf2b37f62a579692f17f937c489ab5ef117670e
Sep 17 17:37:10.362257 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:10.362228    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:10.698410 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:10.698356    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:10.815589 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:10.815552    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-95v8k" event={"ID":"d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea","Type":"ContainerStarted","Data":"46eadde3906e57af2eaba7150cf2b37f62a579692f17f937c489ab5ef117670e"}
Sep 17 17:37:10.819025 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:10.818807    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-js28h" event={"ID":"cdaea3b1-196c-426f-8823-83e199793607","Type":"ContainerStarted","Data":"656fdd03595223849a97b526f72b2fa2b5e0b26ccb3ad59f1fd852f4e0382cf7"}
Sep 17 17:37:10.825702 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:10.825672    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:11.028066 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:11.027794    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:11.029605 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:11.028299    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:11.035955 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:11.035718    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:11.073276 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:11.072869    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:11.829412 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:11.829300    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:11.896928 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:11.896862    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86b4799dc9-k6whf"]
Sep 17 17:37:12.050394 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:12.050315    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:12.149201 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:12.149159    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:12.232916 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:12.232875    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:12.759562 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:12.759531    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:12.831094 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:12.830948    2574 generic.go:356] "Generic (PLEG): container finished" podID="cdaea3b1-196c-426f-8823-83e199793607" containerID="1a055cdd99972ca719c1107aae586b02de26d84a2f2eaba43465a188503ff293" exitCode=0
Sep 17 17:37:12.831094 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:12.831041    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-js28h" event={"ID":"cdaea3b1-196c-426f-8823-83e199793607","Type":"ContainerDied","Data":"1a055cdd99972ca719c1107aae586b02de26d84a2f2eaba43465a188503ff293"}
Sep 17 17:37:12.834171 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:12.834046    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-95v8k" event={"ID":"d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea","Type":"ContainerStarted","Data":"8e03365e58fe1d44e73c48147e55e648863c9ceeacc490c4ab3dd0b5c026c550"}
Sep 17 17:37:12.834171 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:12.834123    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-95v8k" event={"ID":"d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea","Type":"ContainerStarted","Data":"b6bee9b4e740c1bcd5e150292b90c58ab421108dae0e7f004a1de10a61440ee1"}
Sep 17 17:37:12.877648 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:12.877487    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:13.166900 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:13.166868    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:13.840527 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:13.840440    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-95v8k" event={"ID":"d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea","Type":"ContainerStarted","Data":"a4c280ba8bd6760df690702231543ef5ae1ef2ec1b44cde95eee8b00474a0692"}
Sep 17 17:37:13.843425 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:13.843387    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-js28h" event={"ID":"cdaea3b1-196c-426f-8823-83e199793607","Type":"ContainerStarted","Data":"03c1458859bcc929e5ee5f86335e9942c6b863a16127123e43b6538998a19643"}
Sep 17 17:37:13.843547 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:13.843460    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-js28h" event={"ID":"cdaea3b1-196c-426f-8823-83e199793607","Type":"ContainerStarted","Data":"610a1d2874345463e1203fd07cf200d5bf63aa8cd5d5f66c3473bd3a56006f9e"}
Sep 17 17:37:13.865209 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:13.865137    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-95v8k" podStartSLOduration=2.679302627 podStartE2EDuration="4.865118116s" totalImagesPullingTime="2.185815489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:13.863851717 +0000 UTC m=+209.192951341" watchObservedRunningTime="2026-09-17 17:37:13.865118116 +0000 UTC m=+209.194217741"
Sep 17 17:37:13.887336 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:13.887292    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-js28h" podStartSLOduration=2.601948483 podStartE2EDuration="4.887278108s" totalImagesPullingTime="2.212784763s" totalInitContainerRuntime="72.544862ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:13.887014252 +0000 UTC m=+209.216113878" watchObservedRunningTime="2026-09-17 17:37:13.887278108 +0000 UTC m=+209.216377736"
Sep 17 17:37:14.439862 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:14.439820    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:14.547598 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:14.547535    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:14.646250 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:14.646217    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:14.925206 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:14.925171    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:16.008018 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.007985    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:37:16.010302 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.010262    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:16.051352 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.051318    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:37:16.051486 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.051469    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.054129 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.054106    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 17:37:16.054307 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.054199    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 17:37:16.055255 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.055233    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 17:37:16.055421 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.055250    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 17:37:16.055531 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.055296    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4p89r\""
Sep 17 17:37:16.055531 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.055338    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 17:37:16.055531 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.055338    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 17:37:16.055692 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.055360    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 17:37:16.055957 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.055939    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 17:37:16.056360 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.056339    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 17:37:16.056485 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.056470    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 17:37:16.057475 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.057455    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e881d9pi6q0f0\""
Sep 17 17:37:16.058461 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.058443    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 17:37:16.058589 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.058571    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 17:37:16.058589 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.058577    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 17:37:16.074935 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.074913    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 17:37:16.075486 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.075469    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 17:37:16.158933 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.158902    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0067e4a3-813f-4380-89cf-8bd5ca501766-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.159112 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.158939    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0067e4a3-813f-4380-89cf-8bd5ca501766-web-config\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.159112 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.158967    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0067e4a3-813f-4380-89cf-8bd5ca501766-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.159112 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.158988    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0067e4a3-813f-4380-89cf-8bd5ca501766-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.159112 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.159017    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0067e4a3-813f-4380-89cf-8bd5ca501766-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.159112 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.159040    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0067e4a3-813f-4380-89cf-8bd5ca501766-config\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.159112 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.159086    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0067e4a3-813f-4380-89cf-8bd5ca501766-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.159112 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.159109    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0067e4a3-813f-4380-89cf-8bd5ca501766-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.159455 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.159133    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0067e4a3-813f-4380-89cf-8bd5ca501766-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.159455 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.159176    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0067e4a3-813f-4380-89cf-8bd5ca501766-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.159455 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.159200    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0067e4a3-813f-4380-89cf-8bd5ca501766-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.159455 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.159225    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0067e4a3-813f-4380-89cf-8bd5ca501766-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.159455 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.159249    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0067e4a3-813f-4380-89cf-8bd5ca501766-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.159455 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.159297    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0067e4a3-813f-4380-89cf-8bd5ca501766-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.159455 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.159328    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0067e4a3-813f-4380-89cf-8bd5ca501766-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.159455 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.159350    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0067e4a3-813f-4380-89cf-8bd5ca501766-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.159455 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.159376    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0067e4a3-813f-4380-89cf-8bd5ca501766-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.159455 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.159406    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0067e4a3-813f-4380-89cf-8bd5ca501766-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.159455 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.159447    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8grc9\" (UniqueName: \"kubernetes.io/projected/0067e4a3-813f-4380-89cf-8bd5ca501766-kube-api-access-8grc9\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.159775 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.159472    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0067e4a3-813f-4380-89cf-8bd5ca501766-config-out\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.260406 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.260326    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0067e4a3-813f-4380-89cf-8bd5ca501766-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.260406 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.260372    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0067e4a3-813f-4380-89cf-8bd5ca501766-web-config\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.260406 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.260401    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0067e4a3-813f-4380-89cf-8bd5ca501766-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.260658 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.260422    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0067e4a3-813f-4380-89cf-8bd5ca501766-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.260658 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.260449    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0067e4a3-813f-4380-89cf-8bd5ca501766-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.260658 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.260575    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0067e4a3-813f-4380-89cf-8bd5ca501766-config\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.260658 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.260616    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0067e4a3-813f-4380-89cf-8bd5ca501766-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.260658 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.260640    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0067e4a3-813f-4380-89cf-8bd5ca501766-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.260917 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.260665    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0067e4a3-813f-4380-89cf-8bd5ca501766-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.260917 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.260695    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0067e4a3-813f-4380-89cf-8bd5ca501766-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.260917 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.260721    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0067e4a3-813f-4380-89cf-8bd5ca501766-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.260917 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.260747    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0067e4a3-813f-4380-89cf-8bd5ca501766-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.260917 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.260771    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0067e4a3-813f-4380-89cf-8bd5ca501766-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.260917 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.260799    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0067e4a3-813f-4380-89cf-8bd5ca501766-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.260917 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.260822    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0067e4a3-813f-4380-89cf-8bd5ca501766-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.260917 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.260853    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0067e4a3-813f-4380-89cf-8bd5ca501766-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.260917 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.260860    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0067e4a3-813f-4380-89cf-8bd5ca501766-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.260917 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.260877    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0067e4a3-813f-4380-89cf-8bd5ca501766-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.261436 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.260922    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0067e4a3-813f-4380-89cf-8bd5ca501766-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.261436 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.260981    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8grc9\" (UniqueName: \"kubernetes.io/projected/0067e4a3-813f-4380-89cf-8bd5ca501766-kube-api-access-8grc9\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.261436 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.261023    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0067e4a3-813f-4380-89cf-8bd5ca501766-config-out\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.261436 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.261024    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0067e4a3-813f-4380-89cf-8bd5ca501766-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.261659 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.261636    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0067e4a3-813f-4380-89cf-8bd5ca501766-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.263561 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.261969    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0067e4a3-813f-4380-89cf-8bd5ca501766-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.263561 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.262322    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0067e4a3-813f-4380-89cf-8bd5ca501766-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.263561 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.262537    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0067e4a3-813f-4380-89cf-8bd5ca501766-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.264082 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.263959    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0067e4a3-813f-4380-89cf-8bd5ca501766-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.264367 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.264333    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0067e4a3-813f-4380-89cf-8bd5ca501766-config\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.264457 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.264399    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0067e4a3-813f-4380-89cf-8bd5ca501766-web-config\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.264694 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.264672    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0067e4a3-813f-4380-89cf-8bd5ca501766-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.264863 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.264843    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0067e4a3-813f-4380-89cf-8bd5ca501766-config-out\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.264863 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.264847    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0067e4a3-813f-4380-89cf-8bd5ca501766-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.265302 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.265275    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0067e4a3-813f-4380-89cf-8bd5ca501766-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.265627 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.265604    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0067e4a3-813f-4380-89cf-8bd5ca501766-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.266800 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.266775    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0067e4a3-813f-4380-89cf-8bd5ca501766-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.266897 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.266887    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0067e4a3-813f-4380-89cf-8bd5ca501766-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.266963 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.266924    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0067e4a3-813f-4380-89cf-8bd5ca501766-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.267210 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.267191    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0067e4a3-813f-4380-89cf-8bd5ca501766-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.267998 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.267981    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0067e4a3-813f-4380-89cf-8bd5ca501766-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.272312 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.272289    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8grc9\" (UniqueName: \"kubernetes.io/projected/0067e4a3-813f-4380-89cf-8bd5ca501766-kube-api-access-8grc9\") pod \"prometheus-k8s-0\" (UID: \"0067e4a3-813f-4380-89cf-8bd5ca501766\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.366515 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.366485    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.684862 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.684837    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:37:16.686985 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:37:16.686954    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0067e4a3_813f_4380_89cf_8bd5ca501766.slice/crio-87e52d6583737a5485d89a173144bc9d4ba5ef86f038f6b55ce92bb863e8ac93 WatchSource:0}: Error finding container 87e52d6583737a5485d89a173144bc9d4ba5ef86f038f6b55ce92bb863e8ac93: Status 404 returned error can't find the container with id 87e52d6583737a5485d89a173144bc9d4ba5ef86f038f6b55ce92bb863e8ac93
Sep 17 17:37:16.857064 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:16.857015    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0067e4a3-813f-4380-89cf-8bd5ca501766","Type":"ContainerStarted","Data":"87e52d6583737a5485d89a173144bc9d4ba5ef86f038f6b55ce92bb863e8ac93"}
Sep 17 17:37:18.867702 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:18.867668    2574 generic.go:356] "Generic (PLEG): container finished" podID="0067e4a3-813f-4380-89cf-8bd5ca501766" containerID="b845666462bc768fbc509415f84927a762de123cbe6a2d3ab04e92edd58755e9" exitCode=0
Sep 17 17:37:18.868216 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:18.867752    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0067e4a3-813f-4380-89cf-8bd5ca501766","Type":"ContainerDied","Data":"b845666462bc768fbc509415f84927a762de123cbe6a2d3ab04e92edd58755e9"}
Sep 17 17:37:22.229917 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:22.229867    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:22.331927 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:22.331893    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:22.887084 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:22.886443    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0067e4a3-813f-4380-89cf-8bd5ca501766","Type":"ContainerStarted","Data":"03b265ff2061f548218e031452d13c8d83175f79b53d94c42600717dfbbba953"}
Sep 17 17:37:22.887084 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:22.886528    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0067e4a3-813f-4380-89cf-8bd5ca501766","Type":"ContainerStarted","Data":"2e545f613b28490067b3d16888bfe287b679b934d49a55097bfa88d8232babf4"}
Sep 17 17:37:25.671568 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:25.671493    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d7dd577c7-62nv5"]
Sep 17 17:37:25.812945 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:25.812914    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:25.900548 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:25.900521    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0067e4a3-813f-4380-89cf-8bd5ca501766","Type":"ContainerStarted","Data":"fd03fb9eedcfd3c2142cea2cdc17f852e5c601bfee29d4b20ddf9e8d3c78ea72"}
Sep 17 17:37:25.900753 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:25.900592    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0067e4a3-813f-4380-89cf-8bd5ca501766","Type":"ContainerStarted","Data":"628bdea2fbbed056af080f0533677350503812842a2250f88ed02382f11bd2a2"}
Sep 17 17:37:25.900753 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:25.900629    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0067e4a3-813f-4380-89cf-8bd5ca501766","Type":"ContainerStarted","Data":"d0b48d681a80db4fb5807009e0640285ade7380aa0574138eb74073842e5bd43"}
Sep 17 17:37:25.900753 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:25.900673    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0067e4a3-813f-4380-89cf-8bd5ca501766","Type":"ContainerStarted","Data":"218c92a0cec864a9dafbe11d5a2fd373d72316ed606de9758a72786f36bdf131"}
Sep 17 17:37:25.908376 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:25.908348    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:25.934888 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:25.934803    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.528018733 podStartE2EDuration="10.93478851s" totalImagesPullingTime="7.386309141s" totalInitContainerRuntime="20.460636ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:25.932396303 +0000 UTC m=+221.261495927" watchObservedRunningTime="2026-09-17 17:37:25.93478851 +0000 UTC m=+221.263888173"
Sep 17 17:37:25.997883 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:25.997856    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:26.367161 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:26.367138    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:27.283892 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:27.283858    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:27.386125 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:27.386094    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:32.929125 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:32.929091    2574 generic.go:356] "Generic (PLEG): container finished" podID="fb25ecfe-4234-42bb-9148-caf3f6d6fb29" containerID="339618ef531af15c262fd19a6a2fac61dd04963b618f58a2f6b2fd31bd7ae910" exitCode=0
Sep 17 17:37:32.929567 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:32.929170    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hlzfx" event={"ID":"fb25ecfe-4234-42bb-9148-caf3f6d6fb29","Type":"ContainerDied","Data":"339618ef531af15c262fd19a6a2fac61dd04963b618f58a2f6b2fd31bd7ae910"}
Sep 17 17:37:32.929640 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:32.929587    2574 scope.go:117] "RemoveContainer" containerID="339618ef531af15c262fd19a6a2fac61dd04963b618f58a2f6b2fd31bd7ae910"
Sep 17 17:37:33.934493 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:33.934457    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-hlzfx" event={"ID":"fb25ecfe-4234-42bb-9148-caf3f6d6fb29","Type":"ContainerStarted","Data":"310a6e2cd4aebd76ca82d0dec3c27725b5e8437e969eb88b40e2eb6bf4a76e20"}
Sep 17 17:37:36.924998 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:36.924962    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-86b4799dc9-k6whf" podUID="2fc0acd7-708f-44ab-83c5-bb6fa7891010" containerName="console" containerID="cri-o://307a62092062bda243d758c019a832850513ecfba846cae57991f0daa9d5f125" gracePeriod=15
Sep 17 17:37:37.202560 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.202502    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86b4799dc9-k6whf"
Sep 17 17:37:37.338035 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.338006    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gl7dn\" (UniqueName: \"kubernetes.io/projected/2fc0acd7-708f-44ab-83c5-bb6fa7891010-kube-api-access-gl7dn\") pod \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\" (UID: \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\") "
Sep 17 17:37:37.338184 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.338076    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc0acd7-708f-44ab-83c5-bb6fa7891010-service-ca\") pod \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\" (UID: \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\") "
Sep 17 17:37:37.338184 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.338108    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fc0acd7-708f-44ab-83c5-bb6fa7891010-console-config\") pod \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\" (UID: \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\") "
Sep 17 17:37:37.338184 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.338132    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fc0acd7-708f-44ab-83c5-bb6fa7891010-oauth-serving-cert\") pod \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\" (UID: \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\") "
Sep 17 17:37:37.338402 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.338169    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fc0acd7-708f-44ab-83c5-bb6fa7891010-console-serving-cert\") pod \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\" (UID: \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\") "
Sep 17 17:37:37.338402 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.338242    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fc0acd7-708f-44ab-83c5-bb6fa7891010-console-oauth-config\") pod \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\" (UID: \"2fc0acd7-708f-44ab-83c5-bb6fa7891010\") "
Sep 17 17:37:37.338570 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.338540    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fc0acd7-708f-44ab-83c5-bb6fa7891010-service-ca" (OuterVolumeSpecName: "service-ca") pod "2fc0acd7-708f-44ab-83c5-bb6fa7891010" (UID: "2fc0acd7-708f-44ab-83c5-bb6fa7891010"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:37.338630 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.338544    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fc0acd7-708f-44ab-83c5-bb6fa7891010-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2fc0acd7-708f-44ab-83c5-bb6fa7891010" (UID: "2fc0acd7-708f-44ab-83c5-bb6fa7891010"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:37.338694 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.338637    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fc0acd7-708f-44ab-83c5-bb6fa7891010-console-config" (OuterVolumeSpecName: "console-config") pod "2fc0acd7-708f-44ab-83c5-bb6fa7891010" (UID: "2fc0acd7-708f-44ab-83c5-bb6fa7891010"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:37.340286 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.340265    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fc0acd7-708f-44ab-83c5-bb6fa7891010-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2fc0acd7-708f-44ab-83c5-bb6fa7891010" (UID: "2fc0acd7-708f-44ab-83c5-bb6fa7891010"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:37:37.340401 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.340378    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fc0acd7-708f-44ab-83c5-bb6fa7891010-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2fc0acd7-708f-44ab-83c5-bb6fa7891010" (UID: "2fc0acd7-708f-44ab-83c5-bb6fa7891010"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:37:37.340467 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.340436    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc0acd7-708f-44ab-83c5-bb6fa7891010-kube-api-access-gl7dn" (OuterVolumeSpecName: "kube-api-access-gl7dn") pod "2fc0acd7-708f-44ab-83c5-bb6fa7891010" (UID: "2fc0acd7-708f-44ab-83c5-bb6fa7891010"). InnerVolumeSpecName "kube-api-access-gl7dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:37:37.439851 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.439828    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc0acd7-708f-44ab-83c5-bb6fa7891010-service-ca\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:37:37.439851 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.439849    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fc0acd7-708f-44ab-83c5-bb6fa7891010-console-config\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:37:37.439990 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.439859    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fc0acd7-708f-44ab-83c5-bb6fa7891010-oauth-serving-cert\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:37:37.439990 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.439868    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fc0acd7-708f-44ab-83c5-bb6fa7891010-console-serving-cert\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:37:37.439990 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.439876    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fc0acd7-708f-44ab-83c5-bb6fa7891010-console-oauth-config\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:37:37.439990 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.439885    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl7dn\" (UniqueName: \"kubernetes.io/projected/2fc0acd7-708f-44ab-83c5-bb6fa7891010-kube-api-access-gl7dn\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:37:37.950877 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.950837    2574 generic.go:356] "Generic (PLEG): container finished" podID="2fc0acd7-708f-44ab-83c5-bb6fa7891010" containerID="307a62092062bda243d758c019a832850513ecfba846cae57991f0daa9d5f125" exitCode=0
Sep 17 17:37:37.951372 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.950892    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86b4799dc9-k6whf"
Sep 17 17:37:37.951372 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.950924    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86b4799dc9-k6whf" event={"ID":"2fc0acd7-708f-44ab-83c5-bb6fa7891010","Type":"ContainerDied","Data":"307a62092062bda243d758c019a832850513ecfba846cae57991f0daa9d5f125"}
Sep 17 17:37:37.951372 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.951004    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86b4799dc9-k6whf" event={"ID":"2fc0acd7-708f-44ab-83c5-bb6fa7891010","Type":"ContainerDied","Data":"910874de1bffbcd57c66e15242b81c572e0eac51402d1737db3438f886aa7cd7"}
Sep 17 17:37:37.951372 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.951027    2574 scope.go:117] "RemoveContainer" containerID="307a62092062bda243d758c019a832850513ecfba846cae57991f0daa9d5f125"
Sep 17 17:37:37.974255 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.974232    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86b4799dc9-k6whf"]
Sep 17 17:37:37.977041 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.977012    2574 scope.go:117] "RemoveContainer" containerID="307a62092062bda243d758c019a832850513ecfba846cae57991f0daa9d5f125"
Sep 17 17:37:37.977329 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:37:37.977308    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"307a62092062bda243d758c019a832850513ecfba846cae57991f0daa9d5f125\": container with ID starting with 307a62092062bda243d758c019a832850513ecfba846cae57991f0daa9d5f125 not found: ID does not exist" containerID="307a62092062bda243d758c019a832850513ecfba846cae57991f0daa9d5f125"
Sep 17 17:37:37.977388 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.977343    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"307a62092062bda243d758c019a832850513ecfba846cae57991f0daa9d5f125"} err="failed to get container status \"307a62092062bda243d758c019a832850513ecfba846cae57991f0daa9d5f125\": rpc error: code = NotFound desc = could not find container \"307a62092062bda243d758c019a832850513ecfba846cae57991f0daa9d5f125\": container with ID starting with 307a62092062bda243d758c019a832850513ecfba846cae57991f0daa9d5f125 not found: ID does not exist"
Sep 17 17:37:37.980173 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:37.980150    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86b4799dc9-k6whf"]
Sep 17 17:37:39.115348 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:39.115314    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fc0acd7-708f-44ab-83c5-bb6fa7891010" path="/var/lib/kubelet/pods/2fc0acd7-708f-44ab-83c5-bb6fa7891010/volumes"
Sep 17 17:37:42.285996 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:42.285963    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:42.376099 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:42.376073    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:50.690627 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:50.690590    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d7dd577c7-62nv5" podUID="0a2f0ba9-7b23-4965-8b35-2441ef9283eb" containerName="console" containerID="cri-o://0b1362d62167a8063753d92e7a3211b095ad9c13892b5731786de449f09e4c95" gracePeriod=15
Sep 17 17:37:50.956800 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:50.956745    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:51.002531 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.002504    2574 generic.go:356] "Generic (PLEG): container finished" podID="0a2f0ba9-7b23-4965-8b35-2441ef9283eb" containerID="0b1362d62167a8063753d92e7a3211b095ad9c13892b5731786de449f09e4c95" exitCode=0
Sep 17 17:37:51.002644 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.002539    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d7dd577c7-62nv5"
Sep 17 17:37:51.002644 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.002565    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d7dd577c7-62nv5" event={"ID":"0a2f0ba9-7b23-4965-8b35-2441ef9283eb","Type":"ContainerDied","Data":"0b1362d62167a8063753d92e7a3211b095ad9c13892b5731786de449f09e4c95"}
Sep 17 17:37:51.002644 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.002609    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d7dd577c7-62nv5" event={"ID":"0a2f0ba9-7b23-4965-8b35-2441ef9283eb","Type":"ContainerDied","Data":"d763c292c4e970d8c26426cc90754339346e2a9be732848e19a7f1d8ce41df8b"}
Sep 17 17:37:51.002644 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.002636    2574 scope.go:117] "RemoveContainer" containerID="0b1362d62167a8063753d92e7a3211b095ad9c13892b5731786de449f09e4c95"
Sep 17 17:37:51.011693 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.011676    2574 scope.go:117] "RemoveContainer" containerID="0b1362d62167a8063753d92e7a3211b095ad9c13892b5731786de449f09e4c95"
Sep 17 17:37:51.011966 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:37:51.011942    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b1362d62167a8063753d92e7a3211b095ad9c13892b5731786de449f09e4c95\": container with ID starting with 0b1362d62167a8063753d92e7a3211b095ad9c13892b5731786de449f09e4c95 not found: ID does not exist" containerID="0b1362d62167a8063753d92e7a3211b095ad9c13892b5731786de449f09e4c95"
Sep 17 17:37:51.012010 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.011974    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b1362d62167a8063753d92e7a3211b095ad9c13892b5731786de449f09e4c95"} err="failed to get container status \"0b1362d62167a8063753d92e7a3211b095ad9c13892b5731786de449f09e4c95\": rpc error: code = NotFound desc = could not find container \"0b1362d62167a8063753d92e7a3211b095ad9c13892b5731786de449f09e4c95\": container with ID starting with 0b1362d62167a8063753d92e7a3211b095ad9c13892b5731786de449f09e4c95 not found: ID does not exist"
Sep 17 17:37:51.041404 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.041384    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvnfn\" (UniqueName: \"kubernetes.io/projected/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-kube-api-access-vvnfn\") pod \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") "
Sep 17 17:37:51.041490 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.041413    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-trusted-ca-bundle\") pod \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") "
Sep 17 17:37:51.041490 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.041438    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-console-oauth-config\") pod \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") "
Sep 17 17:37:51.041490 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.041467    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-oauth-serving-cert\") pod \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") "
Sep 17 17:37:51.041621 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.041526    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-console-config\") pod \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") "
Sep 17 17:37:51.041621 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.041553    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-service-ca\") pod \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") "
Sep 17 17:37:51.041621 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.041589    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-console-serving-cert\") pod \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\" (UID: \"0a2f0ba9-7b23-4965-8b35-2441ef9283eb\") "
Sep 17 17:37:51.041865 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.041842    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0a2f0ba9-7b23-4965-8b35-2441ef9283eb" (UID: "0a2f0ba9-7b23-4965-8b35-2441ef9283eb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:51.041923 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.041860    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0a2f0ba9-7b23-4965-8b35-2441ef9283eb" (UID: "0a2f0ba9-7b23-4965-8b35-2441ef9283eb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:51.041923 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.041902    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-console-config" (OuterVolumeSpecName: "console-config") pod "0a2f0ba9-7b23-4965-8b35-2441ef9283eb" (UID: "0a2f0ba9-7b23-4965-8b35-2441ef9283eb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:51.042253 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.042226    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-service-ca" (OuterVolumeSpecName: "service-ca") pod "0a2f0ba9-7b23-4965-8b35-2441ef9283eb" (UID: "0a2f0ba9-7b23-4965-8b35-2441ef9283eb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:51.043649 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.043629    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0a2f0ba9-7b23-4965-8b35-2441ef9283eb" (UID: "0a2f0ba9-7b23-4965-8b35-2441ef9283eb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:37:51.043799 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.043783    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0a2f0ba9-7b23-4965-8b35-2441ef9283eb" (UID: "0a2f0ba9-7b23-4965-8b35-2441ef9283eb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:37:51.043862 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.043813    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-kube-api-access-vvnfn" (OuterVolumeSpecName: "kube-api-access-vvnfn") pod "0a2f0ba9-7b23-4965-8b35-2441ef9283eb" (UID: "0a2f0ba9-7b23-4965-8b35-2441ef9283eb"). InnerVolumeSpecName "kube-api-access-vvnfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:37:51.142899 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.142873    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-console-config\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:37:51.142899 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.142894    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-service-ca\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:37:51.143025 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.142908    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-console-serving-cert\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:37:51.143025 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.142921    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvnfn\" (UniqueName: \"kubernetes.io/projected/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-kube-api-access-vvnfn\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:37:51.143025 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.142933    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-trusted-ca-bundle\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:37:51.143025 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.142948    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-console-oauth-config\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:37:51.143025 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.142961    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a2f0ba9-7b23-4965-8b35-2441ef9283eb-oauth-serving-cert\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:37:51.319746 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.319691    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d7dd577c7-62nv5"]
Sep 17 17:37:51.327479 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:51.327449    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d7dd577c7-62nv5"]
Sep 17 17:37:53.011539 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:53.011507    2574 generic.go:356] "Generic (PLEG): container finished" podID="496b2135-6189-4a33-b138-112cfd1a57d2" containerID="bac0a623dd0823c2d552f6ae02e89b22137b1a647017d5d92ab96c4150bb51fd" exitCode=0
Sep 17 17:37:53.011929 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:53.011584    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-dqbxl" event={"ID":"496b2135-6189-4a33-b138-112cfd1a57d2","Type":"ContainerDied","Data":"bac0a623dd0823c2d552f6ae02e89b22137b1a647017d5d92ab96c4150bb51fd"}
Sep 17 17:37:53.011929 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:53.011918    2574 scope.go:117] "RemoveContainer" containerID="bac0a623dd0823c2d552f6ae02e89b22137b1a647017d5d92ab96c4150bb51fd"
Sep 17 17:37:53.115398 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:53.115375    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a2f0ba9-7b23-4965-8b35-2441ef9283eb" path="/var/lib/kubelet/pods/0a2f0ba9-7b23-4965-8b35-2441ef9283eb/volumes"
Sep 17 17:37:54.016791 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:54.016756    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-dqbxl" event={"ID":"496b2135-6189-4a33-b138-112cfd1a57d2","Type":"ContainerStarted","Data":"baeedc5126d9158fbbd710583098e60a11ea46d45fa47e8a4564d560e8fd00ec"}
Sep 17 17:37:54.134135 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:54.134099    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jvdsb_b5dd23d0-8b89-453c-99ee-dfd8a18c683c/dns-node-resolver/0.log"
Sep 17 17:37:56.887283 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:56.887245    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs\") pod \"network-metrics-daemon-kcglc\" (UID: \"c08424bd-0295-4dcb-9c57-3d5a5a6ba21d\") " pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:37:56.889638 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:56.889614    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c08424bd-0295-4dcb-9c57-3d5a5a6ba21d-metrics-certs\") pod \"network-metrics-daemon-kcglc\" (UID: \"c08424bd-0295-4dcb-9c57-3d5a5a6ba21d\") " pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:37:56.913876 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:56.913852    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-745tx\""
Sep 17 17:37:56.921930 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:56.921909    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kcglc"
Sep 17 17:37:57.212254 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:57.212185    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kcglc"]
Sep 17 17:37:57.215211 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:37:57.215183    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08424bd_0295_4dcb_9c57_3d5a5a6ba21d.slice/crio-9bcc844751fa3af0e231196b5cb7bc72d18adb752c90cc2278af9b4064feee11 WatchSource:0}: Error finding container 9bcc844751fa3af0e231196b5cb7bc72d18adb752c90cc2278af9b4064feee11: Status 404 returned error can't find the container with id 9bcc844751fa3af0e231196b5cb7bc72d18adb752c90cc2278af9b4064feee11
Sep 17 17:37:57.442704 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:57.442683    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:58.034036 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:58.033992    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kcglc" event={"ID":"c08424bd-0295-4dcb-9c57-3d5a5a6ba21d","Type":"ContainerStarted","Data":"9bcc844751fa3af0e231196b5cb7bc72d18adb752c90cc2278af9b4064feee11"}
Sep 17 17:37:58.633962 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:58.633933    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:58.717107 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:58.717085    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:59.039763 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:59.039687    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kcglc" event={"ID":"c08424bd-0295-4dcb-9c57-3d5a5a6ba21d","Type":"ContainerStarted","Data":"e3261adb54441a8087e17276dc12e028858f10712a9812dfbf31f97918613814"}
Sep 17 17:37:59.039763 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:59.039736    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kcglc" event={"ID":"c08424bd-0295-4dcb-9c57-3d5a5a6ba21d","Type":"ContainerStarted","Data":"cc6dfc6b75cb51854a1350516f07189b6176ba56c6c3fe0c4afe4d4d741b001a"}
Sep 17 17:37:59.062915 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:59.062873    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kcglc" podStartSLOduration=252.987393875 podStartE2EDuration="4m14.062857028s" totalImagesPullingTime="1.075463153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:59.061445503 +0000 UTC m=+254.390545127" watchObservedRunningTime="2026-09-17 17:37:59.062857028 +0000 UTC m=+254.391956652"
Sep 17 17:37:59.186744 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:59.186705    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:59.287412 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:37:59.287387    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:14.951986 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:14.951905    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:15.027338 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:15.027306    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:16.367679 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:16.367647    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:16.382707 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:16.382680    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:17.123557 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:17.123526    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:25.616943 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:38:25.616904    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-qpdzb" podUID="7656e584-bfe6-4f58-bdac-fe20192d5a44"
Sep 17 17:38:26.139534 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:26.139504    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qpdzb"
Sep 17 17:38:28.926426 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:28.926390    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls\") pod \"dns-default-qpdzb\" (UID: \"7656e584-bfe6-4f58-bdac-fe20192d5a44\") " pod="openshift-dns/dns-default-qpdzb"
Sep 17 17:38:28.928828 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:28.928804    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7656e584-bfe6-4f58-bdac-fe20192d5a44-metrics-tls\") pod \"dns-default-qpdzb\" (UID: \"7656e584-bfe6-4f58-bdac-fe20192d5a44\") " pod="openshift-dns/dns-default-qpdzb"
Sep 17 17:38:29.049261 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:29.049190    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:29.142992 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:29.142964    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vdp8v\""
Sep 17 17:38:29.150630 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:29.150611    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qpdzb"
Sep 17 17:38:29.163259 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:29.163237    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:29.428094 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:29.428069    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qpdzb"]
Sep 17 17:38:29.430075 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:38:29.430038    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7656e584_bfe6_4f58_bdac_fe20192d5a44.slice/crio-2ad1966cdba30ee43697c0c672c166dc512e0cf8db9f8a96459a7b4eb123605c WatchSource:0}: Error finding container 2ad1966cdba30ee43697c0c672c166dc512e0cf8db9f8a96459a7b4eb123605c: Status 404 returned error can't find the container with id 2ad1966cdba30ee43697c0c672c166dc512e0cf8db9f8a96459a7b4eb123605c
Sep 17 17:38:30.157878 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:30.157839    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qpdzb" event={"ID":"7656e584-bfe6-4f58-bdac-fe20192d5a44","Type":"ContainerStarted","Data":"2ad1966cdba30ee43697c0c672c166dc512e0cf8db9f8a96459a7b4eb123605c"}
Sep 17 17:38:31.163661 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:31.163571    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qpdzb" event={"ID":"7656e584-bfe6-4f58-bdac-fe20192d5a44","Type":"ContainerStarted","Data":"df22bd6f5c3796f4ea1e1afbf4f674a68eccaa54a5da50bd856ac61cd8172143"}
Sep 17 17:38:31.163661 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:31.163634    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qpdzb" event={"ID":"7656e584-bfe6-4f58-bdac-fe20192d5a44","Type":"ContainerStarted","Data":"b664b5f7d51ffde2594cbc3c30a5f1e27c25b7df9aef16bffa7e1c862a4ffdf1"}
Sep 17 17:38:31.164114 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:31.163723    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qpdzb"
Sep 17 17:38:31.192529 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:31.192481    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qpdzb" podStartSLOduration=251.750823854 podStartE2EDuration="4m13.192462886s" totalImagesPullingTime="1.441639032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:38:31.191565519 +0000 UTC m=+286.520665144" watchObservedRunningTime="2026-09-17 17:38:31.192462886 +0000 UTC m=+286.521562510"
Sep 17 17:38:32.456243 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:32.456214    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:37.121381 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:37.120440    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:37.237665 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:37.237635    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:37.338879 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:37.338845    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:37.845218 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:37.845192    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:41.169620 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:41.169585    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qpdzb"
Sep 17 17:38:44.380235 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:44.380204    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:44.479668 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:44.479642    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:45.005101 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:45.005048    2574 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 17:38:50.342228 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:50.342193    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:50.442202 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:50.442175    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:56.279130 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:56.279098    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:56.346896 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:56.346854    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:59.251198 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:59.251167    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:59.353160 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:59.353130    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:59.456611 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:38:59.456586    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:12.285988 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:39:12.285944    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:12.355338 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:39:12.355299    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:23.797494 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:39:23.797454    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:23.885939 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:39:23.885901    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:02.391820 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:40:02.391771    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:25.635037 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:41:25.635002    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:38.972432 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:41:38.972399    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:39.063896 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:41:39.063862    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:57.289658 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:41:57.289622    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:57.386426 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:41:57.386396    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:57.461184 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:41:57.461157    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:12.211795 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:42:12.211761    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:23.050844 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:42:23.050810    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:23.149564 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:42:23.149533    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:23.215903 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:42:23.215874    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:37.972570 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:42:37.972490    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:38.071219 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:42:38.071193    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:32.490497 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:43:32.490462    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:32.584251 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:43:32.584223    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:32.658068 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:43:32.658006    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:00.979842 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:44:00.979783    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:01.067619 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:44:01.067582    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:46.610185 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:44:46.610142    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:46.720982 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:44:46.720939    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:52.159837 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:44:52.159805    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:52.492498 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:44:52.492415    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:52.820119 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:44:52.820034    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:53.040454 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:44:53.040425    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:53.163105 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:44:53.163073    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:53.243887 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:44:53.243855    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:53.450748 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:44:53.450668    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:53.552206 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:44:53.552166    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:53.727810 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:44:53.727717    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:53.912182 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:44:53.912153    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:54.027920 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:44:54.027837    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:44.819866 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:45:44.819782    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:44.916722 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:45:44.916694    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:57.926709 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:45:57.926677    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:05.470625 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:46:05.470590    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:12.979786 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:46:12.979753    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:13.080607 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:46:13.080567    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:13.215074 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:46:13.215026    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:13.311376 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:46:13.311303    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:13.586072 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:46:13.586027    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:14.171629 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:46:14.171601    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:14.274259 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:46:14.274231    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:14.366847 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:46:14.366822    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:15.907357 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:15.907282    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:30.605570 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:30.605537    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:30.714077 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:30.714023    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:30.822426 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:30.822392    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:31.049430 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:31.049350    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:31.144805 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:31.144775    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:32.220714 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:32.220684    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:32.329454 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:32.329428    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:48.218969 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:48.218938    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-dnqsq"]
Sep 17 17:47:48.219432 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:48.219244    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a2f0ba9-7b23-4965-8b35-2441ef9283eb" containerName="console"
Sep 17 17:47:48.219432 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:48.219255    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a2f0ba9-7b23-4965-8b35-2441ef9283eb" containerName="console"
Sep 17 17:47:48.219432 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:48.219270    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fc0acd7-708f-44ab-83c5-bb6fa7891010" containerName="console"
Sep 17 17:47:48.219432 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:48.219276    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fc0acd7-708f-44ab-83c5-bb6fa7891010" containerName="console"
Sep 17 17:47:48.219432 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:48.219330    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fc0acd7-708f-44ab-83c5-bb6fa7891010" containerName="console"
Sep 17 17:47:48.219432 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:48.219338    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a2f0ba9-7b23-4965-8b35-2441ef9283eb" containerName="console"
Sep 17 17:47:48.222194 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:48.222175    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-dnqsq"
Sep 17 17:47:48.224477 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:48.224454    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 17:47:48.224600 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:48.224471    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 17:47:48.224600 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:48.224504    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 17:47:48.225410 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:48.225392    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 17:47:48.225523 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:48.225421    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-zkdvg\""
Sep 17 17:47:48.234165 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:48.234144    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-dnqsq"]
Sep 17 17:47:48.288443 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:48.288417    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsbrx\" (UniqueName: \"kubernetes.io/projected/65f40148-a803-4220-8471-527fb92b57a0-kube-api-access-hsbrx\") pod \"s3-init-dnqsq\" (UID: \"65f40148-a803-4220-8471-527fb92b57a0\") " pod="kserve/s3-init-dnqsq"
Sep 17 17:47:48.288565 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:48.288461    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65f40148-a803-4220-8471-527fb92b57a0-model-cache\") pod \"s3-init-dnqsq\" (UID: \"65f40148-a803-4220-8471-527fb92b57a0\") " pod="kserve/s3-init-dnqsq"
Sep 17 17:47:48.389194 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:48.389167    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsbrx\" (UniqueName: \"kubernetes.io/projected/65f40148-a803-4220-8471-527fb92b57a0-kube-api-access-hsbrx\") pod \"s3-init-dnqsq\" (UID: \"65f40148-a803-4220-8471-527fb92b57a0\") " pod="kserve/s3-init-dnqsq"
Sep 17 17:47:48.389299 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:48.389208    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65f40148-a803-4220-8471-527fb92b57a0-model-cache\") pod \"s3-init-dnqsq\" (UID: \"65f40148-a803-4220-8471-527fb92b57a0\") " pod="kserve/s3-init-dnqsq"
Sep 17 17:47:48.389512 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:48.389497    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65f40148-a803-4220-8471-527fb92b57a0-model-cache\") pod \"s3-init-dnqsq\" (UID: \"65f40148-a803-4220-8471-527fb92b57a0\") " pod="kserve/s3-init-dnqsq"
Sep 17 17:47:48.412213 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:48.412182    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsbrx\" (UniqueName: \"kubernetes.io/projected/65f40148-a803-4220-8471-527fb92b57a0-kube-api-access-hsbrx\") pod \"s3-init-dnqsq\" (UID: \"65f40148-a803-4220-8471-527fb92b57a0\") " pod="kserve/s3-init-dnqsq"
Sep 17 17:47:48.542492 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:48.542422    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-dnqsq"
Sep 17 17:47:48.816431 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:48.816349    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-dnqsq"]
Sep 17 17:47:48.819002 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:47:48.818975    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65f40148_a803_4220_8471_527fb92b57a0.slice/crio-ba0d3ddd23c81e40afb71351301ea7ca1da0e832ec1aab0973e3cb0713a255d7 WatchSource:0}: Error finding container ba0d3ddd23c81e40afb71351301ea7ca1da0e832ec1aab0973e3cb0713a255d7: Status 404 returned error can't find the container with id ba0d3ddd23c81e40afb71351301ea7ca1da0e832ec1aab0973e3cb0713a255d7
Sep 17 17:47:48.820934 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:48.820916    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:47:49.223695 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:49.223656    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dnqsq" event={"ID":"65f40148-a803-4220-8471-527fb92b57a0","Type":"ContainerStarted","Data":"ba0d3ddd23c81e40afb71351301ea7ca1da0e832ec1aab0973e3cb0713a255d7"}
Sep 17 17:47:52.238916 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:52.238881    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dnqsq" event={"ID":"65f40148-a803-4220-8471-527fb92b57a0","Type":"ContainerStarted","Data":"dde4065a685e913bb7745d22bc7dec469b7ac912a28727d85da8e03bccec1aae"}
Sep 17 17:47:54.249172 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:54.249093    2574 generic.go:356] "Generic (PLEG): container finished" podID="65f40148-a803-4220-8471-527fb92b57a0" containerID="dde4065a685e913bb7745d22bc7dec469b7ac912a28727d85da8e03bccec1aae" exitCode=0
Sep 17 17:47:54.249576 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:54.249162    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dnqsq" event={"ID":"65f40148-a803-4220-8471-527fb92b57a0","Type":"ContainerDied","Data":"dde4065a685e913bb7745d22bc7dec469b7ac912a28727d85da8e03bccec1aae"}
Sep 17 17:47:59.273898 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:59.273864    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dnqsq" event={"ID":"65f40148-a803-4220-8471-527fb92b57a0","Type":"ContainerStarted","Data":"d85e11b3f5cc3ed135e6313bc609db2e7647844174033ae8067a98619893d5e9"}
Sep 17 17:47:59.300072 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:47:59.299994    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-dnqsq" podStartSLOduration=1.856114617 podStartE2EDuration="11.29997233s" totalImagesPullingTime="7.618406943s" totalInitContainerRuntime="1.82545077s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:47:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:47:59.298456477 +0000 UTC m=+854.627556100" watchObservedRunningTime="2026-09-17 17:47:59.29997233 +0000 UTC m=+854.629071957"
Sep 17 17:49:03.510280 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:03.510243    2574 generic.go:356] "Generic (PLEG): container finished" podID="65f40148-a803-4220-8471-527fb92b57a0" containerID="d85e11b3f5cc3ed135e6313bc609db2e7647844174033ae8067a98619893d5e9" exitCode=0
Sep 17 17:49:03.510630 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:03.510315    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dnqsq" event={"ID":"65f40148-a803-4220-8471-527fb92b57a0","Type":"ContainerDied","Data":"d85e11b3f5cc3ed135e6313bc609db2e7647844174033ae8067a98619893d5e9"}
Sep 17 17:49:04.640842 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:04.640821    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-dnqsq"
Sep 17 17:49:04.743985 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:04.743952    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65f40148-a803-4220-8471-527fb92b57a0-model-cache\") pod \"65f40148-a803-4220-8471-527fb92b57a0\" (UID: \"65f40148-a803-4220-8471-527fb92b57a0\") "
Sep 17 17:49:04.744117 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:04.744011    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsbrx\" (UniqueName: \"kubernetes.io/projected/65f40148-a803-4220-8471-527fb92b57a0-kube-api-access-hsbrx\") pod \"65f40148-a803-4220-8471-527fb92b57a0\" (UID: \"65f40148-a803-4220-8471-527fb92b57a0\") "
Sep 17 17:49:04.746125 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:04.746093    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f40148-a803-4220-8471-527fb92b57a0-kube-api-access-hsbrx" (OuterVolumeSpecName: "kube-api-access-hsbrx") pod "65f40148-a803-4220-8471-527fb92b57a0" (UID: "65f40148-a803-4220-8471-527fb92b57a0"). InnerVolumeSpecName "kube-api-access-hsbrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:49:04.762684 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:04.762658    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f40148-a803-4220-8471-527fb92b57a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "65f40148-a803-4220-8471-527fb92b57a0" (UID: "65f40148-a803-4220-8471-527fb92b57a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:49:04.844643 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:04.844621    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65f40148-a803-4220-8471-527fb92b57a0-model-cache\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:49:04.844731 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:04.844642    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsbrx\" (UniqueName: \"kubernetes.io/projected/65f40148-a803-4220-8471-527fb92b57a0-kube-api-access-hsbrx\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:49:05.520274 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:05.520199    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-dnqsq"
Sep 17 17:49:05.520402 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:05.520199    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dnqsq" event={"ID":"65f40148-a803-4220-8471-527fb92b57a0","Type":"ContainerDied","Data":"ba0d3ddd23c81e40afb71351301ea7ca1da0e832ec1aab0973e3cb0713a255d7"}
Sep 17 17:49:05.520402 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:05.520302    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba0d3ddd23c81e40afb71351301ea7ca1da0e832ec1aab0973e3cb0713a255d7"
Sep 17 17:49:05.967630 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:05.967597    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:06.071183 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:06.071153    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:17.485894 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.485859    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:17.612287 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.612251    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:17.761420 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.761345    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"]
Sep 17 17:49:17.761681 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.761668    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65f40148-a803-4220-8471-527fb92b57a0" containerName="download-hf-model"
Sep 17 17:49:17.761722 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.761683    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="65f40148-a803-4220-8471-527fb92b57a0" containerName="download-hf-model"
Sep 17 17:49:17.761722 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.761692    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65f40148-a803-4220-8471-527fb92b57a0" containerName="s3-init"
Sep 17 17:49:17.761722 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.761698    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="65f40148-a803-4220-8471-527fb92b57a0" containerName="s3-init"
Sep 17 17:49:17.761814 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.761752    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="65f40148-a803-4220-8471-527fb92b57a0" containerName="s3-init"
Sep 17 17:49:17.764839 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.764822    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"
Sep 17 17:49:17.767957 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.767930    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:49:17.768125 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.767944    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 17:49:17.768125 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.767995    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-q5zmc\""
Sep 17 17:49:17.768125 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.768029    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:49:17.768125 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.768027    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:49:17.768125 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.767988    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 17 17:49:17.768833 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.768818    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:49:17.781749 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.781728    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"]
Sep 17 17:49:17.809116 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.809092    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:17.833803 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.833778    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e73d825-ca2f-4c31-829b-58971098973d-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22\" (UID: \"7e73d825-ca2f-4c31-829b-58971098973d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"
Sep 17 17:49:17.833889 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.833818    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e73d825-ca2f-4c31-829b-58971098973d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22\" (UID: \"7e73d825-ca2f-4c31-829b-58971098973d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"
Sep 17 17:49:17.833953 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.833884    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e73d825-ca2f-4c31-829b-58971098973d-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22\" (UID: \"7e73d825-ca2f-4c31-829b-58971098973d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"
Sep 17 17:49:17.834009 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.833970    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdctm\" (UniqueName: \"kubernetes.io/projected/7e73d825-ca2f-4c31-829b-58971098973d-kube-api-access-wdctm\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22\" (UID: \"7e73d825-ca2f-4c31-829b-58971098973d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"
Sep 17 17:49:17.834083 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.834006    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e73d825-ca2f-4c31-829b-58971098973d-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22\" (UID: \"7e73d825-ca2f-4c31-829b-58971098973d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"
Sep 17 17:49:17.847511 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.847490    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"]
Sep 17 17:49:17.850708 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.850692    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"
Sep 17 17:49:17.852951 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.852935    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 17:49:17.867785 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.867761    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"]
Sep 17 17:49:17.934719 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.934694    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d45fa650-d04d-4c22-bc99-a3a622da9090-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh\" (UID: \"d45fa650-d04d-4c22-bc99-a3a622da9090\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"
Sep 17 17:49:17.934840 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.934744    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdctm\" (UniqueName: \"kubernetes.io/projected/7e73d825-ca2f-4c31-829b-58971098973d-kube-api-access-wdctm\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22\" (UID: \"7e73d825-ca2f-4c31-829b-58971098973d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"
Sep 17 17:49:17.934840 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.934794    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e73d825-ca2f-4c31-829b-58971098973d-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22\" (UID: \"7e73d825-ca2f-4c31-829b-58971098973d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"
Sep 17 17:49:17.934840 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.934824    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr2n9\" (UniqueName: \"kubernetes.io/projected/d45fa650-d04d-4c22-bc99-a3a622da9090-kube-api-access-lr2n9\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh\" (UID: \"d45fa650-d04d-4c22-bc99-a3a622da9090\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"
Sep 17 17:49:17.934956 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.934849    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d45fa650-d04d-4c22-bc99-a3a622da9090-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh\" (UID: \"d45fa650-d04d-4c22-bc99-a3a622da9090\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"
Sep 17 17:49:17.934956 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:49:17.934909    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 17 17:49:17.934956 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.934922    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e73d825-ca2f-4c31-829b-58971098973d-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22\" (UID: \"7e73d825-ca2f-4c31-829b-58971098973d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"
Sep 17 17:49:17.935075 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.934961    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d45fa650-d04d-4c22-bc99-a3a622da9090-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh\" (UID: \"d45fa650-d04d-4c22-bc99-a3a622da9090\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"
Sep 17 17:49:17.935075 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:49:17.934979    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e73d825-ca2f-4c31-829b-58971098973d-proxy-tls podName:7e73d825-ca2f-4c31-829b-58971098973d nodeName:}" failed. No retries permitted until 2026-09-17 17:49:18.434964749 +0000 UTC m=+933.764064350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7e73d825-ca2f-4c31-829b-58971098973d-proxy-tls") pod "isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22" (UID: "7e73d825-ca2f-4c31-829b-58971098973d") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 17 17:49:17.935075 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.935035    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e73d825-ca2f-4c31-829b-58971098973d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22\" (UID: \"7e73d825-ca2f-4c31-829b-58971098973d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"
Sep 17 17:49:17.935240 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.935086    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d45fa650-d04d-4c22-bc99-a3a622da9090-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh\" (UID: \"d45fa650-d04d-4c22-bc99-a3a622da9090\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"
Sep 17 17:49:17.935240 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.935113    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e73d825-ca2f-4c31-829b-58971098973d-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22\" (UID: \"7e73d825-ca2f-4c31-829b-58971098973d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"
Sep 17 17:49:17.935423 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.935404    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e73d825-ca2f-4c31-829b-58971098973d-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22\" (UID: \"7e73d825-ca2f-4c31-829b-58971098973d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"
Sep 17 17:49:17.935568 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.935550    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e73d825-ca2f-4c31-829b-58971098973d-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22\" (UID: \"7e73d825-ca2f-4c31-829b-58971098973d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"
Sep 17 17:49:17.935669 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.935651    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e73d825-ca2f-4c31-829b-58971098973d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22\" (UID: \"7e73d825-ca2f-4c31-829b-58971098973d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"
Sep 17 17:49:17.947369 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.947342    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdctm\" (UniqueName: \"kubernetes.io/projected/7e73d825-ca2f-4c31-829b-58971098973d-kube-api-access-wdctm\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22\" (UID: \"7e73d825-ca2f-4c31-829b-58971098973d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"
Sep 17 17:49:17.977907 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:17.977882    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:18.035768 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:18.035717    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d45fa650-d04d-4c22-bc99-a3a622da9090-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh\" (UID: \"d45fa650-d04d-4c22-bc99-a3a622da9090\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"
Sep 17 17:49:18.035768 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:18.035760    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr2n9\" (UniqueName: \"kubernetes.io/projected/d45fa650-d04d-4c22-bc99-a3a622da9090-kube-api-access-lr2n9\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh\" (UID: \"d45fa650-d04d-4c22-bc99-a3a622da9090\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"
Sep 17 17:49:18.035955 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:18.035776    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d45fa650-d04d-4c22-bc99-a3a622da9090-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh\" (UID: \"d45fa650-d04d-4c22-bc99-a3a622da9090\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"
Sep 17 17:49:18.035955 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:18.035810    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d45fa650-d04d-4c22-bc99-a3a622da9090-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh\" (UID: \"d45fa650-d04d-4c22-bc99-a3a622da9090\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"
Sep 17 17:49:18.035955 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:18.035831    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d45fa650-d04d-4c22-bc99-a3a622da9090-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh\" (UID: \"d45fa650-d04d-4c22-bc99-a3a622da9090\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"
Sep 17 17:49:18.036138 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:49:18.035964    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 17:49:18.036138 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:49:18.036024    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d45fa650-d04d-4c22-bc99-a3a622da9090-proxy-tls podName:d45fa650-d04d-4c22-bc99-a3a622da9090 nodeName:}" failed. No retries permitted until 2026-09-17 17:49:18.536006782 +0000 UTC m=+933.865106396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d45fa650-d04d-4c22-bc99-a3a622da9090-proxy-tls") pod "isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh" (UID: "d45fa650-d04d-4c22-bc99-a3a622da9090") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 17:49:18.036265 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:18.036244    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d45fa650-d04d-4c22-bc99-a3a622da9090-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh\" (UID: \"d45fa650-d04d-4c22-bc99-a3a622da9090\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"
Sep 17 17:49:18.036427 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:18.036409    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d45fa650-d04d-4c22-bc99-a3a622da9090-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh\" (UID: \"d45fa650-d04d-4c22-bc99-a3a622da9090\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"
Sep 17 17:49:18.036507 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:18.036491    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d45fa650-d04d-4c22-bc99-a3a622da9090-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh\" (UID: \"d45fa650-d04d-4c22-bc99-a3a622da9090\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"
Sep 17 17:49:18.049917 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:18.049891    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr2n9\" (UniqueName: \"kubernetes.io/projected/d45fa650-d04d-4c22-bc99-a3a622da9090-kube-api-access-lr2n9\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh\" (UID: \"d45fa650-d04d-4c22-bc99-a3a622da9090\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"
Sep 17 17:49:18.176797 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:18.176772    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:18.275332 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:18.275308    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:18.439071 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:18.439022    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e73d825-ca2f-4c31-829b-58971098973d-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22\" (UID: \"7e73d825-ca2f-4c31-829b-58971098973d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"
Sep 17 17:49:18.441403 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:18.441380    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e73d825-ca2f-4c31-829b-58971098973d-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22\" (UID: \"7e73d825-ca2f-4c31-829b-58971098973d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"
Sep 17 17:49:18.539630 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:18.539602    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d45fa650-d04d-4c22-bc99-a3a622da9090-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh\" (UID: \"d45fa650-d04d-4c22-bc99-a3a622da9090\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"
Sep 17 17:49:18.541772 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:18.541752    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d45fa650-d04d-4c22-bc99-a3a622da9090-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh\" (UID: \"d45fa650-d04d-4c22-bc99-a3a622da9090\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"
Sep 17 17:49:18.674561 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:18.674531    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"
Sep 17 17:49:18.760356 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:18.760331    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"
Sep 17 17:49:18.954388 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:18.954358    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"]
Sep 17 17:49:18.956879 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:49:18.956848    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e73d825_ca2f_4c31_829b_58971098973d.slice/crio-47fbcf39ba902246c1913220a072da82e390d6f286227b6e98c4e2bd1c5d1481 WatchSource:0}: Error finding container 47fbcf39ba902246c1913220a072da82e390d6f286227b6e98c4e2bd1c5d1481: Status 404 returned error can't find the container with id 47fbcf39ba902246c1913220a072da82e390d6f286227b6e98c4e2bd1c5d1481
Sep 17 17:49:19.033943 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:19.033907    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"]
Sep 17 17:49:19.576218 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:19.576181    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22" event={"ID":"7e73d825-ca2f-4c31-829b-58971098973d","Type":"ContainerStarted","Data":"71d29de34fe8a2a913e22c0f1bdc47ee5b615189eab99415789a4a18e7214068"}
Sep 17 17:49:19.576674 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:19.576257    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22" event={"ID":"7e73d825-ca2f-4c31-829b-58971098973d","Type":"ContainerStarted","Data":"47fbcf39ba902246c1913220a072da82e390d6f286227b6e98c4e2bd1c5d1481"}
Sep 17 17:49:19.578041 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:19.578009    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh" event={"ID":"d45fa650-d04d-4c22-bc99-a3a622da9090","Type":"ContainerStarted","Data":"df992e7f7f3ec1020d1f039d44086ea6592e45ac9c44d2e89d3dea38d1c4a1b9"}
Sep 17 17:49:19.578164 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:19.578098    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh" event={"ID":"d45fa650-d04d-4c22-bc99-a3a622da9090","Type":"ContainerStarted","Data":"e8123041fc88c100e6b2d7003645ec746e866fa9afb3eef7575b170e595d8562"}
Sep 17 17:49:23.596626 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:23.596595    2574 generic.go:356] "Generic (PLEG): container finished" podID="d45fa650-d04d-4c22-bc99-a3a622da9090" containerID="df992e7f7f3ec1020d1f039d44086ea6592e45ac9c44d2e89d3dea38d1c4a1b9" exitCode=0
Sep 17 17:49:23.597087 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:23.596668    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh" event={"ID":"d45fa650-d04d-4c22-bc99-a3a622da9090","Type":"ContainerDied","Data":"df992e7f7f3ec1020d1f039d44086ea6592e45ac9c44d2e89d3dea38d1c4a1b9"}
Sep 17 17:49:23.598508 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:23.598486    2574 generic.go:356] "Generic (PLEG): container finished" podID="7e73d825-ca2f-4c31-829b-58971098973d" containerID="71d29de34fe8a2a913e22c0f1bdc47ee5b615189eab99415789a4a18e7214068" exitCode=0
Sep 17 17:49:23.598584 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:23.598530    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22" event={"ID":"7e73d825-ca2f-4c31-829b-58971098973d","Type":"ContainerDied","Data":"71d29de34fe8a2a913e22c0f1bdc47ee5b615189eab99415789a4a18e7214068"}
Sep 17 17:49:39.696379 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:39.696348    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh" event={"ID":"d45fa650-d04d-4c22-bc99-a3a622da9090","Type":"ContainerStarted","Data":"6a2b8db96da1d283abb24ed1b37d87e4b79bbe7bed6552260cdd3d199f12e127"}
Sep 17 17:49:40.707989 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:40.707949    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22" event={"ID":"7e73d825-ca2f-4c31-829b-58971098973d","Type":"ContainerStarted","Data":"452703c9644a7cd6a5d39d57187d08c4c31f7856a33a38244addac8c03197e1e"}
Sep 17 17:49:42.717652 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:42.717614    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22" event={"ID":"7e73d825-ca2f-4c31-829b-58971098973d","Type":"ContainerStarted","Data":"7ae51c667166c696613df6e282d921e5e13d261dd6403db5b29d17028d5a0abb"}
Sep 17 17:49:42.718128 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:42.717730    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"
Sep 17 17:49:42.719863 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:42.719834    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh" event={"ID":"d45fa650-d04d-4c22-bc99-a3a622da9090","Type":"ContainerStarted","Data":"36fc56801ff9ed98ef5442bbb39de69118125b24c423eb9bfb41c0f6f6ff8041"}
Sep 17 17:49:42.719973 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:42.719950    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"
Sep 17 17:49:42.742816 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:42.742773    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22" podStartSLOduration=3.01094946 podStartE2EDuration="25.742759018s" totalImagesPullingTime="18.87590471s" totalInitContainerRuntime="3.855904848s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:49:42.741555826 +0000 UTC m=+958.070655472" watchObservedRunningTime="2026-09-17 17:49:42.742759018 +0000 UTC m=+958.071858643"
Sep 17 17:49:42.766963 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:42.766889    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh" podStartSLOduration=3.025844922 podStartE2EDuration="25.766874559s" totalImagesPullingTime="18.846129775s" totalInitContainerRuntime="3.894899862s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:49:42.765628817 +0000 UTC m=+958.094728441" watchObservedRunningTime="2026-09-17 17:49:42.766874559 +0000 UTC m=+958.095974183"
Sep 17 17:49:43.729468 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:43.729427    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"
Sep 17 17:49:43.729880 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:43.729525    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"
Sep 17 17:49:43.730557 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:43.730514    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh" podUID="d45fa650-d04d-4c22-bc99-a3a622da9090" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 17:49:43.730665 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:43.730514    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22" podUID="7e73d825-ca2f-4c31-829b-58971098973d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 17:49:44.734087 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:44.734029    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh" podUID="d45fa650-d04d-4c22-bc99-a3a622da9090" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 17:49:44.734529 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:44.734134    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22" podUID="7e73d825-ca2f-4c31-829b-58971098973d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 17:49:49.738174 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:49.738138    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"
Sep 17 17:49:49.738578 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:49.738557    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"
Sep 17 17:49:49.738623 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:49.738586    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh" podUID="d45fa650-d04d-4c22-bc99-a3a622da9090" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 17:49:49.739012 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:49.738990    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22" podUID="7e73d825-ca2f-4c31-829b-58971098973d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 17:49:59.738716 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:59.738637    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh" podUID="d45fa650-d04d-4c22-bc99-a3a622da9090" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 17:49:59.739206 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:49:59.738954    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22" podUID="7e73d825-ca2f-4c31-829b-58971098973d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 17:50:09.738694 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:50:09.738597    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh" podUID="d45fa650-d04d-4c22-bc99-a3a622da9090" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 17:50:09.739087 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:50:09.738927    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22" podUID="7e73d825-ca2f-4c31-829b-58971098973d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 17:50:19.738672 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:50:19.738628    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh" podUID="d45fa650-d04d-4c22-bc99-a3a622da9090" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 17:50:19.739135 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:50:19.738936    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22" podUID="7e73d825-ca2f-4c31-829b-58971098973d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 17:50:21.799203 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:50:21.799167    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:29.738727 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:50:29.738687    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh" podUID="d45fa650-d04d-4c22-bc99-a3a622da9090" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 17:50:29.739170 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:50:29.738950    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22" podUID="7e73d825-ca2f-4c31-829b-58971098973d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 17:50:39.739239 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:50:39.739202    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22" podUID="7e73d825-ca2f-4c31-829b-58971098973d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 17:50:39.739660 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:50:39.739207    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh" podUID="d45fa650-d04d-4c22-bc99-a3a622da9090" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 17:50:49.738662 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:50:49.738617    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh" podUID="d45fa650-d04d-4c22-bc99-a3a622da9090" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 17:50:49.739082 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:50:49.739024    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22" podUID="7e73d825-ca2f-4c31-829b-58971098973d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 17:50:57.115939 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:50:57.115909    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"
Sep 17 17:50:59.740005 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:50:59.739972    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"
Sep 17 17:51:02.690478 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:02.690440    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:03.027786 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:03.027704    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:03.127233 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:03.127201    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:07.429742 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:07.429665    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"]
Sep 17 17:51:07.430157 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:07.429911    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22" podUID="7e73d825-ca2f-4c31-829b-58971098973d" containerName="kserve-container" containerID="cri-o://452703c9644a7cd6a5d39d57187d08c4c31f7856a33a38244addac8c03197e1e" gracePeriod=30
Sep 17 17:51:07.430157 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:07.429950    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22" podUID="7e73d825-ca2f-4c31-829b-58971098973d" containerName="kube-rbac-proxy" containerID="cri-o://7ae51c667166c696613df6e282d921e5e13d261dd6403db5b29d17028d5a0abb" gracePeriod=30
Sep 17 17:51:07.484892 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:07.484861    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:08.071563 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:08.071525    2574 generic.go:356] "Generic (PLEG): container finished" podID="7e73d825-ca2f-4c31-829b-58971098973d" containerID="7ae51c667166c696613df6e282d921e5e13d261dd6403db5b29d17028d5a0abb" exitCode=2
Sep 17 17:51:08.071734 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:08.071595    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22" event={"ID":"7e73d825-ca2f-4c31-829b-58971098973d","Type":"ContainerDied","Data":"7ae51c667166c696613df6e282d921e5e13d261dd6403db5b29d17028d5a0abb"}
Sep 17 17:51:09.734373 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:09.734326    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22" podUID="7e73d825-ca2f-4c31-829b-58971098973d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 17 17:51:09.739735 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:09.739704    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22" podUID="7e73d825-ca2f-4c31-829b-58971098973d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 17:51:11.469158 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:11.469133    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"
Sep 17 17:51:11.554420 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:11.554365    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e73d825-ca2f-4c31-829b-58971098973d-kube-rbac-proxy-sar-config\") pod \"7e73d825-ca2f-4c31-829b-58971098973d\" (UID: \"7e73d825-ca2f-4c31-829b-58971098973d\") "
Sep 17 17:51:11.554420 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:11.554401    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e73d825-ca2f-4c31-829b-58971098973d-kserve-provision-location\") pod \"7e73d825-ca2f-4c31-829b-58971098973d\" (UID: \"7e73d825-ca2f-4c31-829b-58971098973d\") "
Sep 17 17:51:11.554420 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:11.554419    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdctm\" (UniqueName: \"kubernetes.io/projected/7e73d825-ca2f-4c31-829b-58971098973d-kube-api-access-wdctm\") pod \"7e73d825-ca2f-4c31-829b-58971098973d\" (UID: \"7e73d825-ca2f-4c31-829b-58971098973d\") "
Sep 17 17:51:11.554613 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:11.554493    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e73d825-ca2f-4c31-829b-58971098973d-proxy-tls\") pod \"7e73d825-ca2f-4c31-829b-58971098973d\" (UID: \"7e73d825-ca2f-4c31-829b-58971098973d\") "
Sep 17 17:51:11.554613 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:11.554520    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e73d825-ca2f-4c31-829b-58971098973d-cabundle-cert\") pod \"7e73d825-ca2f-4c31-829b-58971098973d\" (UID: \"7e73d825-ca2f-4c31-829b-58971098973d\") "
Sep 17 17:51:11.554717 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:11.554666    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e73d825-ca2f-4c31-829b-58971098973d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7e73d825-ca2f-4c31-829b-58971098973d" (UID: "7e73d825-ca2f-4c31-829b-58971098973d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:51:11.554774 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:11.554715    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e73d825-ca2f-4c31-829b-58971098973d-kserve-provision-location\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:51:11.554774 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:11.554731    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e73d825-ca2f-4c31-829b-58971098973d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7e73d825-ca2f-4c31-829b-58971098973d" (UID: "7e73d825-ca2f-4c31-829b-58971098973d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:51:11.554985 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:11.554963    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e73d825-ca2f-4c31-829b-58971098973d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7e73d825-ca2f-4c31-829b-58971098973d" (UID: "7e73d825-ca2f-4c31-829b-58971098973d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:51:11.556539 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:11.556514    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e73d825-ca2f-4c31-829b-58971098973d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7e73d825-ca2f-4c31-829b-58971098973d" (UID: "7e73d825-ca2f-4c31-829b-58971098973d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:51:11.556598 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:11.556564    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e73d825-ca2f-4c31-829b-58971098973d-kube-api-access-wdctm" (OuterVolumeSpecName: "kube-api-access-wdctm") pod "7e73d825-ca2f-4c31-829b-58971098973d" (UID: "7e73d825-ca2f-4c31-829b-58971098973d"). InnerVolumeSpecName "kube-api-access-wdctm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:51:11.655572 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:11.655545    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e73d825-ca2f-4c31-829b-58971098973d-proxy-tls\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:51:11.655572 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:11.655573    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e73d825-ca2f-4c31-829b-58971098973d-cabundle-cert\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:51:11.655715 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:11.655586    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e73d825-ca2f-4c31-829b-58971098973d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:51:11.655715 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:11.655595    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdctm\" (UniqueName: \"kubernetes.io/projected/7e73d825-ca2f-4c31-829b-58971098973d-kube-api-access-wdctm\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:51:12.088524 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:12.088490    2574 generic.go:356] "Generic (PLEG): container finished" podID="7e73d825-ca2f-4c31-829b-58971098973d" containerID="452703c9644a7cd6a5d39d57187d08c4c31f7856a33a38244addac8c03197e1e" exitCode=0
Sep 17 17:51:12.088682 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:12.088544    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"
Sep 17 17:51:12.088682 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:12.088580    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22" event={"ID":"7e73d825-ca2f-4c31-829b-58971098973d","Type":"ContainerDied","Data":"452703c9644a7cd6a5d39d57187d08c4c31f7856a33a38244addac8c03197e1e"}
Sep 17 17:51:12.088682 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:12.088640    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22" event={"ID":"7e73d825-ca2f-4c31-829b-58971098973d","Type":"ContainerDied","Data":"47fbcf39ba902246c1913220a072da82e390d6f286227b6e98c4e2bd1c5d1481"}
Sep 17 17:51:12.088682 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:12.088670    2574 scope.go:117] "RemoveContainer" containerID="7ae51c667166c696613df6e282d921e5e13d261dd6403db5b29d17028d5a0abb"
Sep 17 17:51:12.097907 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:12.097890    2574 scope.go:117] "RemoveContainer" containerID="452703c9644a7cd6a5d39d57187d08c4c31f7856a33a38244addac8c03197e1e"
Sep 17 17:51:12.104942 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:12.104925    2574 scope.go:117] "RemoveContainer" containerID="71d29de34fe8a2a913e22c0f1bdc47ee5b615189eab99415789a4a18e7214068"
Sep 17 17:51:12.111899 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:12.111881    2574 scope.go:117] "RemoveContainer" containerID="7ae51c667166c696613df6e282d921e5e13d261dd6403db5b29d17028d5a0abb"
Sep 17 17:51:12.112600 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:51:12.112561    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ae51c667166c696613df6e282d921e5e13d261dd6403db5b29d17028d5a0abb\": container with ID starting with 7ae51c667166c696613df6e282d921e5e13d261dd6403db5b29d17028d5a0abb not found: ID does not exist" containerID="7ae51c667166c696613df6e282d921e5e13d261dd6403db5b29d17028d5a0abb"
Sep 17 17:51:12.112707 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:12.112611    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae51c667166c696613df6e282d921e5e13d261dd6403db5b29d17028d5a0abb"} err="failed to get container status \"7ae51c667166c696613df6e282d921e5e13d261dd6403db5b29d17028d5a0abb\": rpc error: code = NotFound desc = could not find container \"7ae51c667166c696613df6e282d921e5e13d261dd6403db5b29d17028d5a0abb\": container with ID starting with 7ae51c667166c696613df6e282d921e5e13d261dd6403db5b29d17028d5a0abb not found: ID does not exist"
Sep 17 17:51:12.112707 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:12.112636    2574 scope.go:117] "RemoveContainer" containerID="452703c9644a7cd6a5d39d57187d08c4c31f7856a33a38244addac8c03197e1e"
Sep 17 17:51:12.113014 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:51:12.112986    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"452703c9644a7cd6a5d39d57187d08c4c31f7856a33a38244addac8c03197e1e\": container with ID starting with 452703c9644a7cd6a5d39d57187d08c4c31f7856a33a38244addac8c03197e1e not found: ID does not exist" containerID="452703c9644a7cd6a5d39d57187d08c4c31f7856a33a38244addac8c03197e1e"
Sep 17 17:51:12.113101 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:12.113024    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"452703c9644a7cd6a5d39d57187d08c4c31f7856a33a38244addac8c03197e1e"} err="failed to get container status \"452703c9644a7cd6a5d39d57187d08c4c31f7856a33a38244addac8c03197e1e\": rpc error: code = NotFound desc = could not find container \"452703c9644a7cd6a5d39d57187d08c4c31f7856a33a38244addac8c03197e1e\": container with ID starting with 452703c9644a7cd6a5d39d57187d08c4c31f7856a33a38244addac8c03197e1e not found: ID does not exist"
Sep 17 17:51:12.113101 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:12.113042    2574 scope.go:117] "RemoveContainer" containerID="71d29de34fe8a2a913e22c0f1bdc47ee5b615189eab99415789a4a18e7214068"
Sep 17 17:51:12.113656 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:51:12.113633    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71d29de34fe8a2a913e22c0f1bdc47ee5b615189eab99415789a4a18e7214068\": container with ID starting with 71d29de34fe8a2a913e22c0f1bdc47ee5b615189eab99415789a4a18e7214068 not found: ID does not exist" containerID="71d29de34fe8a2a913e22c0f1bdc47ee5b615189eab99415789a4a18e7214068"
Sep 17 17:51:12.113728 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:12.113663    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d29de34fe8a2a913e22c0f1bdc47ee5b615189eab99415789a4a18e7214068"} err="failed to get container status \"71d29de34fe8a2a913e22c0f1bdc47ee5b615189eab99415789a4a18e7214068\": rpc error: code = NotFound desc = could not find container \"71d29de34fe8a2a913e22c0f1bdc47ee5b615189eab99415789a4a18e7214068\": container with ID starting with 71d29de34fe8a2a913e22c0f1bdc47ee5b615189eab99415789a4a18e7214068 not found: ID does not exist"
Sep 17 17:51:12.113961 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:12.113943    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"]
Sep 17 17:51:12.118506 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:12.118478    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-8wc22"]
Sep 17 17:51:13.115654 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:51:13.115620    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e73d825-ca2f-4c31-829b-58971098973d" path="/var/lib/kubelet/pods/7e73d825-ca2f-4c31-829b-58971098973d/volumes"
Sep 17 17:52:37.735880 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:37.735849    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"]
Sep 17 17:52:37.736420 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:37.736108    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh" podUID="d45fa650-d04d-4c22-bc99-a3a622da9090" containerName="kserve-container" containerID="cri-o://6a2b8db96da1d283abb24ed1b37d87e4b79bbe7bed6552260cdd3d199f12e127" gracePeriod=30
Sep 17 17:52:37.736420 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:37.736155    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh" podUID="d45fa650-d04d-4c22-bc99-a3a622da9090" containerName="kube-rbac-proxy" containerID="cri-o://36fc56801ff9ed98ef5442bbb39de69118125b24c423eb9bfb41c0f6f6ff8041" gracePeriod=30
Sep 17 17:52:37.836409 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:37.836383    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:38.414581 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:38.414546    2574 generic.go:356] "Generic (PLEG): container finished" podID="d45fa650-d04d-4c22-bc99-a3a622da9090" containerID="36fc56801ff9ed98ef5442bbb39de69118125b24c423eb9bfb41c0f6f6ff8041" exitCode=2
Sep 17 17:52:38.414581 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:38.414587    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh" event={"ID":"d45fa650-d04d-4c22-bc99-a3a622da9090","Type":"ContainerDied","Data":"36fc56801ff9ed98ef5442bbb39de69118125b24c423eb9bfb41c0f6f6ff8041"}
Sep 17 17:52:39.734226 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:39.734181    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh" podUID="d45fa650-d04d-4c22-bc99-a3a622da9090" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Sep 17 17:52:41.973568 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:41.973544    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"
Sep 17 17:52:42.022839 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.022808    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d45fa650-d04d-4c22-bc99-a3a622da9090-cabundle-cert\") pod \"d45fa650-d04d-4c22-bc99-a3a622da9090\" (UID: \"d45fa650-d04d-4c22-bc99-a3a622da9090\") "
Sep 17 17:52:42.022961 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.022859    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lr2n9\" (UniqueName: \"kubernetes.io/projected/d45fa650-d04d-4c22-bc99-a3a622da9090-kube-api-access-lr2n9\") pod \"d45fa650-d04d-4c22-bc99-a3a622da9090\" (UID: \"d45fa650-d04d-4c22-bc99-a3a622da9090\") "
Sep 17 17:52:42.022961 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.022891    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d45fa650-d04d-4c22-bc99-a3a622da9090-kube-rbac-proxy-sar-config\") pod \"d45fa650-d04d-4c22-bc99-a3a622da9090\" (UID: \"d45fa650-d04d-4c22-bc99-a3a622da9090\") "
Sep 17 17:52:42.022961 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.022931    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d45fa650-d04d-4c22-bc99-a3a622da9090-kserve-provision-location\") pod \"d45fa650-d04d-4c22-bc99-a3a622da9090\" (UID: \"d45fa650-d04d-4c22-bc99-a3a622da9090\") "
Sep 17 17:52:42.023117 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.022991    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d45fa650-d04d-4c22-bc99-a3a622da9090-proxy-tls\") pod \"d45fa650-d04d-4c22-bc99-a3a622da9090\" (UID: \"d45fa650-d04d-4c22-bc99-a3a622da9090\") "
Sep 17 17:52:42.023277 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.023250    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d45fa650-d04d-4c22-bc99-a3a622da9090-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d45fa650-d04d-4c22-bc99-a3a622da9090" (UID: "d45fa650-d04d-4c22-bc99-a3a622da9090"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:52:42.023277 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.023262    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d45fa650-d04d-4c22-bc99-a3a622da9090-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d45fa650-d04d-4c22-bc99-a3a622da9090" (UID: "d45fa650-d04d-4c22-bc99-a3a622da9090"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:52:42.023407 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.023312    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d45fa650-d04d-4c22-bc99-a3a622da9090-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d45fa650-d04d-4c22-bc99-a3a622da9090" (UID: "d45fa650-d04d-4c22-bc99-a3a622da9090"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:52:42.024964 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.024940    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d45fa650-d04d-4c22-bc99-a3a622da9090-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d45fa650-d04d-4c22-bc99-a3a622da9090" (UID: "d45fa650-d04d-4c22-bc99-a3a622da9090"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:52:42.025075 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.024975    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d45fa650-d04d-4c22-bc99-a3a622da9090-kube-api-access-lr2n9" (OuterVolumeSpecName: "kube-api-access-lr2n9") pod "d45fa650-d04d-4c22-bc99-a3a622da9090" (UID: "d45fa650-d04d-4c22-bc99-a3a622da9090"). InnerVolumeSpecName "kube-api-access-lr2n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:52:42.124339 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.124314    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d45fa650-d04d-4c22-bc99-a3a622da9090-kserve-provision-location\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:52:42.124339 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.124338    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d45fa650-d04d-4c22-bc99-a3a622da9090-proxy-tls\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:52:42.124506 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.124347    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d45fa650-d04d-4c22-bc99-a3a622da9090-cabundle-cert\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:52:42.124506 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.124357    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr2n9\" (UniqueName: \"kubernetes.io/projected/d45fa650-d04d-4c22-bc99-a3a622da9090-kube-api-access-lr2n9\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:52:42.124506 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.124365    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d45fa650-d04d-4c22-bc99-a3a622da9090-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:52:42.430903 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.430869    2574 generic.go:356] "Generic (PLEG): container finished" podID="d45fa650-d04d-4c22-bc99-a3a622da9090" containerID="6a2b8db96da1d283abb24ed1b37d87e4b79bbe7bed6552260cdd3d199f12e127" exitCode=0
Sep 17 17:52:42.431085 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.430935    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"
Sep 17 17:52:42.431085 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.430956    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh" event={"ID":"d45fa650-d04d-4c22-bc99-a3a622da9090","Type":"ContainerDied","Data":"6a2b8db96da1d283abb24ed1b37d87e4b79bbe7bed6552260cdd3d199f12e127"}
Sep 17 17:52:42.431085 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.431032    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh" event={"ID":"d45fa650-d04d-4c22-bc99-a3a622da9090","Type":"ContainerDied","Data":"e8123041fc88c100e6b2d7003645ec746e866fa9afb3eef7575b170e595d8562"}
Sep 17 17:52:42.431085 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.431082    2574 scope.go:117] "RemoveContainer" containerID="36fc56801ff9ed98ef5442bbb39de69118125b24c423eb9bfb41c0f6f6ff8041"
Sep 17 17:52:42.439278 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.439257    2574 scope.go:117] "RemoveContainer" containerID="6a2b8db96da1d283abb24ed1b37d87e4b79bbe7bed6552260cdd3d199f12e127"
Sep 17 17:52:42.446034 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.446016    2574 scope.go:117] "RemoveContainer" containerID="df992e7f7f3ec1020d1f039d44086ea6592e45ac9c44d2e89d3dea38d1c4a1b9"
Sep 17 17:52:42.452727 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.452711    2574 scope.go:117] "RemoveContainer" containerID="36fc56801ff9ed98ef5442bbb39de69118125b24c423eb9bfb41c0f6f6ff8041"
Sep 17 17:52:42.453017 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:52:42.452997    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36fc56801ff9ed98ef5442bbb39de69118125b24c423eb9bfb41c0f6f6ff8041\": container with ID starting with 36fc56801ff9ed98ef5442bbb39de69118125b24c423eb9bfb41c0f6f6ff8041 not found: ID does not exist" containerID="36fc56801ff9ed98ef5442bbb39de69118125b24c423eb9bfb41c0f6f6ff8041"
Sep 17 17:52:42.453088 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.453025    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36fc56801ff9ed98ef5442bbb39de69118125b24c423eb9bfb41c0f6f6ff8041"} err="failed to get container status \"36fc56801ff9ed98ef5442bbb39de69118125b24c423eb9bfb41c0f6f6ff8041\": rpc error: code = NotFound desc = could not find container \"36fc56801ff9ed98ef5442bbb39de69118125b24c423eb9bfb41c0f6f6ff8041\": container with ID starting with 36fc56801ff9ed98ef5442bbb39de69118125b24c423eb9bfb41c0f6f6ff8041 not found: ID does not exist"
Sep 17 17:52:42.453088 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.453041    2574 scope.go:117] "RemoveContainer" containerID="6a2b8db96da1d283abb24ed1b37d87e4b79bbe7bed6552260cdd3d199f12e127"
Sep 17 17:52:42.453315 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:52:42.453297    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a2b8db96da1d283abb24ed1b37d87e4b79bbe7bed6552260cdd3d199f12e127\": container with ID starting with 6a2b8db96da1d283abb24ed1b37d87e4b79bbe7bed6552260cdd3d199f12e127 not found: ID does not exist" containerID="6a2b8db96da1d283abb24ed1b37d87e4b79bbe7bed6552260cdd3d199f12e127"
Sep 17 17:52:42.453367 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.453317    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a2b8db96da1d283abb24ed1b37d87e4b79bbe7bed6552260cdd3d199f12e127"} err="failed to get container status \"6a2b8db96da1d283abb24ed1b37d87e4b79bbe7bed6552260cdd3d199f12e127\": rpc error: code = NotFound desc = could not find container \"6a2b8db96da1d283abb24ed1b37d87e4b79bbe7bed6552260cdd3d199f12e127\": container with ID starting with 6a2b8db96da1d283abb24ed1b37d87e4b79bbe7bed6552260cdd3d199f12e127 not found: ID does not exist"
Sep 17 17:52:42.453367 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.453330    2574 scope.go:117] "RemoveContainer" containerID="df992e7f7f3ec1020d1f039d44086ea6592e45ac9c44d2e89d3dea38d1c4a1b9"
Sep 17 17:52:42.453553 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:52:42.453539    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df992e7f7f3ec1020d1f039d44086ea6592e45ac9c44d2e89d3dea38d1c4a1b9\": container with ID starting with df992e7f7f3ec1020d1f039d44086ea6592e45ac9c44d2e89d3dea38d1c4a1b9 not found: ID does not exist" containerID="df992e7f7f3ec1020d1f039d44086ea6592e45ac9c44d2e89d3dea38d1c4a1b9"
Sep 17 17:52:42.453592 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.453556    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df992e7f7f3ec1020d1f039d44086ea6592e45ac9c44d2e89d3dea38d1c4a1b9"} err="failed to get container status \"df992e7f7f3ec1020d1f039d44086ea6592e45ac9c44d2e89d3dea38d1c4a1b9\": rpc error: code = NotFound desc = could not find container \"df992e7f7f3ec1020d1f039d44086ea6592e45ac9c44d2e89d3dea38d1c4a1b9\": container with ID starting with df992e7f7f3ec1020d1f039d44086ea6592e45ac9c44d2e89d3dea38d1c4a1b9 not found: ID does not exist"
Sep 17 17:52:42.460545 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.460525    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"]
Sep 17 17:52:42.471488 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.471462    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-pxvnh"]
Sep 17 17:52:42.826930 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.826839    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:42.929752 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.929722    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"]
Sep 17 17:52:42.930039 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.930026    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e73d825-ca2f-4c31-829b-58971098973d" containerName="storage-initializer"
Sep 17 17:52:42.930111 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.930041    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e73d825-ca2f-4c31-829b-58971098973d" containerName="storage-initializer"
Sep 17 17:52:42.930111 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.930064    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d45fa650-d04d-4c22-bc99-a3a622da9090" containerName="storage-initializer"
Sep 17 17:52:42.930111 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.930074    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d45fa650-d04d-4c22-bc99-a3a622da9090" containerName="storage-initializer"
Sep 17 17:52:42.930111 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.930087    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e73d825-ca2f-4c31-829b-58971098973d" containerName="kserve-container"
Sep 17 17:52:42.930111 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.930093    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e73d825-ca2f-4c31-829b-58971098973d" containerName="kserve-container"
Sep 17 17:52:42.930111 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.930104    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d45fa650-d04d-4c22-bc99-a3a622da9090" containerName="kserve-container"
Sep 17 17:52:42.930111 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.930109    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d45fa650-d04d-4c22-bc99-a3a622da9090" containerName="kserve-container"
Sep 17 17:52:42.930371 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.930116    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d45fa650-d04d-4c22-bc99-a3a622da9090" containerName="kube-rbac-proxy"
Sep 17 17:52:42.930371 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.930121    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d45fa650-d04d-4c22-bc99-a3a622da9090" containerName="kube-rbac-proxy"
Sep 17 17:52:42.930371 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.930129    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e73d825-ca2f-4c31-829b-58971098973d" containerName="kube-rbac-proxy"
Sep 17 17:52:42.930371 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.930135    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e73d825-ca2f-4c31-829b-58971098973d" containerName="kube-rbac-proxy"
Sep 17 17:52:42.930371 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.930193    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e73d825-ca2f-4c31-829b-58971098973d" containerName="kserve-container"
Sep 17 17:52:42.930371 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.930204    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d45fa650-d04d-4c22-bc99-a3a622da9090" containerName="kube-rbac-proxy"
Sep 17 17:52:42.930371 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.930215    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d45fa650-d04d-4c22-bc99-a3a622da9090" containerName="kserve-container"
Sep 17 17:52:42.930371 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.930227    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e73d825-ca2f-4c31-829b-58971098973d" containerName="kube-rbac-proxy"
Sep 17 17:52:42.934682 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.934663    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"
Sep 17 17:52:42.937882 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.937862    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 17 17:52:42.937992 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.937896    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:52:42.938134 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.938119    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-q5zmc\""
Sep 17 17:52:42.938196 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.938179    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 17:52:42.938328 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.938316    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:52:42.938508 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.938490    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:52:42.938717 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.938702    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:52:42.954945 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.954923    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"]
Sep 17 17:52:42.965915 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:42.965896    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:43.032697 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.032673    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8gcf\" (UniqueName: \"kubernetes.io/projected/e44bd846-8fa1-4527-92a2-d712bf30bcfa-kube-api-access-z8gcf\") pod \"isvc-canary-stop-v2-predictor-596664c988-4sbx5\" (UID: \"e44bd846-8fa1-4527-92a2-d712bf30bcfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"
Sep 17 17:52:43.032990 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.032754    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e44bd846-8fa1-4527-92a2-d712bf30bcfa-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-596664c988-4sbx5\" (UID: \"e44bd846-8fa1-4527-92a2-d712bf30bcfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"
Sep 17 17:52:43.032990 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.032795    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e44bd846-8fa1-4527-92a2-d712bf30bcfa-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-596664c988-4sbx5\" (UID: \"e44bd846-8fa1-4527-92a2-d712bf30bcfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"
Sep 17 17:52:43.032990 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.032837    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e44bd846-8fa1-4527-92a2-d712bf30bcfa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-596664c988-4sbx5\" (UID: \"e44bd846-8fa1-4527-92a2-d712bf30bcfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"
Sep 17 17:52:43.032990 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.032859    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e44bd846-8fa1-4527-92a2-d712bf30bcfa-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-596664c988-4sbx5\" (UID: \"e44bd846-8fa1-4527-92a2-d712bf30bcfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"
Sep 17 17:52:43.047284 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.047259    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"]
Sep 17 17:52:43.050567 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.050553    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"
Sep 17 17:52:43.053673 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.053656    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 17:52:43.103526 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.103505    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:43.115438 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.115415    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d45fa650-d04d-4c22-bc99-a3a622da9090" path="/var/lib/kubelet/pods/d45fa650-d04d-4c22-bc99-a3a622da9090/volumes"
Sep 17 17:52:43.126723 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.126699    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"]
Sep 17 17:52:43.133478 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.133454    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87869ebb-dd0c-481e-93ff-c116602baf23-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5cdcf6545-qsl6n\" (UID: \"87869ebb-dd0c-481e-93ff-c116602baf23\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"
Sep 17 17:52:43.133580 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.133497    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e44bd846-8fa1-4527-92a2-d712bf30bcfa-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-596664c988-4sbx5\" (UID: \"e44bd846-8fa1-4527-92a2-d712bf30bcfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"
Sep 17 17:52:43.133580 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.133528    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e44bd846-8fa1-4527-92a2-d712bf30bcfa-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-596664c988-4sbx5\" (UID: \"e44bd846-8fa1-4527-92a2-d712bf30bcfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"
Sep 17 17:52:43.133580 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.133554    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx449\" (UniqueName: \"kubernetes.io/projected/87869ebb-dd0c-481e-93ff-c116602baf23-kube-api-access-qx449\") pod \"isvc-canary-stop-predictor-5cdcf6545-qsl6n\" (UID: \"87869ebb-dd0c-481e-93ff-c116602baf23\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"
Sep 17 17:52:43.133721 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.133592    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87869ebb-dd0c-481e-93ff-c116602baf23-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5cdcf6545-qsl6n\" (UID: \"87869ebb-dd0c-481e-93ff-c116602baf23\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"
Sep 17 17:52:43.133721 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:52:43.133609    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 17 17:52:43.133721 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.133632    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87869ebb-dd0c-481e-93ff-c116602baf23-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-qsl6n\" (UID: \"87869ebb-dd0c-481e-93ff-c116602baf23\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"
Sep 17 17:52:43.133721 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:52:43.133684    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e44bd846-8fa1-4527-92a2-d712bf30bcfa-proxy-tls podName:e44bd846-8fa1-4527-92a2-d712bf30bcfa nodeName:}" failed. No retries permitted until 2026-09-17 17:52:43.633662267 +0000 UTC m=+1138.962761889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e44bd846-8fa1-4527-92a2-d712bf30bcfa-proxy-tls") pod "isvc-canary-stop-v2-predictor-596664c988-4sbx5" (UID: "e44bd846-8fa1-4527-92a2-d712bf30bcfa") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 17 17:52:43.133875 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.133737    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e44bd846-8fa1-4527-92a2-d712bf30bcfa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-596664c988-4sbx5\" (UID: \"e44bd846-8fa1-4527-92a2-d712bf30bcfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"
Sep 17 17:52:43.133875 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.133757    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e44bd846-8fa1-4527-92a2-d712bf30bcfa-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-596664c988-4sbx5\" (UID: \"e44bd846-8fa1-4527-92a2-d712bf30bcfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"
Sep 17 17:52:43.133982 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.133919    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e44bd846-8fa1-4527-92a2-d712bf30bcfa-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-596664c988-4sbx5\" (UID: \"e44bd846-8fa1-4527-92a2-d712bf30bcfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"
Sep 17 17:52:43.133982 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.133877    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87869ebb-dd0c-481e-93ff-c116602baf23-cabundle-cert\") pod \"isvc-canary-stop-predictor-5cdcf6545-qsl6n\" (UID: \"87869ebb-dd0c-481e-93ff-c116602baf23\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"
Sep 17 17:52:43.134104 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.134022    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8gcf\" (UniqueName: \"kubernetes.io/projected/e44bd846-8fa1-4527-92a2-d712bf30bcfa-kube-api-access-z8gcf\") pod \"isvc-canary-stop-v2-predictor-596664c988-4sbx5\" (UID: \"e44bd846-8fa1-4527-92a2-d712bf30bcfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"
Sep 17 17:52:43.134355 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.134337    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e44bd846-8fa1-4527-92a2-d712bf30bcfa-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-596664c988-4sbx5\" (UID: \"e44bd846-8fa1-4527-92a2-d712bf30bcfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"
Sep 17 17:52:43.134524 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.134503    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e44bd846-8fa1-4527-92a2-d712bf30bcfa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-596664c988-4sbx5\" (UID: \"e44bd846-8fa1-4527-92a2-d712bf30bcfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"
Sep 17 17:52:43.150749 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.150730    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8gcf\" (UniqueName: \"kubernetes.io/projected/e44bd846-8fa1-4527-92a2-d712bf30bcfa-kube-api-access-z8gcf\") pod \"isvc-canary-stop-v2-predictor-596664c988-4sbx5\" (UID: \"e44bd846-8fa1-4527-92a2-d712bf30bcfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"
Sep 17 17:52:43.234480 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.234453    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87869ebb-dd0c-481e-93ff-c116602baf23-cabundle-cert\") pod \"isvc-canary-stop-predictor-5cdcf6545-qsl6n\" (UID: \"87869ebb-dd0c-481e-93ff-c116602baf23\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"
Sep 17 17:52:43.234593 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.234499    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87869ebb-dd0c-481e-93ff-c116602baf23-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5cdcf6545-qsl6n\" (UID: \"87869ebb-dd0c-481e-93ff-c116602baf23\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"
Sep 17 17:52:43.234593 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.234540    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx449\" (UniqueName: \"kubernetes.io/projected/87869ebb-dd0c-481e-93ff-c116602baf23-kube-api-access-qx449\") pod \"isvc-canary-stop-predictor-5cdcf6545-qsl6n\" (UID: \"87869ebb-dd0c-481e-93ff-c116602baf23\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"
Sep 17 17:52:43.234593 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.234559    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87869ebb-dd0c-481e-93ff-c116602baf23-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5cdcf6545-qsl6n\" (UID: \"87869ebb-dd0c-481e-93ff-c116602baf23\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"
Sep 17 17:52:43.234593 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.234587    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87869ebb-dd0c-481e-93ff-c116602baf23-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-qsl6n\" (UID: \"87869ebb-dd0c-481e-93ff-c116602baf23\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"
Sep 17 17:52:43.234948 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.234911    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87869ebb-dd0c-481e-93ff-c116602baf23-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5cdcf6545-qsl6n\" (UID: \"87869ebb-dd0c-481e-93ff-c116602baf23\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"
Sep 17 17:52:43.235254 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.235223    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87869ebb-dd0c-481e-93ff-c116602baf23-cabundle-cert\") pod \"isvc-canary-stop-predictor-5cdcf6545-qsl6n\" (UID: \"87869ebb-dd0c-481e-93ff-c116602baf23\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"
Sep 17 17:52:43.235254 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.235260    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87869ebb-dd0c-481e-93ff-c116602baf23-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5cdcf6545-qsl6n\" (UID: \"87869ebb-dd0c-481e-93ff-c116602baf23\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"
Sep 17 17:52:43.235473 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.235384    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:43.236940 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.236922    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87869ebb-dd0c-481e-93ff-c116602baf23-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-qsl6n\" (UID: \"87869ebb-dd0c-481e-93ff-c116602baf23\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"
Sep 17 17:52:43.247479 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.247458    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx449\" (UniqueName: \"kubernetes.io/projected/87869ebb-dd0c-481e-93ff-c116602baf23-kube-api-access-qx449\") pod \"isvc-canary-stop-predictor-5cdcf6545-qsl6n\" (UID: \"87869ebb-dd0c-481e-93ff-c116602baf23\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"
Sep 17 17:52:43.360695 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.360631    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"
Sep 17 17:52:43.637905 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.637820    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e44bd846-8fa1-4527-92a2-d712bf30bcfa-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-596664c988-4sbx5\" (UID: \"e44bd846-8fa1-4527-92a2-d712bf30bcfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"
Sep 17 17:52:43.640144 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.640119    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e44bd846-8fa1-4527-92a2-d712bf30bcfa-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-596664c988-4sbx5\" (UID: \"e44bd846-8fa1-4527-92a2-d712bf30bcfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"
Sep 17 17:52:43.652844 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.652811    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"]
Sep 17 17:52:43.653487 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:52:43.653457    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87869ebb_dd0c_481e_93ff_c116602baf23.slice/crio-a919245c0aad63b619e0d286769bdebe0c3e10cf993601fd73faea7cda5abd67 WatchSource:0}: Error finding container a919245c0aad63b619e0d286769bdebe0c3e10cf993601fd73faea7cda5abd67: Status 404 returned error can't find the container with id a919245c0aad63b619e0d286769bdebe0c3e10cf993601fd73faea7cda5abd67
Sep 17 17:52:43.844798 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:43.844761    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"
Sep 17 17:52:44.128174 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:44.127989    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"]
Sep 17 17:52:44.130674 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:52:44.130647    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44bd846_8fa1_4527_92a2_d712bf30bcfa.slice/crio-fe8620f407a0082cd874324358abd27c3235da437e2416b81dd254b76fa43b6a WatchSource:0}: Error finding container fe8620f407a0082cd874324358abd27c3235da437e2416b81dd254b76fa43b6a: Status 404 returned error can't find the container with id fe8620f407a0082cd874324358abd27c3235da437e2416b81dd254b76fa43b6a
Sep 17 17:52:44.442580 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:44.442490    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n" event={"ID":"87869ebb-dd0c-481e-93ff-c116602baf23","Type":"ContainerStarted","Data":"c7262bd8b674dbfb457b1b6a6cdd28e1709e58641f96e98a8739e1ebf4f26f5f"}
Sep 17 17:52:44.442580 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:44.442562    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n" event={"ID":"87869ebb-dd0c-481e-93ff-c116602baf23","Type":"ContainerStarted","Data":"a919245c0aad63b619e0d286769bdebe0c3e10cf993601fd73faea7cda5abd67"}
Sep 17 17:52:44.444121 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:44.444095    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5" event={"ID":"e44bd846-8fa1-4527-92a2-d712bf30bcfa","Type":"ContainerStarted","Data":"63fa42d62d955e820ec788460b26841b99df22f2c6245a11564b9c9d4a9ba6f1"}
Sep 17 17:52:44.444239 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:44.444151    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5" event={"ID":"e44bd846-8fa1-4527-92a2-d712bf30bcfa","Type":"ContainerStarted","Data":"fe8620f407a0082cd874324358abd27c3235da437e2416b81dd254b76fa43b6a"}
Sep 17 17:52:48.460554 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:48.460477    2574 generic.go:356] "Generic (PLEG): container finished" podID="e44bd846-8fa1-4527-92a2-d712bf30bcfa" containerID="63fa42d62d955e820ec788460b26841b99df22f2c6245a11564b9c9d4a9ba6f1" exitCode=0
Sep 17 17:52:48.460926 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:48.460549    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5" event={"ID":"e44bd846-8fa1-4527-92a2-d712bf30bcfa","Type":"ContainerDied","Data":"63fa42d62d955e820ec788460b26841b99df22f2c6245a11564b9c9d4a9ba6f1"}
Sep 17 17:52:48.462267 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:48.462242    2574 generic.go:356] "Generic (PLEG): container finished" podID="87869ebb-dd0c-481e-93ff-c116602baf23" containerID="c7262bd8b674dbfb457b1b6a6cdd28e1709e58641f96e98a8739e1ebf4f26f5f" exitCode=0
Sep 17 17:52:48.462376 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:48.462280    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n" event={"ID":"87869ebb-dd0c-481e-93ff-c116602baf23","Type":"ContainerDied","Data":"c7262bd8b674dbfb457b1b6a6cdd28e1709e58641f96e98a8739e1ebf4f26f5f"}
Sep 17 17:52:49.468326 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:49.468285    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n" event={"ID":"87869ebb-dd0c-481e-93ff-c116602baf23","Type":"ContainerStarted","Data":"230b83960581681a6fd1c5737e1496659bcc10270bc5f7757bc0966bc278eac9"}
Sep 17 17:52:49.468788 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:49.468371    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n" event={"ID":"87869ebb-dd0c-481e-93ff-c116602baf23","Type":"ContainerStarted","Data":"c11f7b01fcf0e446be973d962d981a1577a9a487db8f3ced69c00cd10016d6dd"}
Sep 17 17:52:49.468788 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:49.468565    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"
Sep 17 17:52:49.470576 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:49.470555    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5" event={"ID":"e44bd846-8fa1-4527-92a2-d712bf30bcfa","Type":"ContainerStarted","Data":"da1418c667e5474c1a3c06d2fe0b036a31ba07f179ac5a4a9fe8364394ff47d4"}
Sep 17 17:52:49.470698 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:49.470598    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5" event={"ID":"e44bd846-8fa1-4527-92a2-d712bf30bcfa","Type":"ContainerStarted","Data":"cf7c4330d4ab12496b1433f787f5c9cb20853a1eff8ea570bbd1982dd5f25cee"}
Sep 17 17:52:49.470857 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:49.470838    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"
Sep 17 17:52:49.470902 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:49.470892    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"
Sep 17 17:52:49.472042 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:49.472014    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5" podUID="e44bd846-8fa1-4527-92a2-d712bf30bcfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:52:49.492922 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:49.492878    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n" podStartSLOduration=2.667271595 podStartE2EDuration="6.492865632s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.825594037s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:49.490920821 +0000 UTC m=+1144.820020436" watchObservedRunningTime="2026-09-17 17:52:49.492865632 +0000 UTC m=+1144.821965255"
Sep 17 17:52:49.511999 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:49.511959    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5" podStartSLOduration=3.530349347 podStartE2EDuration="7.511946944s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.981597597s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:49.511466999 +0000 UTC m=+1144.840566624" watchObservedRunningTime="2026-09-17 17:52:49.511946944 +0000 UTC m=+1144.841046566"
Sep 17 17:52:50.475343 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:50.475292    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5" podUID="e44bd846-8fa1-4527-92a2-d712bf30bcfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:52:50.475791 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:50.475396    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"
Sep 17 17:52:50.476752 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:50.476728    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n" podUID="87869ebb-dd0c-481e-93ff-c116602baf23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:52:51.480540 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:51.480501    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n" podUID="87869ebb-dd0c-481e-93ff-c116602baf23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:52:52.916272 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:52.916241    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:53.060253 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:53.060223    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:53.193459 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:53.193374    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:53.301499 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:53.301466    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:53.405571 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:53.405538    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"]
Sep 17 17:52:53.409357 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:53.409338    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"
Sep 17 17:52:53.411988 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:53.411966    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-68381-predictor-serving-cert\""
Sep 17 17:52:53.411988 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:53.411982    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-68381-kube-rbac-proxy-sar-config\""
Sep 17 17:52:53.434348 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:53.434324    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"]
Sep 17 17:52:53.434522 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:53.434507    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:53.514436 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:53.514356    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdd78e43-e851-466a-8613-0bf5abfa8c8b-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw\" (UID: \"fdd78e43-e851-466a-8613-0bf5abfa8c8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"
Sep 17 17:52:53.514436 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:53.514402    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd78e43-e851-466a-8613-0bf5abfa8c8b-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw\" (UID: \"fdd78e43-e851-466a-8613-0bf5abfa8c8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"
Sep 17 17:52:53.514599 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:53.514460    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdd78e43-e851-466a-8613-0bf5abfa8c8b-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw\" (UID: \"fdd78e43-e851-466a-8613-0bf5abfa8c8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"
Sep 17 17:52:53.514599 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:53.514502    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdd78e43-e851-466a-8613-0bf5abfa8c8b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw\" (UID: \"fdd78e43-e851-466a-8613-0bf5abfa8c8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"
Sep 17 17:52:53.514599 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:53.514530    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r4ll\" (UniqueName: \"kubernetes.io/projected/fdd78e43-e851-466a-8613-0bf5abfa8c8b-kube-api-access-7r4ll\") pod \"isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw\" (UID: \"fdd78e43-e851-466a-8613-0bf5abfa8c8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"
Sep 17 17:52:53.615770 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:53.615734    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r4ll\" (UniqueName: \"kubernetes.io/projected/fdd78e43-e851-466a-8613-0bf5abfa8c8b-kube-api-access-7r4ll\") pod \"isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw\" (UID: \"fdd78e43-e851-466a-8613-0bf5abfa8c8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"
Sep 17 17:52:53.615931 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:53.615800    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdd78e43-e851-466a-8613-0bf5abfa8c8b-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw\" (UID: \"fdd78e43-e851-466a-8613-0bf5abfa8c8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"
Sep 17 17:52:53.615931 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:53.615834    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd78e43-e851-466a-8613-0bf5abfa8c8b-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw\" (UID: \"fdd78e43-e851-466a-8613-0bf5abfa8c8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"
Sep 17 17:52:53.615931 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:53.615856    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdd78e43-e851-466a-8613-0bf5abfa8c8b-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw\" (UID: \"fdd78e43-e851-466a-8613-0bf5abfa8c8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"
Sep 17 17:52:53.615931 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:53.615876    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdd78e43-e851-466a-8613-0bf5abfa8c8b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw\" (UID: \"fdd78e43-e851-466a-8613-0bf5abfa8c8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"
Sep 17 17:52:53.615931 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:52:53.615905    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-68381-predictor-serving-cert" not found
Sep 17 17:52:53.616226 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:52:53.615977    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd78e43-e851-466a-8613-0bf5abfa8c8b-proxy-tls podName:fdd78e43-e851-466a-8613-0bf5abfa8c8b nodeName:}" failed. No retries permitted until 2026-09-17 17:52:54.115955209 +0000 UTC m=+1149.445054829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fdd78e43-e851-466a-8613-0bf5abfa8c8b-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw" (UID: "fdd78e43-e851-466a-8613-0bf5abfa8c8b") : secret "isvc-xgboost-graph-raw-hpa-68381-predictor-serving-cert" not found
Sep 17 17:52:53.616345 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:53.616324    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd78e43-e851-466a-8613-0bf5abfa8c8b-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw\" (UID: \"fdd78e43-e851-466a-8613-0bf5abfa8c8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"
Sep 17 17:52:53.616515 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:53.616497    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdd78e43-e851-466a-8613-0bf5abfa8c8b-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw\" (UID: \"fdd78e43-e851-466a-8613-0bf5abfa8c8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"
Sep 17 17:52:53.616589 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:53.616570    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdd78e43-e851-466a-8613-0bf5abfa8c8b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw\" (UID: \"fdd78e43-e851-466a-8613-0bf5abfa8c8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"
Sep 17 17:52:53.628445 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:53.628422    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r4ll\" (UniqueName: \"kubernetes.io/projected/fdd78e43-e851-466a-8613-0bf5abfa8c8b-kube-api-access-7r4ll\") pod \"isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw\" (UID: \"fdd78e43-e851-466a-8613-0bf5abfa8c8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"
Sep 17 17:52:53.944400 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:53.944364    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:54.119269 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:54.119232    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdd78e43-e851-466a-8613-0bf5abfa8c8b-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw\" (UID: \"fdd78e43-e851-466a-8613-0bf5abfa8c8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"
Sep 17 17:52:54.121562 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:54.121540    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdd78e43-e851-466a-8613-0bf5abfa8c8b-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw\" (UID: \"fdd78e43-e851-466a-8613-0bf5abfa8c8b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"
Sep 17 17:52:54.318935 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:54.318860    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"
Sep 17 17:52:54.598148 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:54.598124    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"]
Sep 17 17:52:55.480329 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:55.480296    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"
Sep 17 17:52:55.480937 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:55.480900    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5" podUID="e44bd846-8fa1-4527-92a2-d712bf30bcfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:52:55.497906 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:55.497872    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw" event={"ID":"fdd78e43-e851-466a-8613-0bf5abfa8c8b","Type":"ContainerStarted","Data":"0f0dd96a57fdbc3674ce589d2c98f3636d6fc8d15e3091a84aae5316d9bc03b4"}
Sep 17 17:52:55.498022 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:55.497945    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw" event={"ID":"fdd78e43-e851-466a-8613-0bf5abfa8c8b","Type":"ContainerStarted","Data":"43b947d5fe86278c21d6138a6e06bd3690ea08060dbce6c3f101a66f74db94eb"}
Sep 17 17:52:56.484956 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:56.484923    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"
Sep 17 17:52:56.485596 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:56.485559    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n" podUID="87869ebb-dd0c-481e-93ff-c116602baf23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:52:59.517432 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:59.517398    2574 generic.go:356] "Generic (PLEG): container finished" podID="fdd78e43-e851-466a-8613-0bf5abfa8c8b" containerID="0f0dd96a57fdbc3674ce589d2c98f3636d6fc8d15e3091a84aae5316d9bc03b4" exitCode=0
Sep 17 17:52:59.517819 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:59.517454    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw" event={"ID":"fdd78e43-e851-466a-8613-0bf5abfa8c8b","Type":"ContainerDied","Data":"0f0dd96a57fdbc3674ce589d2c98f3636d6fc8d15e3091a84aae5316d9bc03b4"}
Sep 17 17:52:59.518750 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:52:59.518728    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:53:05.481582 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:53:05.481540    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5" podUID="e44bd846-8fa1-4527-92a2-d712bf30bcfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:53:06.485465 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:53:06.485424    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n" podUID="87869ebb-dd0c-481e-93ff-c116602baf23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:53:15.480922 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:53:15.480828    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5" podUID="e44bd846-8fa1-4527-92a2-d712bf30bcfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:53:16.485936 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:53:16.485893    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n" podUID="87869ebb-dd0c-481e-93ff-c116602baf23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:53:18.608484 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:53:18.608457    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw" event={"ID":"fdd78e43-e851-466a-8613-0bf5abfa8c8b","Type":"ContainerStarted","Data":"43c3cac3623fbc91866ee1dd47033a1a9b79c29e3c73cd68172f2321a320f5fa"}
Sep 17 17:53:19.614727 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:53:19.614687    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw" event={"ID":"fdd78e43-e851-466a-8613-0bf5abfa8c8b","Type":"ContainerStarted","Data":"63b106eaee2c66b2673fb58f508c01d835d529a82f9209319c6f1ded7834b4b5"}
Sep 17 17:53:19.615173 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:53:19.614758    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"
Sep 17 17:53:19.615173 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:53:19.614902    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"
Sep 17 17:53:19.616066 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:53:19.616033    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw" podUID="fdd78e43-e851-466a-8613-0bf5abfa8c8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 17:53:19.637159 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:53:19.637113    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw" podStartSLOduration=3.621274399 podStartE2EDuration="26.637099882s" totalImagesPullingTime="18.992830019s" totalInitContainerRuntime="4.022995464s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:53:19.635211501 +0000 UTC m=+1174.964311160" watchObservedRunningTime="2026-09-17 17:53:19.637099882 +0000 UTC m=+1174.966199505"
Sep 17 17:53:20.618682 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:53:20.618638    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw" podUID="fdd78e43-e851-466a-8613-0bf5abfa8c8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 17:53:25.481599 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:53:25.481551    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5" podUID="e44bd846-8fa1-4527-92a2-d712bf30bcfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:53:25.623222 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:53:25.623196    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"
Sep 17 17:53:25.623700 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:53:25.623673    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw" podUID="fdd78e43-e851-466a-8613-0bf5abfa8c8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 17:53:26.486398 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:53:26.486361    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n" podUID="87869ebb-dd0c-481e-93ff-c116602baf23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:53:35.480805 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:53:35.480761    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5" podUID="e44bd846-8fa1-4527-92a2-d712bf30bcfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:53:35.624623 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:53:35.624580    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw" podUID="fdd78e43-e851-466a-8613-0bf5abfa8c8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 17:53:36.485438 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:53:36.485400    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n" podUID="87869ebb-dd0c-481e-93ff-c116602baf23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:53:45.481757 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:53:45.481714    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5" podUID="e44bd846-8fa1-4527-92a2-d712bf30bcfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:53:45.624459 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:53:45.624419    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw" podUID="fdd78e43-e851-466a-8613-0bf5abfa8c8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 17:53:46.486432 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:53:46.486395    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n" podUID="87869ebb-dd0c-481e-93ff-c116602baf23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:53:55.481748 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:53:55.481699    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5" podUID="e44bd846-8fa1-4527-92a2-d712bf30bcfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:53:55.624506 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:53:55.624457    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw" podUID="fdd78e43-e851-466a-8613-0bf5abfa8c8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 17:53:56.485962 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:53:56.485917    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n" podUID="87869ebb-dd0c-481e-93ff-c116602baf23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:53:59.111216 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:53:59.111161    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n" podUID="87869ebb-dd0c-481e-93ff-c116602baf23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:54:05.481490 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:05.481438    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5" podUID="e44bd846-8fa1-4527-92a2-d712bf30bcfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:54:05.623614 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:05.623577    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw" podUID="fdd78e43-e851-466a-8613-0bf5abfa8c8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 17:54:09.114838 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:09.114812    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"
Sep 17 17:54:15.482203 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:15.482170    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"
Sep 17 17:54:15.623732 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:15.623690    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw" podUID="fdd78e43-e851-466a-8613-0bf5abfa8c8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 17:54:15.692066 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:15.692018    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"]
Sep 17 17:54:15.692279 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:15.692257    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n" podUID="87869ebb-dd0c-481e-93ff-c116602baf23" containerName="kserve-container" containerID="cri-o://c11f7b01fcf0e446be973d962d981a1577a9a487db8f3ced69c00cd10016d6dd" gracePeriod=30
Sep 17 17:54:15.692363 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:15.692314    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n" podUID="87869ebb-dd0c-481e-93ff-c116602baf23" containerName="kube-rbac-proxy" containerID="cri-o://230b83960581681a6fd1c5737e1496659bcc10270bc5f7757bc0966bc278eac9" gracePeriod=30
Sep 17 17:54:15.849170 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:15.849138    2574 generic.go:356] "Generic (PLEG): container finished" podID="87869ebb-dd0c-481e-93ff-c116602baf23" containerID="230b83960581681a6fd1c5737e1496659bcc10270bc5f7757bc0966bc278eac9" exitCode=2
Sep 17 17:54:15.849304 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:15.849208    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n" event={"ID":"87869ebb-dd0c-481e-93ff-c116602baf23","Type":"ContainerDied","Data":"230b83960581681a6fd1c5737e1496659bcc10270bc5f7757bc0966bc278eac9"}
Sep 17 17:54:16.480940 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:16.480899    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n" podUID="87869ebb-dd0c-481e-93ff-c116602baf23" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 17 17:54:17.830900 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:17.830868    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"]
Sep 17 17:54:17.831377 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:17.831095    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5" podUID="e44bd846-8fa1-4527-92a2-d712bf30bcfa" containerName="kserve-container" containerID="cri-o://cf7c4330d4ab12496b1433f787f5c9cb20853a1eff8ea570bbd1982dd5f25cee" gracePeriod=30
Sep 17 17:54:17.831377 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:17.831121    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5" podUID="e44bd846-8fa1-4527-92a2-d712bf30bcfa" containerName="kube-rbac-proxy" containerID="cri-o://da1418c667e5474c1a3c06d2fe0b036a31ba07f179ac5a4a9fe8364394ff47d4" gracePeriod=30
Sep 17 17:54:17.883616 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:17.883590    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:18.102261 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:18.102171    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:18.863213 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:18.863180    2574 generic.go:356] "Generic (PLEG): container finished" podID="e44bd846-8fa1-4527-92a2-d712bf30bcfa" containerID="da1418c667e5474c1a3c06d2fe0b036a31ba07f179ac5a4a9fe8364394ff47d4" exitCode=2
Sep 17 17:54:18.863589 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:18.863255    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5" event={"ID":"e44bd846-8fa1-4527-92a2-d712bf30bcfa","Type":"ContainerDied","Data":"da1418c667e5474c1a3c06d2fe0b036a31ba07f179ac5a4a9fe8364394ff47d4"}
Sep 17 17:54:19.111985 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:19.111945    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n" podUID="87869ebb-dd0c-481e-93ff-c116602baf23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:54:20.231002 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.230979    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"
Sep 17 17:54:20.313939 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.313905    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87869ebb-dd0c-481e-93ff-c116602baf23-kube-rbac-proxy-sar-config\") pod \"87869ebb-dd0c-481e-93ff-c116602baf23\" (UID: \"87869ebb-dd0c-481e-93ff-c116602baf23\") "
Sep 17 17:54:20.314124 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.313956    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87869ebb-dd0c-481e-93ff-c116602baf23-cabundle-cert\") pod \"87869ebb-dd0c-481e-93ff-c116602baf23\" (UID: \"87869ebb-dd0c-481e-93ff-c116602baf23\") "
Sep 17 17:54:20.314124 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.314003    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qx449\" (UniqueName: \"kubernetes.io/projected/87869ebb-dd0c-481e-93ff-c116602baf23-kube-api-access-qx449\") pod \"87869ebb-dd0c-481e-93ff-c116602baf23\" (UID: \"87869ebb-dd0c-481e-93ff-c116602baf23\") "
Sep 17 17:54:20.314124 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.314076    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87869ebb-dd0c-481e-93ff-c116602baf23-kserve-provision-location\") pod \"87869ebb-dd0c-481e-93ff-c116602baf23\" (UID: \"87869ebb-dd0c-481e-93ff-c116602baf23\") "
Sep 17 17:54:20.314124 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.314119    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87869ebb-dd0c-481e-93ff-c116602baf23-proxy-tls\") pod \"87869ebb-dd0c-481e-93ff-c116602baf23\" (UID: \"87869ebb-dd0c-481e-93ff-c116602baf23\") "
Sep 17 17:54:20.314445 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.314417    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87869ebb-dd0c-481e-93ff-c116602baf23-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "87869ebb-dd0c-481e-93ff-c116602baf23" (UID: "87869ebb-dd0c-481e-93ff-c116602baf23"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:54:20.314445 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.314431    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87869ebb-dd0c-481e-93ff-c116602baf23-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "87869ebb-dd0c-481e-93ff-c116602baf23" (UID: "87869ebb-dd0c-481e-93ff-c116602baf23"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:54:20.314604 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.314429    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87869ebb-dd0c-481e-93ff-c116602baf23-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "87869ebb-dd0c-481e-93ff-c116602baf23" (UID: "87869ebb-dd0c-481e-93ff-c116602baf23"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:54:20.316108 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.316074    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87869ebb-dd0c-481e-93ff-c116602baf23-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "87869ebb-dd0c-481e-93ff-c116602baf23" (UID: "87869ebb-dd0c-481e-93ff-c116602baf23"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:54:20.316213 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.316174    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87869ebb-dd0c-481e-93ff-c116602baf23-kube-api-access-qx449" (OuterVolumeSpecName: "kube-api-access-qx449") pod "87869ebb-dd0c-481e-93ff-c116602baf23" (UID: "87869ebb-dd0c-481e-93ff-c116602baf23"). InnerVolumeSpecName "kube-api-access-qx449". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:54:20.415675 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.415645    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87869ebb-dd0c-481e-93ff-c116602baf23-proxy-tls\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:54:20.415675 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.415675    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87869ebb-dd0c-481e-93ff-c116602baf23-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:54:20.415865 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.415690    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87869ebb-dd0c-481e-93ff-c116602baf23-cabundle-cert\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:54:20.415865 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.415704    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx449\" (UniqueName: \"kubernetes.io/projected/87869ebb-dd0c-481e-93ff-c116602baf23-kube-api-access-qx449\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:54:20.415865 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.415716    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87869ebb-dd0c-481e-93ff-c116602baf23-kserve-provision-location\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:54:20.476407 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.476369    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5" podUID="e44bd846-8fa1-4527-92a2-d712bf30bcfa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Sep 17 17:54:20.872227 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.872189    2574 generic.go:356] "Generic (PLEG): container finished" podID="87869ebb-dd0c-481e-93ff-c116602baf23" containerID="c11f7b01fcf0e446be973d962d981a1577a9a487db8f3ced69c00cd10016d6dd" exitCode=0
Sep 17 17:54:20.872389 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.872253    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"
Sep 17 17:54:20.872389 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.872278    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n" event={"ID":"87869ebb-dd0c-481e-93ff-c116602baf23","Type":"ContainerDied","Data":"c11f7b01fcf0e446be973d962d981a1577a9a487db8f3ced69c00cd10016d6dd"}
Sep 17 17:54:20.872483 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.872390    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n" event={"ID":"87869ebb-dd0c-481e-93ff-c116602baf23","Type":"ContainerDied","Data":"a919245c0aad63b619e0d286769bdebe0c3e10cf993601fd73faea7cda5abd67"}
Sep 17 17:54:20.872483 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.872435    2574 scope.go:117] "RemoveContainer" containerID="230b83960581681a6fd1c5737e1496659bcc10270bc5f7757bc0966bc278eac9"
Sep 17 17:54:20.883993 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.883966    2574 scope.go:117] "RemoveContainer" containerID="c11f7b01fcf0e446be973d962d981a1577a9a487db8f3ced69c00cd10016d6dd"
Sep 17 17:54:20.891456 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.891439    2574 scope.go:117] "RemoveContainer" containerID="c7262bd8b674dbfb457b1b6a6cdd28e1709e58641f96e98a8739e1ebf4f26f5f"
Sep 17 17:54:20.899045 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.899021    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"]
Sep 17 17:54:20.899388 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.899374    2574 scope.go:117] "RemoveContainer" containerID="230b83960581681a6fd1c5737e1496659bcc10270bc5f7757bc0966bc278eac9"
Sep 17 17:54:20.899656 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:54:20.899635    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"230b83960581681a6fd1c5737e1496659bcc10270bc5f7757bc0966bc278eac9\": container with ID starting with 230b83960581681a6fd1c5737e1496659bcc10270bc5f7757bc0966bc278eac9 not found: ID does not exist" containerID="230b83960581681a6fd1c5737e1496659bcc10270bc5f7757bc0966bc278eac9"
Sep 17 17:54:20.899719 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.899666    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"230b83960581681a6fd1c5737e1496659bcc10270bc5f7757bc0966bc278eac9"} err="failed to get container status \"230b83960581681a6fd1c5737e1496659bcc10270bc5f7757bc0966bc278eac9\": rpc error: code = NotFound desc = could not find container \"230b83960581681a6fd1c5737e1496659bcc10270bc5f7757bc0966bc278eac9\": container with ID starting with 230b83960581681a6fd1c5737e1496659bcc10270bc5f7757bc0966bc278eac9 not found: ID does not exist"
Sep 17 17:54:20.899719 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.899684    2574 scope.go:117] "RemoveContainer" containerID="c11f7b01fcf0e446be973d962d981a1577a9a487db8f3ced69c00cd10016d6dd"
Sep 17 17:54:20.899948 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:54:20.899926    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c11f7b01fcf0e446be973d962d981a1577a9a487db8f3ced69c00cd10016d6dd\": container with ID starting with c11f7b01fcf0e446be973d962d981a1577a9a487db8f3ced69c00cd10016d6dd not found: ID does not exist" containerID="c11f7b01fcf0e446be973d962d981a1577a9a487db8f3ced69c00cd10016d6dd"
Sep 17 17:54:20.899997 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.899955    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c11f7b01fcf0e446be973d962d981a1577a9a487db8f3ced69c00cd10016d6dd"} err="failed to get container status \"c11f7b01fcf0e446be973d962d981a1577a9a487db8f3ced69c00cd10016d6dd\": rpc error: code = NotFound desc = could not find container \"c11f7b01fcf0e446be973d962d981a1577a9a487db8f3ced69c00cd10016d6dd\": container with ID starting with c11f7b01fcf0e446be973d962d981a1577a9a487db8f3ced69c00cd10016d6dd not found: ID does not exist"
Sep 17 17:54:20.899997 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.899977    2574 scope.go:117] "RemoveContainer" containerID="c7262bd8b674dbfb457b1b6a6cdd28e1709e58641f96e98a8739e1ebf4f26f5f"
Sep 17 17:54:20.900245 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:54:20.900229    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7262bd8b674dbfb457b1b6a6cdd28e1709e58641f96e98a8739e1ebf4f26f5f\": container with ID starting with c7262bd8b674dbfb457b1b6a6cdd28e1709e58641f96e98a8739e1ebf4f26f5f not found: ID does not exist" containerID="c7262bd8b674dbfb457b1b6a6cdd28e1709e58641f96e98a8739e1ebf4f26f5f"
Sep 17 17:54:20.900298 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.900247    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7262bd8b674dbfb457b1b6a6cdd28e1709e58641f96e98a8739e1ebf4f26f5f"} err="failed to get container status \"c7262bd8b674dbfb457b1b6a6cdd28e1709e58641f96e98a8739e1ebf4f26f5f\": rpc error: code = NotFound desc = could not find container \"c7262bd8b674dbfb457b1b6a6cdd28e1709e58641f96e98a8739e1ebf4f26f5f\": container with ID starting with c7262bd8b674dbfb457b1b6a6cdd28e1709e58641f96e98a8739e1ebf4f26f5f not found: ID does not exist"
Sep 17 17:54:20.903278 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:20.903260    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-qsl6n"]
Sep 17 17:54:21.115382 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:21.115349    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87869ebb-dd0c-481e-93ff-c116602baf23" path="/var/lib/kubelet/pods/87869ebb-dd0c-481e-93ff-c116602baf23/volumes"
Sep 17 17:54:22.674305 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.674283    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"
Sep 17 17:54:22.835367 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.835344    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8gcf\" (UniqueName: \"kubernetes.io/projected/e44bd846-8fa1-4527-92a2-d712bf30bcfa-kube-api-access-z8gcf\") pod \"e44bd846-8fa1-4527-92a2-d712bf30bcfa\" (UID: \"e44bd846-8fa1-4527-92a2-d712bf30bcfa\") "
Sep 17 17:54:22.835537 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.835387    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e44bd846-8fa1-4527-92a2-d712bf30bcfa-kserve-provision-location\") pod \"e44bd846-8fa1-4527-92a2-d712bf30bcfa\" (UID: \"e44bd846-8fa1-4527-92a2-d712bf30bcfa\") "
Sep 17 17:54:22.835537 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.835412    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e44bd846-8fa1-4527-92a2-d712bf30bcfa-cabundle-cert\") pod \"e44bd846-8fa1-4527-92a2-d712bf30bcfa\" (UID: \"e44bd846-8fa1-4527-92a2-d712bf30bcfa\") "
Sep 17 17:54:22.835649 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.835532    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e44bd846-8fa1-4527-92a2-d712bf30bcfa-proxy-tls\") pod \"e44bd846-8fa1-4527-92a2-d712bf30bcfa\" (UID: \"e44bd846-8fa1-4527-92a2-d712bf30bcfa\") "
Sep 17 17:54:22.835649 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.835588    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e44bd846-8fa1-4527-92a2-d712bf30bcfa-kube-rbac-proxy-sar-config\") pod \"e44bd846-8fa1-4527-92a2-d712bf30bcfa\" (UID: \"e44bd846-8fa1-4527-92a2-d712bf30bcfa\") "
Sep 17 17:54:22.835749 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.835709    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44bd846-8fa1-4527-92a2-d712bf30bcfa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e44bd846-8fa1-4527-92a2-d712bf30bcfa" (UID: "e44bd846-8fa1-4527-92a2-d712bf30bcfa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:54:22.835805 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.835758    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e44bd846-8fa1-4527-92a2-d712bf30bcfa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e44bd846-8fa1-4527-92a2-d712bf30bcfa" (UID: "e44bd846-8fa1-4527-92a2-d712bf30bcfa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:54:22.835951 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.835929    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e44bd846-8fa1-4527-92a2-d712bf30bcfa-kserve-provision-location\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:54:22.836003 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.835949    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e44bd846-8fa1-4527-92a2-d712bf30bcfa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e44bd846-8fa1-4527-92a2-d712bf30bcfa" (UID: "e44bd846-8fa1-4527-92a2-d712bf30bcfa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:54:22.836003 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.835958    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e44bd846-8fa1-4527-92a2-d712bf30bcfa-cabundle-cert\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:54:22.837461 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.837442    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e44bd846-8fa1-4527-92a2-d712bf30bcfa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e44bd846-8fa1-4527-92a2-d712bf30bcfa" (UID: "e44bd846-8fa1-4527-92a2-d712bf30bcfa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:54:22.837537 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.837519    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44bd846-8fa1-4527-92a2-d712bf30bcfa-kube-api-access-z8gcf" (OuterVolumeSpecName: "kube-api-access-z8gcf") pod "e44bd846-8fa1-4527-92a2-d712bf30bcfa" (UID: "e44bd846-8fa1-4527-92a2-d712bf30bcfa"). InnerVolumeSpecName "kube-api-access-z8gcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:54:22.884333 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.884307    2574 generic.go:356] "Generic (PLEG): container finished" podID="e44bd846-8fa1-4527-92a2-d712bf30bcfa" containerID="cf7c4330d4ab12496b1433f787f5c9cb20853a1eff8ea570bbd1982dd5f25cee" exitCode=0
Sep 17 17:54:22.884451 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.884364    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"
Sep 17 17:54:22.884451 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.884390    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5" event={"ID":"e44bd846-8fa1-4527-92a2-d712bf30bcfa","Type":"ContainerDied","Data":"cf7c4330d4ab12496b1433f787f5c9cb20853a1eff8ea570bbd1982dd5f25cee"}
Sep 17 17:54:22.884559 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.884469    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5" event={"ID":"e44bd846-8fa1-4527-92a2-d712bf30bcfa","Type":"ContainerDied","Data":"fe8620f407a0082cd874324358abd27c3235da437e2416b81dd254b76fa43b6a"}
Sep 17 17:54:22.884559 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.884511    2574 scope.go:117] "RemoveContainer" containerID="da1418c667e5474c1a3c06d2fe0b036a31ba07f179ac5a4a9fe8364394ff47d4"
Sep 17 17:54:22.893322 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.893166    2574 scope.go:117] "RemoveContainer" containerID="cf7c4330d4ab12496b1433f787f5c9cb20853a1eff8ea570bbd1982dd5f25cee"
Sep 17 17:54:22.900380 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.900364    2574 scope.go:117] "RemoveContainer" containerID="63fa42d62d955e820ec788460b26841b99df22f2c6245a11564b9c9d4a9ba6f1"
Sep 17 17:54:22.907252 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.907234    2574 scope.go:117] "RemoveContainer" containerID="da1418c667e5474c1a3c06d2fe0b036a31ba07f179ac5a4a9fe8364394ff47d4"
Sep 17 17:54:22.907506 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:54:22.907486    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da1418c667e5474c1a3c06d2fe0b036a31ba07f179ac5a4a9fe8364394ff47d4\": container with ID starting with da1418c667e5474c1a3c06d2fe0b036a31ba07f179ac5a4a9fe8364394ff47d4 not found: ID does not exist" containerID="da1418c667e5474c1a3c06d2fe0b036a31ba07f179ac5a4a9fe8364394ff47d4"
Sep 17 17:54:22.907600 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.907513    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da1418c667e5474c1a3c06d2fe0b036a31ba07f179ac5a4a9fe8364394ff47d4"} err="failed to get container status \"da1418c667e5474c1a3c06d2fe0b036a31ba07f179ac5a4a9fe8364394ff47d4\": rpc error: code = NotFound desc = could not find container \"da1418c667e5474c1a3c06d2fe0b036a31ba07f179ac5a4a9fe8364394ff47d4\": container with ID starting with da1418c667e5474c1a3c06d2fe0b036a31ba07f179ac5a4a9fe8364394ff47d4 not found: ID does not exist"
Sep 17 17:54:22.907600 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.907531    2574 scope.go:117] "RemoveContainer" containerID="cf7c4330d4ab12496b1433f787f5c9cb20853a1eff8ea570bbd1982dd5f25cee"
Sep 17 17:54:22.907769 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:54:22.907756    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf7c4330d4ab12496b1433f787f5c9cb20853a1eff8ea570bbd1982dd5f25cee\": container with ID starting with cf7c4330d4ab12496b1433f787f5c9cb20853a1eff8ea570bbd1982dd5f25cee not found: ID does not exist" containerID="cf7c4330d4ab12496b1433f787f5c9cb20853a1eff8ea570bbd1982dd5f25cee"
Sep 17 17:54:22.907811 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.907771    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf7c4330d4ab12496b1433f787f5c9cb20853a1eff8ea570bbd1982dd5f25cee"} err="failed to get container status \"cf7c4330d4ab12496b1433f787f5c9cb20853a1eff8ea570bbd1982dd5f25cee\": rpc error: code = NotFound desc = could not find container \"cf7c4330d4ab12496b1433f787f5c9cb20853a1eff8ea570bbd1982dd5f25cee\": container with ID starting with cf7c4330d4ab12496b1433f787f5c9cb20853a1eff8ea570bbd1982dd5f25cee not found: ID does not exist"
Sep 17 17:54:22.907811 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.907784    2574 scope.go:117] "RemoveContainer" containerID="63fa42d62d955e820ec788460b26841b99df22f2c6245a11564b9c9d4a9ba6f1"
Sep 17 17:54:22.907974 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:54:22.907954    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63fa42d62d955e820ec788460b26841b99df22f2c6245a11564b9c9d4a9ba6f1\": container with ID starting with 63fa42d62d955e820ec788460b26841b99df22f2c6245a11564b9c9d4a9ba6f1 not found: ID does not exist" containerID="63fa42d62d955e820ec788460b26841b99df22f2c6245a11564b9c9d4a9ba6f1"
Sep 17 17:54:22.908013 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.907981    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63fa42d62d955e820ec788460b26841b99df22f2c6245a11564b9c9d4a9ba6f1"} err="failed to get container status \"63fa42d62d955e820ec788460b26841b99df22f2c6245a11564b9c9d4a9ba6f1\": rpc error: code = NotFound desc = could not find container \"63fa42d62d955e820ec788460b26841b99df22f2c6245a11564b9c9d4a9ba6f1\": container with ID starting with 63fa42d62d955e820ec788460b26841b99df22f2c6245a11564b9c9d4a9ba6f1 not found: ID does not exist"
Sep 17 17:54:22.914089 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.914067    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"]
Sep 17 17:54:22.924823 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.924797    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-4sbx5"]
Sep 17 17:54:22.936787 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.936767    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e44bd846-8fa1-4527-92a2-d712bf30bcfa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:54:22.936868 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.936792    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8gcf\" (UniqueName: \"kubernetes.io/projected/e44bd846-8fa1-4527-92a2-d712bf30bcfa-kube-api-access-z8gcf\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:54:22.936868 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:22.936809    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e44bd846-8fa1-4527-92a2-d712bf30bcfa-proxy-tls\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:54:23.114922 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:23.114844    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e44bd846-8fa1-4527-92a2-d712bf30bcfa" path="/var/lib/kubelet/pods/e44bd846-8fa1-4527-92a2-d712bf30bcfa/volumes"
Sep 17 17:54:25.625175 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:25.625147    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"
Sep 17 17:54:28.145271 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:28.145226    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:28.420221 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:28.420141    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:32.490672 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:32.490639    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:33.179128 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:33.179094    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:33.414595 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:33.414566    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:43.302048 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:43.301972    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:43.389784 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:43.389756    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"]
Sep 17 17:54:43.390015 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:43.389988    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw" podUID="fdd78e43-e851-466a-8613-0bf5abfa8c8b" containerName="kserve-container" containerID="cri-o://43c3cac3623fbc91866ee1dd47033a1a9b79c29e3c73cd68172f2321a320f5fa" gracePeriod=30
Sep 17 17:54:43.390152 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:43.390030    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw" podUID="fdd78e43-e851-466a-8613-0bf5abfa8c8b" containerName="kube-rbac-proxy" containerID="cri-o://63b106eaee2c66b2673fb58f508c01d835d529a82f9209319c6f1ded7834b4b5" gracePeriod=30
Sep 17 17:54:43.445639 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:43.445610    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:43.573617 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:43.573542    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:43.781877 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:43.781845    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:43.968683 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:43.968654    2574 generic.go:356] "Generic (PLEG): container finished" podID="fdd78e43-e851-466a-8613-0bf5abfa8c8b" containerID="63b106eaee2c66b2673fb58f508c01d835d529a82f9209319c6f1ded7834b4b5" exitCode=2
Sep 17 17:54:43.968829 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:43.968693    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw" event={"ID":"fdd78e43-e851-466a-8613-0bf5abfa8c8b","Type":"ContainerDied","Data":"63b106eaee2c66b2673fb58f508c01d835d529a82f9209319c6f1ded7834b4b5"}
Sep 17 17:54:45.619037 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:45.618996    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw" podUID="fdd78e43-e851-466a-8613-0bf5abfa8c8b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 17 17:54:45.623607 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:45.623582    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw" podUID="fdd78e43-e851-466a-8613-0bf5abfa8c8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 17:54:47.232908 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:47.232883    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"
Sep 17 17:54:47.312096 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:47.312067    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd78e43-e851-466a-8613-0bf5abfa8c8b-kserve-provision-location\") pod \"fdd78e43-e851-466a-8613-0bf5abfa8c8b\" (UID: \"fdd78e43-e851-466a-8613-0bf5abfa8c8b\") "
Sep 17 17:54:47.312260 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:47.312115    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7r4ll\" (UniqueName: \"kubernetes.io/projected/fdd78e43-e851-466a-8613-0bf5abfa8c8b-kube-api-access-7r4ll\") pod \"fdd78e43-e851-466a-8613-0bf5abfa8c8b\" (UID: \"fdd78e43-e851-466a-8613-0bf5abfa8c8b\") "
Sep 17 17:54:47.312260 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:47.312161    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdd78e43-e851-466a-8613-0bf5abfa8c8b-cabundle-cert\") pod \"fdd78e43-e851-466a-8613-0bf5abfa8c8b\" (UID: \"fdd78e43-e851-466a-8613-0bf5abfa8c8b\") "
Sep 17 17:54:47.312260 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:47.312182    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdd78e43-e851-466a-8613-0bf5abfa8c8b-proxy-tls\") pod \"fdd78e43-e851-466a-8613-0bf5abfa8c8b\" (UID: \"fdd78e43-e851-466a-8613-0bf5abfa8c8b\") "
Sep 17 17:54:47.312260 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:47.312243    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdd78e43-e851-466a-8613-0bf5abfa8c8b-kube-rbac-proxy-sar-config\") pod \"fdd78e43-e851-466a-8613-0bf5abfa8c8b\" (UID: \"fdd78e43-e851-466a-8613-0bf5abfa8c8b\") "
Sep 17 17:54:47.312475 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:47.312443    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd78e43-e851-466a-8613-0bf5abfa8c8b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fdd78e43-e851-466a-8613-0bf5abfa8c8b" (UID: "fdd78e43-e851-466a-8613-0bf5abfa8c8b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:54:47.312680 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:47.312564    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd78e43-e851-466a-8613-0bf5abfa8c8b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fdd78e43-e851-466a-8613-0bf5abfa8c8b" (UID: "fdd78e43-e851-466a-8613-0bf5abfa8c8b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:54:47.312831 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:47.312732    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd78e43-e851-466a-8613-0bf5abfa8c8b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fdd78e43-e851-466a-8613-0bf5abfa8c8b" (UID: "fdd78e43-e851-466a-8613-0bf5abfa8c8b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:54:47.314314 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:47.314285    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdd78e43-e851-466a-8613-0bf5abfa8c8b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fdd78e43-e851-466a-8613-0bf5abfa8c8b" (UID: "fdd78e43-e851-466a-8613-0bf5abfa8c8b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:54:47.314619 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:47.314591    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd78e43-e851-466a-8613-0bf5abfa8c8b-kube-api-access-7r4ll" (OuterVolumeSpecName: "kube-api-access-7r4ll") pod "fdd78e43-e851-466a-8613-0bf5abfa8c8b" (UID: "fdd78e43-e851-466a-8613-0bf5abfa8c8b"). InnerVolumeSpecName "kube-api-access-7r4ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:54:47.413262 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:47.413228    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdd78e43-e851-466a-8613-0bf5abfa8c8b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:54:47.413262 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:47.413257    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd78e43-e851-466a-8613-0bf5abfa8c8b-kserve-provision-location\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:54:47.413262 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:47.413269    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r4ll\" (UniqueName: \"kubernetes.io/projected/fdd78e43-e851-466a-8613-0bf5abfa8c8b-kube-api-access-7r4ll\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:54:47.413486 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:47.413283    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdd78e43-e851-466a-8613-0bf5abfa8c8b-cabundle-cert\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:54:47.413486 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:47.413295    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdd78e43-e851-466a-8613-0bf5abfa8c8b-proxy-tls\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:54:47.986495 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:47.986457    2574 generic.go:356] "Generic (PLEG): container finished" podID="fdd78e43-e851-466a-8613-0bf5abfa8c8b" containerID="43c3cac3623fbc91866ee1dd47033a1a9b79c29e3c73cd68172f2321a320f5fa" exitCode=0
Sep 17 17:54:47.986770 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:47.986520    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw" event={"ID":"fdd78e43-e851-466a-8613-0bf5abfa8c8b","Type":"ContainerDied","Data":"43c3cac3623fbc91866ee1dd47033a1a9b79c29e3c73cd68172f2321a320f5fa"}
Sep 17 17:54:47.986770 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:47.986531    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"
Sep 17 17:54:47.986770 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:47.986591    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw" event={"ID":"fdd78e43-e851-466a-8613-0bf5abfa8c8b","Type":"ContainerDied","Data":"43b947d5fe86278c21d6138a6e06bd3690ea08060dbce6c3f101a66f74db94eb"}
Sep 17 17:54:47.986770 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:47.986619    2574 scope.go:117] "RemoveContainer" containerID="63b106eaee2c66b2673fb58f508c01d835d529a82f9209319c6f1ded7834b4b5"
Sep 17 17:54:47.994834 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:47.994816    2574 scope.go:117] "RemoveContainer" containerID="43c3cac3623fbc91866ee1dd47033a1a9b79c29e3c73cd68172f2321a320f5fa"
Sep 17 17:54:48.001934 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:48.001912    2574 scope.go:117] "RemoveContainer" containerID="0f0dd96a57fdbc3674ce589d2c98f3636d6fc8d15e3091a84aae5316d9bc03b4"
Sep 17 17:54:48.008727 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:48.008709    2574 scope.go:117] "RemoveContainer" containerID="63b106eaee2c66b2673fb58f508c01d835d529a82f9209319c6f1ded7834b4b5"
Sep 17 17:54:48.008970 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:54:48.008952    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63b106eaee2c66b2673fb58f508c01d835d529a82f9209319c6f1ded7834b4b5\": container with ID starting with 63b106eaee2c66b2673fb58f508c01d835d529a82f9209319c6f1ded7834b4b5 not found: ID does not exist" containerID="63b106eaee2c66b2673fb58f508c01d835d529a82f9209319c6f1ded7834b4b5"
Sep 17 17:54:48.009017 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:48.008978    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63b106eaee2c66b2673fb58f508c01d835d529a82f9209319c6f1ded7834b4b5"} err="failed to get container status \"63b106eaee2c66b2673fb58f508c01d835d529a82f9209319c6f1ded7834b4b5\": rpc error: code = NotFound desc = could not find container \"63b106eaee2c66b2673fb58f508c01d835d529a82f9209319c6f1ded7834b4b5\": container with ID starting with 63b106eaee2c66b2673fb58f508c01d835d529a82f9209319c6f1ded7834b4b5 not found: ID does not exist"
Sep 17 17:54:48.009017 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:48.008993    2574 scope.go:117] "RemoveContainer" containerID="43c3cac3623fbc91866ee1dd47033a1a9b79c29e3c73cd68172f2321a320f5fa"
Sep 17 17:54:48.009210 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:54:48.009196    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43c3cac3623fbc91866ee1dd47033a1a9b79c29e3c73cd68172f2321a320f5fa\": container with ID starting with 43c3cac3623fbc91866ee1dd47033a1a9b79c29e3c73cd68172f2321a320f5fa not found: ID does not exist" containerID="43c3cac3623fbc91866ee1dd47033a1a9b79c29e3c73cd68172f2321a320f5fa"
Sep 17 17:54:48.009257 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:48.009214    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43c3cac3623fbc91866ee1dd47033a1a9b79c29e3c73cd68172f2321a320f5fa"} err="failed to get container status \"43c3cac3623fbc91866ee1dd47033a1a9b79c29e3c73cd68172f2321a320f5fa\": rpc error: code = NotFound desc = could not find container \"43c3cac3623fbc91866ee1dd47033a1a9b79c29e3c73cd68172f2321a320f5fa\": container with ID starting with 43c3cac3623fbc91866ee1dd47033a1a9b79c29e3c73cd68172f2321a320f5fa not found: ID does not exist"
Sep 17 17:54:48.009257 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:48.009226    2574 scope.go:117] "RemoveContainer" containerID="0f0dd96a57fdbc3674ce589d2c98f3636d6fc8d15e3091a84aae5316d9bc03b4"
Sep 17 17:54:48.011877 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:54:48.011852    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f0dd96a57fdbc3674ce589d2c98f3636d6fc8d15e3091a84aae5316d9bc03b4\": container with ID starting with 0f0dd96a57fdbc3674ce589d2c98f3636d6fc8d15e3091a84aae5316d9bc03b4 not found: ID does not exist" containerID="0f0dd96a57fdbc3674ce589d2c98f3636d6fc8d15e3091a84aae5316d9bc03b4"
Sep 17 17:54:48.011958 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:48.011881    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f0dd96a57fdbc3674ce589d2c98f3636d6fc8d15e3091a84aae5316d9bc03b4"} err="failed to get container status \"0f0dd96a57fdbc3674ce589d2c98f3636d6fc8d15e3091a84aae5316d9bc03b4\": rpc error: code = NotFound desc = could not find container \"0f0dd96a57fdbc3674ce589d2c98f3636d6fc8d15e3091a84aae5316d9bc03b4\": container with ID starting with 0f0dd96a57fdbc3674ce589d2c98f3636d6fc8d15e3091a84aae5316d9bc03b4 not found: ID does not exist"
Sep 17 17:54:48.014581 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:48.014559    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"]
Sep 17 17:54:48.018992 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:48.018968    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-68381-predictor-7fd7ff945c-wrqmw"]
Sep 17 17:54:49.114629 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:54:49.114595    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdd78e43-e851-466a-8613-0bf5abfa8c8b" path="/var/lib/kubelet/pods/fdd78e43-e851-466a-8613-0bf5abfa8c8b/volumes"
Sep 17 17:55:03.475883 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:55:03.475849    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:03.582069 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:55:03.582018    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:13.970820 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:55:13.970786    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:14.090104 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:55:14.090069    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:19.015225 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:55:19.015190    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:19.091149 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:55:19.091115    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:13.275580 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.275492    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n"]
Sep 17 17:56:13.276515 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.275916    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e44bd846-8fa1-4527-92a2-d712bf30bcfa" containerName="storage-initializer"
Sep 17 17:56:13.276515 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.275934    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e44bd846-8fa1-4527-92a2-d712bf30bcfa" containerName="storage-initializer"
Sep 17 17:56:13.276515 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.275948    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e44bd846-8fa1-4527-92a2-d712bf30bcfa" containerName="kserve-container"
Sep 17 17:56:13.276515 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.275957    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e44bd846-8fa1-4527-92a2-d712bf30bcfa" containerName="kserve-container"
Sep 17 17:56:13.276515 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.275971    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd78e43-e851-466a-8613-0bf5abfa8c8b" containerName="storage-initializer"
Sep 17 17:56:13.276515 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.275980    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd78e43-e851-466a-8613-0bf5abfa8c8b" containerName="storage-initializer"
Sep 17 17:56:13.276515 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.275991    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd78e43-e851-466a-8613-0bf5abfa8c8b" containerName="kube-rbac-proxy"
Sep 17 17:56:13.276515 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.276002    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd78e43-e851-466a-8613-0bf5abfa8c8b" containerName="kube-rbac-proxy"
Sep 17 17:56:13.276515 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.276021    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd78e43-e851-466a-8613-0bf5abfa8c8b" containerName="kserve-container"
Sep 17 17:56:13.276515 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.276031    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd78e43-e851-466a-8613-0bf5abfa8c8b" containerName="kserve-container"
Sep 17 17:56:13.276515 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.276048    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87869ebb-dd0c-481e-93ff-c116602baf23" containerName="kserve-container"
Sep 17 17:56:13.276515 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.276082    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="87869ebb-dd0c-481e-93ff-c116602baf23" containerName="kserve-container"
Sep 17 17:56:13.276515 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.276103    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87869ebb-dd0c-481e-93ff-c116602baf23" containerName="storage-initializer"
Sep 17 17:56:13.276515 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.276112    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="87869ebb-dd0c-481e-93ff-c116602baf23" containerName="storage-initializer"
Sep 17 17:56:13.276515 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.276120    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e44bd846-8fa1-4527-92a2-d712bf30bcfa" containerName="kube-rbac-proxy"
Sep 17 17:56:13.276515 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.276128    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e44bd846-8fa1-4527-92a2-d712bf30bcfa" containerName="kube-rbac-proxy"
Sep 17 17:56:13.276515 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.276139    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87869ebb-dd0c-481e-93ff-c116602baf23" containerName="kube-rbac-proxy"
Sep 17 17:56:13.276515 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.276147    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="87869ebb-dd0c-481e-93ff-c116602baf23" containerName="kube-rbac-proxy"
Sep 17 17:56:13.276515 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.276220    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="87869ebb-dd0c-481e-93ff-c116602baf23" containerName="kserve-container"
Sep 17 17:56:13.276515 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.276234    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdd78e43-e851-466a-8613-0bf5abfa8c8b" containerName="kube-rbac-proxy"
Sep 17 17:56:13.276515 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.276246    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="87869ebb-dd0c-481e-93ff-c116602baf23" containerName="kube-rbac-proxy"
Sep 17 17:56:13.276515 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.276257    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e44bd846-8fa1-4527-92a2-d712bf30bcfa" containerName="kube-rbac-proxy"
Sep 17 17:56:13.276515 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.276267    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e44bd846-8fa1-4527-92a2-d712bf30bcfa" containerName="kserve-container"
Sep 17 17:56:13.276515 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.276275    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdd78e43-e851-466a-8613-0bf5abfa8c8b" containerName="kserve-container"
Sep 17 17:56:13.280004 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.279980    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n"
Sep 17 17:56:13.282860 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.282458    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:56:13.282860 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.282518    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-a7cd73-dockercfg-gh9ds\""
Sep 17 17:56:13.282860 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.282616    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-a7cd73\""
Sep 17 17:56:13.282860 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.282646    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-a7cd73-predictor-serving-cert\""
Sep 17 17:56:13.282860 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.282697    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:56:13.282860 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.282731    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-a7cd73-kube-rbac-proxy-sar-config\""
Sep 17 17:56:13.283426 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.283410    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:56:13.284635 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.284614    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:13.290911 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.290886    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n"]
Sep 17 17:56:13.345890 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.345866    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bcf18be6-1954-4a5e-8578-0a7551454bc5-cabundle-cert\") pod \"isvc-secondary-a7cd73-predictor-676bb47b74-db44n\" (UID: \"bcf18be6-1954-4a5e-8578-0a7551454bc5\") " pod="kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n"
Sep 17 17:56:13.346017 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.345913    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcf18be6-1954-4a5e-8578-0a7551454bc5-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a7cd73-predictor-676bb47b74-db44n\" (UID: \"bcf18be6-1954-4a5e-8578-0a7551454bc5\") " pod="kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n"
Sep 17 17:56:13.346017 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.345971    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcf18be6-1954-4a5e-8578-0a7551454bc5-kserve-provision-location\") pod \"isvc-secondary-a7cd73-predictor-676bb47b74-db44n\" (UID: \"bcf18be6-1954-4a5e-8578-0a7551454bc5\") " pod="kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n"
Sep 17 17:56:13.346017 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.346006    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmddh\" (UniqueName: \"kubernetes.io/projected/bcf18be6-1954-4a5e-8578-0a7551454bc5-kube-api-access-qmddh\") pod \"isvc-secondary-a7cd73-predictor-676bb47b74-db44n\" (UID: \"bcf18be6-1954-4a5e-8578-0a7551454bc5\") " pod="kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n"
Sep 17 17:56:13.346174 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.346036    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcf18be6-1954-4a5e-8578-0a7551454bc5-proxy-tls\") pod \"isvc-secondary-a7cd73-predictor-676bb47b74-db44n\" (UID: \"bcf18be6-1954-4a5e-8578-0a7551454bc5\") " pod="kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n"
Sep 17 17:56:13.422927 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.422896    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:13.446855 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.446832    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcf18be6-1954-4a5e-8578-0a7551454bc5-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a7cd73-predictor-676bb47b74-db44n\" (UID: \"bcf18be6-1954-4a5e-8578-0a7551454bc5\") " pod="kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n"
Sep 17 17:56:13.446959 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.446869    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcf18be6-1954-4a5e-8578-0a7551454bc5-kserve-provision-location\") pod \"isvc-secondary-a7cd73-predictor-676bb47b74-db44n\" (UID: \"bcf18be6-1954-4a5e-8578-0a7551454bc5\") " pod="kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n"
Sep 17 17:56:13.446959 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.446898    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmddh\" (UniqueName: \"kubernetes.io/projected/bcf18be6-1954-4a5e-8578-0a7551454bc5-kube-api-access-qmddh\") pod \"isvc-secondary-a7cd73-predictor-676bb47b74-db44n\" (UID: \"bcf18be6-1954-4a5e-8578-0a7551454bc5\") " pod="kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n"
Sep 17 17:56:13.446959 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.446928    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcf18be6-1954-4a5e-8578-0a7551454bc5-proxy-tls\") pod \"isvc-secondary-a7cd73-predictor-676bb47b74-db44n\" (UID: \"bcf18be6-1954-4a5e-8578-0a7551454bc5\") " pod="kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n"
Sep 17 17:56:13.446959 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.446955    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bcf18be6-1954-4a5e-8578-0a7551454bc5-cabundle-cert\") pod \"isvc-secondary-a7cd73-predictor-676bb47b74-db44n\" (UID: \"bcf18be6-1954-4a5e-8578-0a7551454bc5\") " pod="kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n"
Sep 17 17:56:13.447304 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.447283    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcf18be6-1954-4a5e-8578-0a7551454bc5-kserve-provision-location\") pod \"isvc-secondary-a7cd73-predictor-676bb47b74-db44n\" (UID: \"bcf18be6-1954-4a5e-8578-0a7551454bc5\") " pod="kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n"
Sep 17 17:56:13.447582 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.447558    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcf18be6-1954-4a5e-8578-0a7551454bc5-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a7cd73-predictor-676bb47b74-db44n\" (UID: \"bcf18be6-1954-4a5e-8578-0a7551454bc5\") " pod="kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n"
Sep 17 17:56:13.447685 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.447562    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bcf18be6-1954-4a5e-8578-0a7551454bc5-cabundle-cert\") pod \"isvc-secondary-a7cd73-predictor-676bb47b74-db44n\" (UID: \"bcf18be6-1954-4a5e-8578-0a7551454bc5\") " pod="kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n"
Sep 17 17:56:13.449353 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.449333    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcf18be6-1954-4a5e-8578-0a7551454bc5-proxy-tls\") pod \"isvc-secondary-a7cd73-predictor-676bb47b74-db44n\" (UID: \"bcf18be6-1954-4a5e-8578-0a7551454bc5\") " pod="kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n"
Sep 17 17:56:13.456110 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.456089    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmddh\" (UniqueName: \"kubernetes.io/projected/bcf18be6-1954-4a5e-8578-0a7551454bc5-kube-api-access-qmddh\") pod \"isvc-secondary-a7cd73-predictor-676bb47b74-db44n\" (UID: \"bcf18be6-1954-4a5e-8578-0a7551454bc5\") " pod="kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n"
Sep 17 17:56:13.592774 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.592736    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n"
Sep 17 17:56:13.865903 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:13.865833    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n"]
Sep 17 17:56:13.868702 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:56:13.868676    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf18be6_1954_4a5e_8578_0a7551454bc5.slice/crio-2478e65d8362c7756b2f875136ee1ae6fd09cbddfaf2e9fbeeaace0bb4a1d499 WatchSource:0}: Error finding container 2478e65d8362c7756b2f875136ee1ae6fd09cbddfaf2e9fbeeaace0bb4a1d499: Status 404 returned error can't find the container with id 2478e65d8362c7756b2f875136ee1ae6fd09cbddfaf2e9fbeeaace0bb4a1d499
Sep 17 17:56:14.321276 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:14.321192    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n" event={"ID":"bcf18be6-1954-4a5e-8578-0a7551454bc5","Type":"ContainerStarted","Data":"c78e9a99d8016294a9c866b6e11c77deb50c680f2df23f7c1849b2f20bfbe9ab"}
Sep 17 17:56:14.321276 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:14.321259    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n" event={"ID":"bcf18be6-1954-4a5e-8578-0a7551454bc5","Type":"ContainerStarted","Data":"2478e65d8362c7756b2f875136ee1ae6fd09cbddfaf2e9fbeeaace0bb4a1d499"}
Sep 17 17:56:19.340389 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:19.340355    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a7cd73-predictor-676bb47b74-db44n_bcf18be6-1954-4a5e-8578-0a7551454bc5/storage-initializer/0.log"
Sep 17 17:56:19.340763 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:19.340402    2574 generic.go:356] "Generic (PLEG): container finished" podID="bcf18be6-1954-4a5e-8578-0a7551454bc5" containerID="c78e9a99d8016294a9c866b6e11c77deb50c680f2df23f7c1849b2f20bfbe9ab" exitCode=1
Sep 17 17:56:19.340763 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:19.340479    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n" event={"ID":"bcf18be6-1954-4a5e-8578-0a7551454bc5","Type":"ContainerDied","Data":"c78e9a99d8016294a9c866b6e11c77deb50c680f2df23f7c1849b2f20bfbe9ab"}
Sep 17 17:56:20.346130 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:20.346104    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a7cd73-predictor-676bb47b74-db44n_bcf18be6-1954-4a5e-8578-0a7551454bc5/storage-initializer/0.log"
Sep 17 17:56:20.346495 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:20.346167    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n" event={"ID":"bcf18be6-1954-4a5e-8578-0a7551454bc5","Type":"ContainerStarted","Data":"6008e9a0b2cf047e6e94e73b2e5091f38b2616cfeb1432e405f4008fd3a30911"}
Sep 17 17:56:25.367728 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:25.367694    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a7cd73-predictor-676bb47b74-db44n_bcf18be6-1954-4a5e-8578-0a7551454bc5/storage-initializer/1.log"
Sep 17 17:56:25.368197 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:25.368073    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a7cd73-predictor-676bb47b74-db44n_bcf18be6-1954-4a5e-8578-0a7551454bc5/storage-initializer/0.log"
Sep 17 17:56:25.368197 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:25.368111    2574 generic.go:356] "Generic (PLEG): container finished" podID="bcf18be6-1954-4a5e-8578-0a7551454bc5" containerID="6008e9a0b2cf047e6e94e73b2e5091f38b2616cfeb1432e405f4008fd3a30911" exitCode=1
Sep 17 17:56:25.368197 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:25.368143    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n" event={"ID":"bcf18be6-1954-4a5e-8578-0a7551454bc5","Type":"ContainerDied","Data":"6008e9a0b2cf047e6e94e73b2e5091f38b2616cfeb1432e405f4008fd3a30911"}
Sep 17 17:56:25.368317 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:25.368204    2574 scope.go:117] "RemoveContainer" containerID="c78e9a99d8016294a9c866b6e11c77deb50c680f2df23f7c1849b2f20bfbe9ab"
Sep 17 17:56:25.368588 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:25.368567    2574 scope.go:117] "RemoveContainer" containerID="c78e9a99d8016294a9c866b6e11c77deb50c680f2df23f7c1849b2f20bfbe9ab"
Sep 17 17:56:25.379582 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:56:25.379560    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-a7cd73-predictor-676bb47b74-db44n_kserve-ci-e2e-test(bcf18be6-1954-4a5e-8578-0a7551454bc5)\"" pod="kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n" podUID="bcf18be6-1954-4a5e-8578-0a7551454bc5"
Sep 17 17:56:25.379879 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:56:25.379855    2574 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-a7cd73-predictor-676bb47b74-db44n_kserve-ci-e2e-test_bcf18be6-1954-4a5e-8578-0a7551454bc5_0 in pod sandbox 2478e65d8362c7756b2f875136ee1ae6fd09cbddfaf2e9fbeeaace0bb4a1d499 from index: no such id: 'c78e9a99d8016294a9c866b6e11c77deb50c680f2df23f7c1849b2f20bfbe9ab'" containerID="c78e9a99d8016294a9c866b6e11c77deb50c680f2df23f7c1849b2f20bfbe9ab"
Sep 17 17:56:25.379949 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:25.379886    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c78e9a99d8016294a9c866b6e11c77deb50c680f2df23f7c1849b2f20bfbe9ab"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-a7cd73-predictor-676bb47b74-db44n_kserve-ci-e2e-test_bcf18be6-1954-4a5e-8578-0a7551454bc5_0 in pod sandbox 2478e65d8362c7756b2f875136ee1ae6fd09cbddfaf2e9fbeeaace0bb4a1d499 from index: no such id: 'c78e9a99d8016294a9c866b6e11c77deb50c680f2df23f7c1849b2f20bfbe9ab'"
Sep 17 17:56:26.373981 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:26.373954    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a7cd73-predictor-676bb47b74-db44n_bcf18be6-1954-4a5e-8578-0a7551454bc5/storage-initializer/1.log"
Sep 17 17:56:31.379609 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:31.379571    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n"]
Sep 17 17:56:31.427859 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:31.427826    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:31.514804 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:31.514782    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a7cd73-predictor-676bb47b74-db44n_bcf18be6-1954-4a5e-8578-0a7551454bc5/storage-initializer/1.log"
Sep 17 17:56:31.518455 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:31.518438    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n"
Sep 17 17:56:31.537414 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:31.537390    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:31.590990 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:31.590970    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bcf18be6-1954-4a5e-8578-0a7551454bc5-cabundle-cert\") pod \"bcf18be6-1954-4a5e-8578-0a7551454bc5\" (UID: \"bcf18be6-1954-4a5e-8578-0a7551454bc5\") "
Sep 17 17:56:31.591133 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:31.591031    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcf18be6-1954-4a5e-8578-0a7551454bc5-kube-rbac-proxy-sar-config\") pod \"bcf18be6-1954-4a5e-8578-0a7551454bc5\" (UID: \"bcf18be6-1954-4a5e-8578-0a7551454bc5\") "
Sep 17 17:56:31.591133 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:31.591080    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcf18be6-1954-4a5e-8578-0a7551454bc5-proxy-tls\") pod \"bcf18be6-1954-4a5e-8578-0a7551454bc5\" (UID: \"bcf18be6-1954-4a5e-8578-0a7551454bc5\") "
Sep 17 17:56:31.591133 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:31.591107    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmddh\" (UniqueName: \"kubernetes.io/projected/bcf18be6-1954-4a5e-8578-0a7551454bc5-kube-api-access-qmddh\") pod \"bcf18be6-1954-4a5e-8578-0a7551454bc5\" (UID: \"bcf18be6-1954-4a5e-8578-0a7551454bc5\") "
Sep 17 17:56:31.591289 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:31.591170    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcf18be6-1954-4a5e-8578-0a7551454bc5-kserve-provision-location\") pod \"bcf18be6-1954-4a5e-8578-0a7551454bc5\" (UID: \"bcf18be6-1954-4a5e-8578-0a7551454bc5\") "
Sep 17 17:56:31.591359 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:31.591335    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf18be6-1954-4a5e-8578-0a7551454bc5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bcf18be6-1954-4a5e-8578-0a7551454bc5" (UID: "bcf18be6-1954-4a5e-8578-0a7551454bc5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:56:31.591508 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:31.591484    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf18be6-1954-4a5e-8578-0a7551454bc5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bcf18be6-1954-4a5e-8578-0a7551454bc5" (UID: "bcf18be6-1954-4a5e-8578-0a7551454bc5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:31.591562 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:31.591491    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf18be6-1954-4a5e-8578-0a7551454bc5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bcf18be6-1954-4a5e-8578-0a7551454bc5" (UID: "bcf18be6-1954-4a5e-8578-0a7551454bc5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:56:31.593097 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:31.593065    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf18be6-1954-4a5e-8578-0a7551454bc5-kube-api-access-qmddh" (OuterVolumeSpecName: "kube-api-access-qmddh") pod "bcf18be6-1954-4a5e-8578-0a7551454bc5" (UID: "bcf18be6-1954-4a5e-8578-0a7551454bc5"). InnerVolumeSpecName "kube-api-access-qmddh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:56:31.593176 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:31.593118    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcf18be6-1954-4a5e-8578-0a7551454bc5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bcf18be6-1954-4a5e-8578-0a7551454bc5" (UID: "bcf18be6-1954-4a5e-8578-0a7551454bc5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:56:31.692182 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:31.692125    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcf18be6-1954-4a5e-8578-0a7551454bc5-kserve-provision-location\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:56:31.692182 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:31.692149    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bcf18be6-1954-4a5e-8578-0a7551454bc5-cabundle-cert\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:56:31.692182 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:31.692163    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcf18be6-1954-4a5e-8578-0a7551454bc5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:56:31.692182 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:31.692176    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcf18be6-1954-4a5e-8578-0a7551454bc5-proxy-tls\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:56:31.692352 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:31.692189    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmddh\" (UniqueName: \"kubernetes.io/projected/bcf18be6-1954-4a5e-8578-0a7551454bc5-kube-api-access-qmddh\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:56:32.399135 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:32.399106    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a7cd73-predictor-676bb47b74-db44n_bcf18be6-1954-4a5e-8578-0a7551454bc5/storage-initializer/1.log"
Sep 17 17:56:32.399569 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:32.399160    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n" event={"ID":"bcf18be6-1954-4a5e-8578-0a7551454bc5","Type":"ContainerDied","Data":"2478e65d8362c7756b2f875136ee1ae6fd09cbddfaf2e9fbeeaace0bb4a1d499"}
Sep 17 17:56:32.399569 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:32.399199    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n"
Sep 17 17:56:32.399569 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:32.399228    2574 scope.go:117] "RemoveContainer" containerID="6008e9a0b2cf047e6e94e73b2e5091f38b2616cfeb1432e405f4008fd3a30911"
Sep 17 17:56:32.440524 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:32.440499    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n"]
Sep 17 17:56:32.443445 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:32.443422    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a7cd73-predictor-676bb47b74-db44n"]
Sep 17 17:56:33.114743 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:33.114713    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcf18be6-1954-4a5e-8578-0a7551454bc5" path="/var/lib/kubelet/pods/bcf18be6-1954-4a5e-8578-0a7551454bc5/volumes"
Sep 17 17:56:36.670459 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.670423    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:36.681542 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.681513    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j"]
Sep 17 17:56:36.681878 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.681861    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcf18be6-1954-4a5e-8578-0a7551454bc5" containerName="storage-initializer"
Sep 17 17:56:36.681973 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.681881    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcf18be6-1954-4a5e-8578-0a7551454bc5" containerName="storage-initializer"
Sep 17 17:56:36.682031 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.681989    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcf18be6-1954-4a5e-8578-0a7551454bc5" containerName="storage-initializer"
Sep 17 17:56:36.682111 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.682099    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcf18be6-1954-4a5e-8578-0a7551454bc5" containerName="storage-initializer"
Sep 17 17:56:36.682167 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.682114    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcf18be6-1954-4a5e-8578-0a7551454bc5" containerName="storage-initializer"
Sep 17 17:56:36.682215 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.682185    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcf18be6-1954-4a5e-8578-0a7551454bc5" containerName="storage-initializer"
Sep 17 17:56:36.686448 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.686428    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j"
Sep 17 17:56:36.688939 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.688917    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-da0909-predictor-serving-cert\""
Sep 17 17:56:36.689047 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.688917    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:56:36.689047 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.688918    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:56:36.689047 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.689010    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-da0909\""
Sep 17 17:56:36.689229 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.689157    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:56:36.689855 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.689837    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-da0909-kube-rbac-proxy-sar-config\""
Sep 17 17:56:36.690005 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.689868    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-da0909-dockercfg-8wr9g\""
Sep 17 17:56:36.699698 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.699679    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j"]
Sep 17 17:56:36.830757 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.830720    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79b6b81a-44b3-416b-bf7c-d671cb414f15-cabundle-cert\") pod \"isvc-init-fail-da0909-predictor-f56d96658-7q88j\" (UID: \"79b6b81a-44b3-416b-bf7c-d671cb414f15\") " pod="kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j"
Sep 17 17:56:36.830884 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.830777    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79b6b81a-44b3-416b-bf7c-d671cb414f15-proxy-tls\") pod \"isvc-init-fail-da0909-predictor-f56d96658-7q88j\" (UID: \"79b6b81a-44b3-416b-bf7c-d671cb414f15\") " pod="kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j"
Sep 17 17:56:36.830884 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.830842    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79b6b81a-44b3-416b-bf7c-d671cb414f15-kserve-provision-location\") pod \"isvc-init-fail-da0909-predictor-f56d96658-7q88j\" (UID: \"79b6b81a-44b3-416b-bf7c-d671cb414f15\") " pod="kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j"
Sep 17 17:56:36.830965 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.830905    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79b6b81a-44b3-416b-bf7c-d671cb414f15-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-da0909-predictor-f56d96658-7q88j\" (UID: \"79b6b81a-44b3-416b-bf7c-d671cb414f15\") " pod="kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j"
Sep 17 17:56:36.830965 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.830928    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs6k7\" (UniqueName: \"kubernetes.io/projected/79b6b81a-44b3-416b-bf7c-d671cb414f15-kube-api-access-fs6k7\") pod \"isvc-init-fail-da0909-predictor-f56d96658-7q88j\" (UID: \"79b6b81a-44b3-416b-bf7c-d671cb414f15\") " pod="kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j"
Sep 17 17:56:36.931631 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.931557    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79b6b81a-44b3-416b-bf7c-d671cb414f15-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-da0909-predictor-f56d96658-7q88j\" (UID: \"79b6b81a-44b3-416b-bf7c-d671cb414f15\") " pod="kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j"
Sep 17 17:56:36.931631 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.931598    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs6k7\" (UniqueName: \"kubernetes.io/projected/79b6b81a-44b3-416b-bf7c-d671cb414f15-kube-api-access-fs6k7\") pod \"isvc-init-fail-da0909-predictor-f56d96658-7q88j\" (UID: \"79b6b81a-44b3-416b-bf7c-d671cb414f15\") " pod="kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j"
Sep 17 17:56:36.931813 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.931654    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79b6b81a-44b3-416b-bf7c-d671cb414f15-cabundle-cert\") pod \"isvc-init-fail-da0909-predictor-f56d96658-7q88j\" (UID: \"79b6b81a-44b3-416b-bf7c-d671cb414f15\") " pod="kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j"
Sep 17 17:56:36.931813 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.931701    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79b6b81a-44b3-416b-bf7c-d671cb414f15-proxy-tls\") pod \"isvc-init-fail-da0909-predictor-f56d96658-7q88j\" (UID: \"79b6b81a-44b3-416b-bf7c-d671cb414f15\") " pod="kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j"
Sep 17 17:56:36.931813 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.931745    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79b6b81a-44b3-416b-bf7c-d671cb414f15-kserve-provision-location\") pod \"isvc-init-fail-da0909-predictor-f56d96658-7q88j\" (UID: \"79b6b81a-44b3-416b-bf7c-d671cb414f15\") " pod="kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j"
Sep 17 17:56:36.932104 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.932086    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79b6b81a-44b3-416b-bf7c-d671cb414f15-kserve-provision-location\") pod \"isvc-init-fail-da0909-predictor-f56d96658-7q88j\" (UID: \"79b6b81a-44b3-416b-bf7c-d671cb414f15\") " pod="kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j"
Sep 17 17:56:36.932264 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.932242    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79b6b81a-44b3-416b-bf7c-d671cb414f15-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-da0909-predictor-f56d96658-7q88j\" (UID: \"79b6b81a-44b3-416b-bf7c-d671cb414f15\") " pod="kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j"
Sep 17 17:56:36.932351 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.932333    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79b6b81a-44b3-416b-bf7c-d671cb414f15-cabundle-cert\") pod \"isvc-init-fail-da0909-predictor-f56d96658-7q88j\" (UID: \"79b6b81a-44b3-416b-bf7c-d671cb414f15\") " pod="kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j"
Sep 17 17:56:36.934181 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.934159    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79b6b81a-44b3-416b-bf7c-d671cb414f15-proxy-tls\") pod \"isvc-init-fail-da0909-predictor-f56d96658-7q88j\" (UID: \"79b6b81a-44b3-416b-bf7c-d671cb414f15\") " pod="kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j"
Sep 17 17:56:36.942382 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.942361    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs6k7\" (UniqueName: \"kubernetes.io/projected/79b6b81a-44b3-416b-bf7c-d671cb414f15-kube-api-access-fs6k7\") pod \"isvc-init-fail-da0909-predictor-f56d96658-7q88j\" (UID: \"79b6b81a-44b3-416b-bf7c-d671cb414f15\") " pod="kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j"
Sep 17 17:56:36.955545 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.955522    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:36.996185 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:36.996167    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j"
Sep 17 17:56:37.278640 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:37.278560    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j"]
Sep 17 17:56:37.281494 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:56:37.281464    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b6b81a_44b3_416b_bf7c_d671cb414f15.slice/crio-f0475ec070f6eff98b28ad65d64206852c01a66e5f7ba3e58b6a060c2207261d WatchSource:0}: Error finding container f0475ec070f6eff98b28ad65d64206852c01a66e5f7ba3e58b6a060c2207261d: Status 404 returned error can't find the container with id f0475ec070f6eff98b28ad65d64206852c01a66e5f7ba3e58b6a060c2207261d
Sep 17 17:56:37.420645 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:37.420610    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j" event={"ID":"79b6b81a-44b3-416b-bf7c-d671cb414f15","Type":"ContainerStarted","Data":"effae981bfad7dec04fde400e4ee4d2d5c6826f8f526cd4bab9c333bc6027d95"}
Sep 17 17:56:37.420826 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:37.420671    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j" event={"ID":"79b6b81a-44b3-416b-bf7c-d671cb414f15","Type":"ContainerStarted","Data":"f0475ec070f6eff98b28ad65d64206852c01a66e5f7ba3e58b6a060c2207261d"}
Sep 17 17:56:38.847486 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:38.847442    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:39.035886 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:39.035848    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:39.265999 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:39.265915    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:40.435987 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:40.435955    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-da0909-predictor-f56d96658-7q88j_79b6b81a-44b3-416b-bf7c-d671cb414f15/storage-initializer/0.log"
Sep 17 17:56:40.436394 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:40.435996    2574 generic.go:356] "Generic (PLEG): container finished" podID="79b6b81a-44b3-416b-bf7c-d671cb414f15" containerID="effae981bfad7dec04fde400e4ee4d2d5c6826f8f526cd4bab9c333bc6027d95" exitCode=1
Sep 17 17:56:40.436394 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:40.436079    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j" event={"ID":"79b6b81a-44b3-416b-bf7c-d671cb414f15","Type":"ContainerDied","Data":"effae981bfad7dec04fde400e4ee4d2d5c6826f8f526cd4bab9c333bc6027d95"}
Sep 17 17:56:41.441036 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:41.441010    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-da0909-predictor-f56d96658-7q88j_79b6b81a-44b3-416b-bf7c-d671cb414f15/storage-initializer/0.log"
Sep 17 17:56:41.441392 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:41.441071    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j" event={"ID":"79b6b81a-44b3-416b-bf7c-d671cb414f15","Type":"ContainerStarted","Data":"545b0a4b81d4f37dc2e11295c1176bfd05d15f7769cb3ca07d401905e2600d8a"}
Sep 17 17:56:41.617913 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:41.617880    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j"]
Sep 17 17:56:41.678721 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:41.678690    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:42.444989 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:42.444927    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j" podUID="79b6b81a-44b3-416b-bf7c-d671cb414f15" containerName="storage-initializer" containerID="cri-o://545b0a4b81d4f37dc2e11295c1176bfd05d15f7769cb3ca07d401905e2600d8a" gracePeriod=30
Sep 17 17:56:44.978694 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:44.978673    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-da0909-predictor-f56d96658-7q88j_79b6b81a-44b3-416b-bf7c-d671cb414f15/storage-initializer/1.log"
Sep 17 17:56:44.979086 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:44.979072    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-da0909-predictor-f56d96658-7q88j_79b6b81a-44b3-416b-bf7c-d671cb414f15/storage-initializer/0.log"
Sep 17 17:56:44.981679 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:44.981663    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j"
Sep 17 17:56:45.102924 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:45.102901    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79b6b81a-44b3-416b-bf7c-d671cb414f15-proxy-tls\") pod \"79b6b81a-44b3-416b-bf7c-d671cb414f15\" (UID: \"79b6b81a-44b3-416b-bf7c-d671cb414f15\") "
Sep 17 17:56:45.103050 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:45.102934    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fs6k7\" (UniqueName: \"kubernetes.io/projected/79b6b81a-44b3-416b-bf7c-d671cb414f15-kube-api-access-fs6k7\") pod \"79b6b81a-44b3-416b-bf7c-d671cb414f15\" (UID: \"79b6b81a-44b3-416b-bf7c-d671cb414f15\") "
Sep 17 17:56:45.103050 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:45.102968    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79b6b81a-44b3-416b-bf7c-d671cb414f15-cabundle-cert\") pod \"79b6b81a-44b3-416b-bf7c-d671cb414f15\" (UID: \"79b6b81a-44b3-416b-bf7c-d671cb414f15\") "
Sep 17 17:56:45.103050 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:45.103023    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79b6b81a-44b3-416b-bf7c-d671cb414f15-kube-rbac-proxy-sar-config\") pod \"79b6b81a-44b3-416b-bf7c-d671cb414f15\" (UID: \"79b6b81a-44b3-416b-bf7c-d671cb414f15\") "
Sep 17 17:56:45.103259 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:45.103087    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79b6b81a-44b3-416b-bf7c-d671cb414f15-kserve-provision-location\") pod \"79b6b81a-44b3-416b-bf7c-d671cb414f15\" (UID: \"79b6b81a-44b3-416b-bf7c-d671cb414f15\") "
Sep 17 17:56:45.103384 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:45.103345    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b6b81a-44b3-416b-bf7c-d671cb414f15-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "79b6b81a-44b3-416b-bf7c-d671cb414f15" (UID: "79b6b81a-44b3-416b-bf7c-d671cb414f15"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:45.103384 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:45.103370    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b6b81a-44b3-416b-bf7c-d671cb414f15-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "79b6b81a-44b3-416b-bf7c-d671cb414f15" (UID: "79b6b81a-44b3-416b-bf7c-d671cb414f15"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:56:45.103489 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:45.103413    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b6b81a-44b3-416b-bf7c-d671cb414f15-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "79b6b81a-44b3-416b-bf7c-d671cb414f15" (UID: "79b6b81a-44b3-416b-bf7c-d671cb414f15"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:56:45.104971 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:45.104952    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b6b81a-44b3-416b-bf7c-d671cb414f15-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "79b6b81a-44b3-416b-bf7c-d671cb414f15" (UID: "79b6b81a-44b3-416b-bf7c-d671cb414f15"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:56:45.105046 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:45.105023    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b6b81a-44b3-416b-bf7c-d671cb414f15-kube-api-access-fs6k7" (OuterVolumeSpecName: "kube-api-access-fs6k7") pod "79b6b81a-44b3-416b-bf7c-d671cb414f15" (UID: "79b6b81a-44b3-416b-bf7c-d671cb414f15"). InnerVolumeSpecName "kube-api-access-fs6k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:56:45.203997 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:45.203966    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79b6b81a-44b3-416b-bf7c-d671cb414f15-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:56:45.203997 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:45.203994    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79b6b81a-44b3-416b-bf7c-d671cb414f15-kserve-provision-location\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:56:45.204186 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:45.204004    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79b6b81a-44b3-416b-bf7c-d671cb414f15-proxy-tls\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:56:45.204186 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:45.204012    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs6k7\" (UniqueName: \"kubernetes.io/projected/79b6b81a-44b3-416b-bf7c-d671cb414f15-kube-api-access-fs6k7\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:56:45.204186 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:45.204021    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79b6b81a-44b3-416b-bf7c-d671cb414f15-cabundle-cert\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:56:45.457371 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:45.457294    2574 generic.go:356] "Generic (PLEG): container finished" podID="79b6b81a-44b3-416b-bf7c-d671cb414f15" containerID="545b0a4b81d4f37dc2e11295c1176bfd05d15f7769cb3ca07d401905e2600d8a" exitCode=1
Sep 17 17:56:45.457371 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:45.457351    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j" event={"ID":"79b6b81a-44b3-416b-bf7c-d671cb414f15","Type":"ContainerDied","Data":"545b0a4b81d4f37dc2e11295c1176bfd05d15f7769cb3ca07d401905e2600d8a"}
Sep 17 17:56:45.457547 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:45.457370    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j"
Sep 17 17:56:45.457547 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:45.457402    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j" event={"ID":"79b6b81a-44b3-416b-bf7c-d671cb414f15","Type":"ContainerDied","Data":"f0475ec070f6eff98b28ad65d64206852c01a66e5f7ba3e58b6a060c2207261d"}
Sep 17 17:56:45.457547 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:45.457432    2574 scope.go:117] "RemoveContainer" containerID="545b0a4b81d4f37dc2e11295c1176bfd05d15f7769cb3ca07d401905e2600d8a"
Sep 17 17:56:45.465289 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:45.465263    2574 scope.go:117] "RemoveContainer" containerID="effae981bfad7dec04fde400e4ee4d2d5c6826f8f526cd4bab9c333bc6027d95"
Sep 17 17:56:45.472030 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:45.472002    2574 scope.go:117] "RemoveContainer" containerID="545b0a4b81d4f37dc2e11295c1176bfd05d15f7769cb3ca07d401905e2600d8a"
Sep 17 17:56:45.472300 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:56:45.472283    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"545b0a4b81d4f37dc2e11295c1176bfd05d15f7769cb3ca07d401905e2600d8a\": container with ID starting with 545b0a4b81d4f37dc2e11295c1176bfd05d15f7769cb3ca07d401905e2600d8a not found: ID does not exist" containerID="545b0a4b81d4f37dc2e11295c1176bfd05d15f7769cb3ca07d401905e2600d8a"
Sep 17 17:56:45.472362 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:45.472306    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"545b0a4b81d4f37dc2e11295c1176bfd05d15f7769cb3ca07d401905e2600d8a"} err="failed to get container status \"545b0a4b81d4f37dc2e11295c1176bfd05d15f7769cb3ca07d401905e2600d8a\": rpc error: code = NotFound desc = could not find container \"545b0a4b81d4f37dc2e11295c1176bfd05d15f7769cb3ca07d401905e2600d8a\": container with ID starting with 545b0a4b81d4f37dc2e11295c1176bfd05d15f7769cb3ca07d401905e2600d8a not found: ID does not exist"
Sep 17 17:56:45.472362 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:45.472322    2574 scope.go:117] "RemoveContainer" containerID="effae981bfad7dec04fde400e4ee4d2d5c6826f8f526cd4bab9c333bc6027d95"
Sep 17 17:56:45.472577 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:56:45.472558    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"effae981bfad7dec04fde400e4ee4d2d5c6826f8f526cd4bab9c333bc6027d95\": container with ID starting with effae981bfad7dec04fde400e4ee4d2d5c6826f8f526cd4bab9c333bc6027d95 not found: ID does not exist" containerID="effae981bfad7dec04fde400e4ee4d2d5c6826f8f526cd4bab9c333bc6027d95"
Sep 17 17:56:45.472630 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:45.472586    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"effae981bfad7dec04fde400e4ee4d2d5c6826f8f526cd4bab9c333bc6027d95"} err="failed to get container status \"effae981bfad7dec04fde400e4ee4d2d5c6826f8f526cd4bab9c333bc6027d95\": rpc error: code = NotFound desc = could not find container \"effae981bfad7dec04fde400e4ee4d2d5c6826f8f526cd4bab9c333bc6027d95\": container with ID starting with effae981bfad7dec04fde400e4ee4d2d5c6826f8f526cd4bab9c333bc6027d95 not found: ID does not exist"
Sep 17 17:56:45.517253 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:45.517226    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j"]
Sep 17 17:56:45.519663 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:45.519641    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-da0909-predictor-f56d96658-7q88j"]
Sep 17 17:56:46.829326 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:46.829291    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"]
Sep 17 17:56:46.829700 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:46.829640    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79b6b81a-44b3-416b-bf7c-d671cb414f15" containerName="storage-initializer"
Sep 17 17:56:46.829700 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:46.829654    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="79b6b81a-44b3-416b-bf7c-d671cb414f15" containerName="storage-initializer"
Sep 17 17:56:46.829700 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:46.829667    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79b6b81a-44b3-416b-bf7c-d671cb414f15" containerName="storage-initializer"
Sep 17 17:56:46.829700 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:46.829673    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="79b6b81a-44b3-416b-bf7c-d671cb414f15" containerName="storage-initializer"
Sep 17 17:56:46.829892 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:46.829726    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="79b6b81a-44b3-416b-bf7c-d671cb414f15" containerName="storage-initializer"
Sep 17 17:56:46.829892 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:46.829740    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="79b6b81a-44b3-416b-bf7c-d671cb414f15" containerName="storage-initializer"
Sep 17 17:56:46.832766 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:46.832750    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"
Sep 17 17:56:46.835317 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:46.835290    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-103f5-predictor-serving-cert\""
Sep 17 17:56:46.835444 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:46.835307    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:56:46.835444 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:46.835313    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:56:46.835444 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:46.835321    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-103f5-kube-rbac-proxy-sar-config\""
Sep 17 17:56:46.835605 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:46.835333    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:56:46.835605 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:46.835360    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:56:46.836227 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:46.836211    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-q5zmc\""
Sep 17 17:56:46.842482 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:46.842462    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"]
Sep 17 17:56:46.906349 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:46.906322    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:46.915426 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:46.915402    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1faa7d9b-70ec-4e07-b774-8427593f98b1-proxy-tls\") pod \"raw-sklearn-103f5-predictor-6594b785b8-857v5\" (UID: \"1faa7d9b-70ec-4e07-b774-8427593f98b1\") " pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"
Sep 17 17:56:46.915508 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:46.915433    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ttdt\" (UniqueName: \"kubernetes.io/projected/1faa7d9b-70ec-4e07-b774-8427593f98b1-kube-api-access-2ttdt\") pod \"raw-sklearn-103f5-predictor-6594b785b8-857v5\" (UID: \"1faa7d9b-70ec-4e07-b774-8427593f98b1\") " pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"
Sep 17 17:56:46.915508 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:46.915461    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1faa7d9b-70ec-4e07-b774-8427593f98b1-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-103f5-predictor-6594b785b8-857v5\" (UID: \"1faa7d9b-70ec-4e07-b774-8427593f98b1\") " pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"
Sep 17 17:56:46.915582 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:46.915517    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1faa7d9b-70ec-4e07-b774-8427593f98b1-kserve-provision-location\") pod \"raw-sklearn-103f5-predictor-6594b785b8-857v5\" (UID: \"1faa7d9b-70ec-4e07-b774-8427593f98b1\") " pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"
Sep 17 17:56:46.915582 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:46.915555    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1faa7d9b-70ec-4e07-b774-8427593f98b1-cabundle-cert\") pod \"raw-sklearn-103f5-predictor-6594b785b8-857v5\" (UID: \"1faa7d9b-70ec-4e07-b774-8427593f98b1\") " pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"
Sep 17 17:56:47.016721 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:47.016691    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1faa7d9b-70ec-4e07-b774-8427593f98b1-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-103f5-predictor-6594b785b8-857v5\" (UID: \"1faa7d9b-70ec-4e07-b774-8427593f98b1\") " pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"
Sep 17 17:56:47.016862 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:47.016732    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1faa7d9b-70ec-4e07-b774-8427593f98b1-kserve-provision-location\") pod \"raw-sklearn-103f5-predictor-6594b785b8-857v5\" (UID: \"1faa7d9b-70ec-4e07-b774-8427593f98b1\") " pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"
Sep 17 17:56:47.016862 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:47.016769    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1faa7d9b-70ec-4e07-b774-8427593f98b1-cabundle-cert\") pod \"raw-sklearn-103f5-predictor-6594b785b8-857v5\" (UID: \"1faa7d9b-70ec-4e07-b774-8427593f98b1\") " pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"
Sep 17 17:56:47.016862 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:47.016799    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1faa7d9b-70ec-4e07-b774-8427593f98b1-proxy-tls\") pod \"raw-sklearn-103f5-predictor-6594b785b8-857v5\" (UID: \"1faa7d9b-70ec-4e07-b774-8427593f98b1\") " pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"
Sep 17 17:56:47.016862 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:47.016818    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ttdt\" (UniqueName: \"kubernetes.io/projected/1faa7d9b-70ec-4e07-b774-8427593f98b1-kube-api-access-2ttdt\") pod \"raw-sklearn-103f5-predictor-6594b785b8-857v5\" (UID: \"1faa7d9b-70ec-4e07-b774-8427593f98b1\") " pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"
Sep 17 17:56:47.017116 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:56:47.016936    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-103f5-predictor-serving-cert: secret "raw-sklearn-103f5-predictor-serving-cert" not found
Sep 17 17:56:47.017116 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:56:47.017026    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1faa7d9b-70ec-4e07-b774-8427593f98b1-proxy-tls podName:1faa7d9b-70ec-4e07-b774-8427593f98b1 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:47.516991462 +0000 UTC m=+1382.846091074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1faa7d9b-70ec-4e07-b774-8427593f98b1-proxy-tls") pod "raw-sklearn-103f5-predictor-6594b785b8-857v5" (UID: "1faa7d9b-70ec-4e07-b774-8427593f98b1") : secret "raw-sklearn-103f5-predictor-serving-cert" not found
Sep 17 17:56:47.017240 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:47.017147    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1faa7d9b-70ec-4e07-b774-8427593f98b1-kserve-provision-location\") pod \"raw-sklearn-103f5-predictor-6594b785b8-857v5\" (UID: \"1faa7d9b-70ec-4e07-b774-8427593f98b1\") " pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"
Sep 17 17:56:47.017417 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:47.017395    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1faa7d9b-70ec-4e07-b774-8427593f98b1-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-103f5-predictor-6594b785b8-857v5\" (UID: \"1faa7d9b-70ec-4e07-b774-8427593f98b1\") " pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"
Sep 17 17:56:47.017534 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:47.017514    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1faa7d9b-70ec-4e07-b774-8427593f98b1-cabundle-cert\") pod \"raw-sklearn-103f5-predictor-6594b785b8-857v5\" (UID: \"1faa7d9b-70ec-4e07-b774-8427593f98b1\") " pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"
Sep 17 17:56:47.033282 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:47.033258    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ttdt\" (UniqueName: \"kubernetes.io/projected/1faa7d9b-70ec-4e07-b774-8427593f98b1-kube-api-access-2ttdt\") pod \"raw-sklearn-103f5-predictor-6594b785b8-857v5\" (UID: \"1faa7d9b-70ec-4e07-b774-8427593f98b1\") " pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"
Sep 17 17:56:47.116371 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:47.116348    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79b6b81a-44b3-416b-bf7c-d671cb414f15" path="/var/lib/kubelet/pods/79b6b81a-44b3-416b-bf7c-d671cb414f15/volumes"
Sep 17 17:56:47.294364 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:47.294326    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:47.521490 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:47.521401    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1faa7d9b-70ec-4e07-b774-8427593f98b1-proxy-tls\") pod \"raw-sklearn-103f5-predictor-6594b785b8-857v5\" (UID: \"1faa7d9b-70ec-4e07-b774-8427593f98b1\") " pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"
Sep 17 17:56:47.523745 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:47.523720    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1faa7d9b-70ec-4e07-b774-8427593f98b1-proxy-tls\") pod \"raw-sklearn-103f5-predictor-6594b785b8-857v5\" (UID: \"1faa7d9b-70ec-4e07-b774-8427593f98b1\") " pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"
Sep 17 17:56:47.743398 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:47.743363    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"
Sep 17 17:56:48.017089 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:48.017045    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"]
Sep 17 17:56:48.019196 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:56:48.019163    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1faa7d9b_70ec_4e07_b774_8427593f98b1.slice/crio-04e694d36855ee2d388da46c539de1b28f134334d32eabd3cd47216a21f3b756 WatchSource:0}: Error finding container 04e694d36855ee2d388da46c539de1b28f134334d32eabd3cd47216a21f3b756: Status 404 returned error can't find the container with id 04e694d36855ee2d388da46c539de1b28f134334d32eabd3cd47216a21f3b756
Sep 17 17:56:48.477217 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:48.477167    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5" event={"ID":"1faa7d9b-70ec-4e07-b774-8427593f98b1","Type":"ContainerStarted","Data":"013f6af8eac6749198c091d972cfeb0e77c29a88a9cab33a6b929b1ee56f0991"}
Sep 17 17:56:48.477435 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:48.477238    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5" event={"ID":"1faa7d9b-70ec-4e07-b774-8427593f98b1","Type":"ContainerStarted","Data":"04e694d36855ee2d388da46c539de1b28f134334d32eabd3cd47216a21f3b756"}
Sep 17 17:56:52.494332 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:52.494299    2574 generic.go:356] "Generic (PLEG): container finished" podID="1faa7d9b-70ec-4e07-b774-8427593f98b1" containerID="013f6af8eac6749198c091d972cfeb0e77c29a88a9cab33a6b929b1ee56f0991" exitCode=0
Sep 17 17:56:52.494733 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:52.494376    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5" event={"ID":"1faa7d9b-70ec-4e07-b774-8427593f98b1","Type":"ContainerDied","Data":"013f6af8eac6749198c091d972cfeb0e77c29a88a9cab33a6b929b1ee56f0991"}
Sep 17 17:56:53.501071 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:53.501032    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5" event={"ID":"1faa7d9b-70ec-4e07-b774-8427593f98b1","Type":"ContainerStarted","Data":"42b9bdba438a3d466007ec6981cdd19723f8065ea9a4e7dfc715f56e4ac3dd43"}
Sep 17 17:56:53.501556 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:53.501127    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5" event={"ID":"1faa7d9b-70ec-4e07-b774-8427593f98b1","Type":"ContainerStarted","Data":"a065a171d3708e8f6021e99d2fc5437ac3eea1185567873d43013cc58f87136f"}
Sep 17 17:56:53.501556 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:53.501354    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"
Sep 17 17:56:53.532508 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:53.532461    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5" podStartSLOduration=3.654418298 podStartE2EDuration="7.532445765s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.878027467s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:56:53.531354234 +0000 UTC m=+1388.860453858" watchObservedRunningTime="2026-09-17 17:56:53.532445765 +0000 UTC m=+1388.861545389"
Sep 17 17:56:54.505611 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:54.505573    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"
Sep 17 17:56:54.506703 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:54.506674    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5" podUID="1faa7d9b-70ec-4e07-b774-8427593f98b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 17:56:55.509492 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:56:55.509445    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5" podUID="1faa7d9b-70ec-4e07-b774-8427593f98b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 17:57:00.513974 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:57:00.513912    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"
Sep 17 17:57:00.514570 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:57:00.514541    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5" podUID="1faa7d9b-70ec-4e07-b774-8427593f98b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 17:57:10.515299 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:57:10.515261    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5" podUID="1faa7d9b-70ec-4e07-b774-8427593f98b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 17:57:20.515117 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:57:20.515070    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5" podUID="1faa7d9b-70ec-4e07-b774-8427593f98b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 17:57:30.514956 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:57:30.514918    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5" podUID="1faa7d9b-70ec-4e07-b774-8427593f98b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 17:57:40.515488 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:57:40.515400    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5" podUID="1faa7d9b-70ec-4e07-b774-8427593f98b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 17:57:50.515296 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:57:50.515256    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5" podUID="1faa7d9b-70ec-4e07-b774-8427593f98b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 17:58:00.515406 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:00.515359    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5" podUID="1faa7d9b-70ec-4e07-b774-8427593f98b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 17:58:10.516014 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:10.515982    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"
Sep 17 17:58:16.985096 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:16.985032    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"]
Sep 17 17:58:16.985611 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:16.985300    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5" podUID="1faa7d9b-70ec-4e07-b774-8427593f98b1" containerName="kserve-container" containerID="cri-o://a065a171d3708e8f6021e99d2fc5437ac3eea1185567873d43013cc58f87136f" gracePeriod=30
Sep 17 17:58:16.985611 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:16.985325    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5" podUID="1faa7d9b-70ec-4e07-b774-8427593f98b1" containerName="kube-rbac-proxy" containerID="cri-o://42b9bdba438a3d466007ec6981cdd19723f8065ea9a4e7dfc715f56e4ac3dd43" gracePeriod=30
Sep 17 17:58:17.010489 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:17.010463    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:17.833353 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:17.833318    2574 generic.go:356] "Generic (PLEG): container finished" podID="1faa7d9b-70ec-4e07-b774-8427593f98b1" containerID="42b9bdba438a3d466007ec6981cdd19723f8065ea9a4e7dfc715f56e4ac3dd43" exitCode=2
Sep 17 17:58:17.833548 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:17.833398    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5" event={"ID":"1faa7d9b-70ec-4e07-b774-8427593f98b1","Type":"ContainerDied","Data":"42b9bdba438a3d466007ec6981cdd19723f8065ea9a4e7dfc715f56e4ac3dd43"}
Sep 17 17:58:20.510292 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:20.510251    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5" podUID="1faa7d9b-70ec-4e07-b774-8427593f98b1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 17 17:58:20.514597 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:20.514571    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5" podUID="1faa7d9b-70ec-4e07-b774-8427593f98b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 17:58:21.226184 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.226159    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"
Sep 17 17:58:21.242906 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.242882    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1faa7d9b-70ec-4e07-b774-8427593f98b1-kube-rbac-proxy-sar-config\") pod \"1faa7d9b-70ec-4e07-b774-8427593f98b1\" (UID: \"1faa7d9b-70ec-4e07-b774-8427593f98b1\") "
Sep 17 17:58:21.243024 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.242925    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1faa7d9b-70ec-4e07-b774-8427593f98b1-proxy-tls\") pod \"1faa7d9b-70ec-4e07-b774-8427593f98b1\" (UID: \"1faa7d9b-70ec-4e07-b774-8427593f98b1\") "
Sep 17 17:58:21.243024 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.242966    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1faa7d9b-70ec-4e07-b774-8427593f98b1-kserve-provision-location\") pod \"1faa7d9b-70ec-4e07-b774-8427593f98b1\" (UID: \"1faa7d9b-70ec-4e07-b774-8427593f98b1\") "
Sep 17 17:58:21.243024 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.242986    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ttdt\" (UniqueName: \"kubernetes.io/projected/1faa7d9b-70ec-4e07-b774-8427593f98b1-kube-api-access-2ttdt\") pod \"1faa7d9b-70ec-4e07-b774-8427593f98b1\" (UID: \"1faa7d9b-70ec-4e07-b774-8427593f98b1\") "
Sep 17 17:58:21.243024 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.243013    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1faa7d9b-70ec-4e07-b774-8427593f98b1-cabundle-cert\") pod \"1faa7d9b-70ec-4e07-b774-8427593f98b1\" (UID: \"1faa7d9b-70ec-4e07-b774-8427593f98b1\") "
Sep 17 17:58:21.243335 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.243301    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1faa7d9b-70ec-4e07-b774-8427593f98b1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1faa7d9b-70ec-4e07-b774-8427593f98b1" (UID: "1faa7d9b-70ec-4e07-b774-8427593f98b1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:58:21.243440 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.243389    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1faa7d9b-70ec-4e07-b774-8427593f98b1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1faa7d9b-70ec-4e07-b774-8427593f98b1" (UID: "1faa7d9b-70ec-4e07-b774-8427593f98b1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:58:21.243524 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.243499    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1faa7d9b-70ec-4e07-b774-8427593f98b1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1faa7d9b-70ec-4e07-b774-8427593f98b1" (UID: "1faa7d9b-70ec-4e07-b774-8427593f98b1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:58:21.245033 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.245012    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1faa7d9b-70ec-4e07-b774-8427593f98b1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1faa7d9b-70ec-4e07-b774-8427593f98b1" (UID: "1faa7d9b-70ec-4e07-b774-8427593f98b1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:58:21.245135 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.245012    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1faa7d9b-70ec-4e07-b774-8427593f98b1-kube-api-access-2ttdt" (OuterVolumeSpecName: "kube-api-access-2ttdt") pod "1faa7d9b-70ec-4e07-b774-8427593f98b1" (UID: "1faa7d9b-70ec-4e07-b774-8427593f98b1"). InnerVolumeSpecName "kube-api-access-2ttdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:58:21.344512 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.344449    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1faa7d9b-70ec-4e07-b774-8427593f98b1-cabundle-cert\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:58:21.344512 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.344472    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1faa7d9b-70ec-4e07-b774-8427593f98b1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:58:21.344512 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.344482    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1faa7d9b-70ec-4e07-b774-8427593f98b1-proxy-tls\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:58:21.344512 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.344491    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1faa7d9b-70ec-4e07-b774-8427593f98b1-kserve-provision-location\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:58:21.344512 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.344500    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ttdt\" (UniqueName: \"kubernetes.io/projected/1faa7d9b-70ec-4e07-b774-8427593f98b1-kube-api-access-2ttdt\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 17:58:21.849895 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.849864    2574 generic.go:356] "Generic (PLEG): container finished" podID="1faa7d9b-70ec-4e07-b774-8427593f98b1" containerID="a065a171d3708e8f6021e99d2fc5437ac3eea1185567873d43013cc58f87136f" exitCode=0
Sep 17 17:58:21.850292 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.849936    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"
Sep 17 17:58:21.850292 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.849940    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5" event={"ID":"1faa7d9b-70ec-4e07-b774-8427593f98b1","Type":"ContainerDied","Data":"a065a171d3708e8f6021e99d2fc5437ac3eea1185567873d43013cc58f87136f"}
Sep 17 17:58:21.850292 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.850083    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5" event={"ID":"1faa7d9b-70ec-4e07-b774-8427593f98b1","Type":"ContainerDied","Data":"04e694d36855ee2d388da46c539de1b28f134334d32eabd3cd47216a21f3b756"}
Sep 17 17:58:21.850292 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.850111    2574 scope.go:117] "RemoveContainer" containerID="42b9bdba438a3d466007ec6981cdd19723f8065ea9a4e7dfc715f56e4ac3dd43"
Sep 17 17:58:21.858223 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.858198    2574 scope.go:117] "RemoveContainer" containerID="a065a171d3708e8f6021e99d2fc5437ac3eea1185567873d43013cc58f87136f"
Sep 17 17:58:21.865237 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.865219    2574 scope.go:117] "RemoveContainer" containerID="013f6af8eac6749198c091d972cfeb0e77c29a88a9cab33a6b929b1ee56f0991"
Sep 17 17:58:21.872384 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.872367    2574 scope.go:117] "RemoveContainer" containerID="42b9bdba438a3d466007ec6981cdd19723f8065ea9a4e7dfc715f56e4ac3dd43"
Sep 17 17:58:21.872633 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:58:21.872614    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42b9bdba438a3d466007ec6981cdd19723f8065ea9a4e7dfc715f56e4ac3dd43\": container with ID starting with 42b9bdba438a3d466007ec6981cdd19723f8065ea9a4e7dfc715f56e4ac3dd43 not found: ID does not exist" containerID="42b9bdba438a3d466007ec6981cdd19723f8065ea9a4e7dfc715f56e4ac3dd43"
Sep 17 17:58:21.872686 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.872640    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42b9bdba438a3d466007ec6981cdd19723f8065ea9a4e7dfc715f56e4ac3dd43"} err="failed to get container status \"42b9bdba438a3d466007ec6981cdd19723f8065ea9a4e7dfc715f56e4ac3dd43\": rpc error: code = NotFound desc = could not find container \"42b9bdba438a3d466007ec6981cdd19723f8065ea9a4e7dfc715f56e4ac3dd43\": container with ID starting with 42b9bdba438a3d466007ec6981cdd19723f8065ea9a4e7dfc715f56e4ac3dd43 not found: ID does not exist"
Sep 17 17:58:21.872686 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.872655    2574 scope.go:117] "RemoveContainer" containerID="a065a171d3708e8f6021e99d2fc5437ac3eea1185567873d43013cc58f87136f"
Sep 17 17:58:21.872902 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:58:21.872887    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a065a171d3708e8f6021e99d2fc5437ac3eea1185567873d43013cc58f87136f\": container with ID starting with a065a171d3708e8f6021e99d2fc5437ac3eea1185567873d43013cc58f87136f not found: ID does not exist" containerID="a065a171d3708e8f6021e99d2fc5437ac3eea1185567873d43013cc58f87136f"
Sep 17 17:58:21.872945 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.872905    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a065a171d3708e8f6021e99d2fc5437ac3eea1185567873d43013cc58f87136f"} err="failed to get container status \"a065a171d3708e8f6021e99d2fc5437ac3eea1185567873d43013cc58f87136f\": rpc error: code = NotFound desc = could not find container \"a065a171d3708e8f6021e99d2fc5437ac3eea1185567873d43013cc58f87136f\": container with ID starting with a065a171d3708e8f6021e99d2fc5437ac3eea1185567873d43013cc58f87136f not found: ID does not exist"
Sep 17 17:58:21.872945 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.872917    2574 scope.go:117] "RemoveContainer" containerID="013f6af8eac6749198c091d972cfeb0e77c29a88a9cab33a6b929b1ee56f0991"
Sep 17 17:58:21.873216 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:58:21.873199    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"013f6af8eac6749198c091d972cfeb0e77c29a88a9cab33a6b929b1ee56f0991\": container with ID starting with 013f6af8eac6749198c091d972cfeb0e77c29a88a9cab33a6b929b1ee56f0991 not found: ID does not exist" containerID="013f6af8eac6749198c091d972cfeb0e77c29a88a9cab33a6b929b1ee56f0991"
Sep 17 17:58:21.873282 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.873220    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"013f6af8eac6749198c091d972cfeb0e77c29a88a9cab33a6b929b1ee56f0991"} err="failed to get container status \"013f6af8eac6749198c091d972cfeb0e77c29a88a9cab33a6b929b1ee56f0991\": rpc error: code = NotFound desc = could not find container \"013f6af8eac6749198c091d972cfeb0e77c29a88a9cab33a6b929b1ee56f0991\": container with ID starting with 013f6af8eac6749198c091d972cfeb0e77c29a88a9cab33a6b929b1ee56f0991 not found: ID does not exist"
Sep 17 17:58:21.876715 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.876695    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"]
Sep 17 17:58:21.888657 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:21.888636    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-103f5-predictor-6594b785b8-857v5"]
Sep 17 17:58:22.105526 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:22.105447    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:22.290875 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:22.290825    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:23.115508 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:23.115471    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1faa7d9b-70ec-4e07-b774-8427593f98b1" path="/var/lib/kubelet/pods/1faa7d9b-70ec-4e07-b774-8427593f98b1/volumes"
Sep 17 17:58:55.361708 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:55.361672    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:59.853259 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:58:59.853228    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:28.971805 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:28.971772    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:34.143897 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.143865    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:34.155696 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.155670    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"]
Sep 17 17:59:34.155953 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.155941    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1faa7d9b-70ec-4e07-b774-8427593f98b1" containerName="kserve-container"
Sep 17 17:59:34.155999 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.155954    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1faa7d9b-70ec-4e07-b774-8427593f98b1" containerName="kserve-container"
Sep 17 17:59:34.155999 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.155967    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1faa7d9b-70ec-4e07-b774-8427593f98b1" containerName="kube-rbac-proxy"
Sep 17 17:59:34.155999 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.155972    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1faa7d9b-70ec-4e07-b774-8427593f98b1" containerName="kube-rbac-proxy"
Sep 17 17:59:34.155999 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.155989    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1faa7d9b-70ec-4e07-b774-8427593f98b1" containerName="storage-initializer"
Sep 17 17:59:34.155999 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.155994    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1faa7d9b-70ec-4e07-b774-8427593f98b1" containerName="storage-initializer"
Sep 17 17:59:34.156195 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.156040    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1faa7d9b-70ec-4e07-b774-8427593f98b1" containerName="kserve-container"
Sep 17 17:59:34.156195 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.156049    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1faa7d9b-70ec-4e07-b774-8427593f98b1" containerName="kube-rbac-proxy"
Sep 17 17:59:34.159279 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.159261    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"
Sep 17 17:59:34.162065 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.162031    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:59:34.162181 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.162040    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 17:59:34.162181 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.162085    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:59:34.163088 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.163063    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 17 17:59:34.163248 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.163071    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-q5zmc\""
Sep 17 17:59:34.163248 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.163105    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:59:34.163446 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.163433    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:59:34.172313 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.172291    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"]
Sep 17 17:59:34.249418 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.249386    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"]
Sep 17 17:59:34.252711 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.252694    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"
Sep 17 17:59:34.255194 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.255167    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 17:59:34.272717 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.272698    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"]
Sep 17 17:59:34.277824 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.277798    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-proxy-tls\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-h8tpl\" (UID: \"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"
Sep 17 17:59:34.277942 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.277853    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kdwz\" (UniqueName: \"kubernetes.io/projected/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-kube-api-access-7kdwz\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-h8tpl\" (UID: \"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"
Sep 17 17:59:34.277942 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.277930    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-h8tpl\" (UID: \"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"
Sep 17 17:59:34.278029 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.278002    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-h8tpl\" (UID: \"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"
Sep 17 17:59:34.278092 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.278031    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-h8tpl\" (UID: \"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"
Sep 17 17:59:34.315384 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.315361    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:34.378409 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.378384    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ba84ca0-1c91-4404-9df6-e4aa1e841401-kserve-provision-location\") pod \"isvc-canary-create-predictor-6474b875b9-6qfbj\" (UID: \"5ba84ca0-1c91-4404-9df6-e4aa1e841401\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"
Sep 17 17:59:34.378505 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.378420    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kdwz\" (UniqueName: \"kubernetes.io/projected/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-kube-api-access-7kdwz\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-h8tpl\" (UID: \"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"
Sep 17 17:59:34.378505 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.378442    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-h8tpl\" (UID: \"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"
Sep 17 17:59:34.378505 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.378459    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ba84ca0-1c91-4404-9df6-e4aa1e841401-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6474b875b9-6qfbj\" (UID: \"5ba84ca0-1c91-4404-9df6-e4aa1e841401\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"
Sep 17 17:59:34.378505 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.378488    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ba84ca0-1c91-4404-9df6-e4aa1e841401-proxy-tls\") pod \"isvc-canary-create-predictor-6474b875b9-6qfbj\" (UID: \"5ba84ca0-1c91-4404-9df6-e4aa1e841401\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"
Sep 17 17:59:34.378727 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.378506    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ba84ca0-1c91-4404-9df6-e4aa1e841401-cabundle-cert\") pod \"isvc-canary-create-predictor-6474b875b9-6qfbj\" (UID: \"5ba84ca0-1c91-4404-9df6-e4aa1e841401\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"
Sep 17 17:59:34.378727 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.378525    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-h8tpl\" (UID: \"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"
Sep 17 17:59:34.378727 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.378551    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-h8tpl\" (UID: \"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"
Sep 17 17:59:34.378727 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.378662    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhdbs\" (UniqueName: \"kubernetes.io/projected/5ba84ca0-1c91-4404-9df6-e4aa1e841401-kube-api-access-fhdbs\") pod \"isvc-canary-create-predictor-6474b875b9-6qfbj\" (UID: \"5ba84ca0-1c91-4404-9df6-e4aa1e841401\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"
Sep 17 17:59:34.378904 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.378725    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-proxy-tls\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-h8tpl\" (UID: \"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"
Sep 17 17:59:34.378904 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:59:34.378853    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 17 17:59:34.378904 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.378869    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-h8tpl\" (UID: \"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"
Sep 17 17:59:34.379002 ip-10-0-142-4 kubenswrapper[2574]: E0917 17:59:34.378916    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-proxy-tls podName:ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e nodeName:}" failed. No retries permitted until 2026-09-17 17:59:34.878897118 +0000 UTC m=+1550.207996727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-proxy-tls") pod "isvc-canary-create-v2-predictor-96f77bdbd-h8tpl" (UID: "ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 17 17:59:34.379277 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.379261    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-h8tpl\" (UID: \"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"
Sep 17 17:59:34.379323 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.379280    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-h8tpl\" (UID: \"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"
Sep 17 17:59:34.395730 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.395672    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kdwz\" (UniqueName: \"kubernetes.io/projected/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-kube-api-access-7kdwz\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-h8tpl\" (UID: \"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"
Sep 17 17:59:34.413048 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.413022    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:34.479737 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.479712    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ba84ca0-1c91-4404-9df6-e4aa1e841401-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6474b875b9-6qfbj\" (UID: \"5ba84ca0-1c91-4404-9df6-e4aa1e841401\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"
Sep 17 17:59:34.479821 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.479750    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ba84ca0-1c91-4404-9df6-e4aa1e841401-proxy-tls\") pod \"isvc-canary-create-predictor-6474b875b9-6qfbj\" (UID: \"5ba84ca0-1c91-4404-9df6-e4aa1e841401\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"
Sep 17 17:59:34.479821 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.479766    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ba84ca0-1c91-4404-9df6-e4aa1e841401-cabundle-cert\") pod \"isvc-canary-create-predictor-6474b875b9-6qfbj\" (UID: \"5ba84ca0-1c91-4404-9df6-e4aa1e841401\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"
Sep 17 17:59:34.479821 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.479799    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhdbs\" (UniqueName: \"kubernetes.io/projected/5ba84ca0-1c91-4404-9df6-e4aa1e841401-kube-api-access-fhdbs\") pod \"isvc-canary-create-predictor-6474b875b9-6qfbj\" (UID: \"5ba84ca0-1c91-4404-9df6-e4aa1e841401\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"
Sep 17 17:59:34.479935 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.479864    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ba84ca0-1c91-4404-9df6-e4aa1e841401-kserve-provision-location\") pod \"isvc-canary-create-predictor-6474b875b9-6qfbj\" (UID: \"5ba84ca0-1c91-4404-9df6-e4aa1e841401\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"
Sep 17 17:59:34.480253 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.480232    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ba84ca0-1c91-4404-9df6-e4aa1e841401-kserve-provision-location\") pod \"isvc-canary-create-predictor-6474b875b9-6qfbj\" (UID: \"5ba84ca0-1c91-4404-9df6-e4aa1e841401\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"
Sep 17 17:59:34.480366 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.480349    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ba84ca0-1c91-4404-9df6-e4aa1e841401-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6474b875b9-6qfbj\" (UID: \"5ba84ca0-1c91-4404-9df6-e4aa1e841401\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"
Sep 17 17:59:34.480439 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.480422    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ba84ca0-1c91-4404-9df6-e4aa1e841401-cabundle-cert\") pod \"isvc-canary-create-predictor-6474b875b9-6qfbj\" (UID: \"5ba84ca0-1c91-4404-9df6-e4aa1e841401\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"
Sep 17 17:59:34.482160 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.482138    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ba84ca0-1c91-4404-9df6-e4aa1e841401-proxy-tls\") pod \"isvc-canary-create-predictor-6474b875b9-6qfbj\" (UID: \"5ba84ca0-1c91-4404-9df6-e4aa1e841401\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"
Sep 17 17:59:34.491202 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.491182    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhdbs\" (UniqueName: \"kubernetes.io/projected/5ba84ca0-1c91-4404-9df6-e4aa1e841401-kube-api-access-fhdbs\") pod \"isvc-canary-create-predictor-6474b875b9-6qfbj\" (UID: \"5ba84ca0-1c91-4404-9df6-e4aa1e841401\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"
Sep 17 17:59:34.550703 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.550680    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:34.562664 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.562644    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"
Sep 17 17:59:34.849649 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.849624    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"]
Sep 17 17:59:34.852149 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:59:34.852116    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba84ca0_1c91_4404_9df6_e4aa1e841401.slice/crio-77a00cc0db15def58fa0e5c665038e6926c8eb158b50acd50e5191075935365b WatchSource:0}: Error finding container 77a00cc0db15def58fa0e5c665038e6926c8eb158b50acd50e5191075935365b: Status 404 returned error can't find the container with id 77a00cc0db15def58fa0e5c665038e6926c8eb158b50acd50e5191075935365b
Sep 17 17:59:34.882553 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.882529    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-proxy-tls\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-h8tpl\" (UID: \"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"
Sep 17 17:59:34.884692 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:34.884669    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-proxy-tls\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-h8tpl\" (UID: \"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"
Sep 17 17:59:35.069487 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:35.069450    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"
Sep 17 17:59:35.134517 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:35.134488    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj" event={"ID":"5ba84ca0-1c91-4404-9df6-e4aa1e841401","Type":"ContainerStarted","Data":"c37525df2c6ab41ea6f8ccf060c46cc6efa728974737b4bf104e4de2b0b89071"}
Sep 17 17:59:35.134652 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:35.134553    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj" event={"ID":"5ba84ca0-1c91-4404-9df6-e4aa1e841401","Type":"ContainerStarted","Data":"77a00cc0db15def58fa0e5c665038e6926c8eb158b50acd50e5191075935365b"}
Sep 17 17:59:35.345444 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:35.345418    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"]
Sep 17 17:59:35.347823 ip-10-0-142-4 kubenswrapper[2574]: W0917 17:59:35.347795    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce340a4c_3fdb_4bd2_8ffc_0b5919b3591e.slice/crio-c73af51cfa6e2d5729116a6f7157842b88fcd8fb72b953be731ace35dad2fb3f WatchSource:0}: Error finding container c73af51cfa6e2d5729116a6f7157842b88fcd8fb72b953be731ace35dad2fb3f: Status 404 returned error can't find the container with id c73af51cfa6e2d5729116a6f7157842b88fcd8fb72b953be731ace35dad2fb3f
Sep 17 17:59:36.139652 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:36.139614    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl" event={"ID":"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e","Type":"ContainerStarted","Data":"357be23dac09aa373188535e00187e23f554a0bb9020e233c5c60fd4c123aa16"}
Sep 17 17:59:36.139853 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:36.139672    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl" event={"ID":"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e","Type":"ContainerStarted","Data":"c73af51cfa6e2d5729116a6f7157842b88fcd8fb72b953be731ace35dad2fb3f"}
Sep 17 17:59:39.153651 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:39.153618    2574 generic.go:356] "Generic (PLEG): container finished" podID="5ba84ca0-1c91-4404-9df6-e4aa1e841401" containerID="c37525df2c6ab41ea6f8ccf060c46cc6efa728974737b4bf104e4de2b0b89071" exitCode=0
Sep 17 17:59:39.154135 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:39.153677    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj" event={"ID":"5ba84ca0-1c91-4404-9df6-e4aa1e841401","Type":"ContainerDied","Data":"c37525df2c6ab41ea6f8ccf060c46cc6efa728974737b4bf104e4de2b0b89071"}
Sep 17 17:59:40.160235 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:40.160199    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj" event={"ID":"5ba84ca0-1c91-4404-9df6-e4aa1e841401","Type":"ContainerStarted","Data":"b0d20e70de42ed6acb5971cff60724916e8e7d7d2be517c0c46124d69966fb18"}
Sep 17 17:59:40.160668 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:40.160273    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj" event={"ID":"5ba84ca0-1c91-4404-9df6-e4aa1e841401","Type":"ContainerStarted","Data":"86d89a1eff5adf1821b6f5e535711a4f2256a62ef0898e59e492296f9497395d"}
Sep 17 17:59:40.160668 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:40.160578    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"
Sep 17 17:59:40.160784 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:40.160721    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"
Sep 17 17:59:40.161950 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:40.161923    2574 generic.go:356] "Generic (PLEG): container finished" podID="ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" containerID="357be23dac09aa373188535e00187e23f554a0bb9020e233c5c60fd4c123aa16" exitCode=0
Sep 17 17:59:40.162083 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:40.161990    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl" event={"ID":"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e","Type":"ContainerDied","Data":"357be23dac09aa373188535e00187e23f554a0bb9020e233c5c60fd4c123aa16"}
Sep 17 17:59:40.162157 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:40.162096    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj" podUID="5ba84ca0-1c91-4404-9df6-e4aa1e841401" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 17:59:40.182628 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:40.182588    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj" podStartSLOduration=2.270260429 podStartE2EDuration="6.182575271s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.912314842s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:40.180968628 +0000 UTC m=+1555.510068251" watchObservedRunningTime="2026-09-17 17:59:40.182575271 +0000 UTC m=+1555.511674895"
Sep 17 17:59:41.170424 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:41.170390    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl" event={"ID":"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e","Type":"ContainerStarted","Data":"9bc1dd20697033b1b4268f10089745f0607d2f5c29eb02d90aef6c7ab5c1d13e"}
Sep 17 17:59:41.170786 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:41.170460    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl" event={"ID":"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e","Type":"ContainerStarted","Data":"1af6ba855e9cf5b9ff77af5973640ecc1cd8601d5c1205a30cfa9d46c76e00f1"}
Sep 17 17:59:41.170786 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:41.170778    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"
Sep 17 17:59:41.170892 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:41.170802    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj" podUID="5ba84ca0-1c91-4404-9df6-e4aa1e841401" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 17:59:41.190852 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:41.190807    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl" podStartSLOduration=3.430881036 podStartE2EDuration="7.190793837s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.759912801s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:41.189559851 +0000 UTC m=+1556.518659477" watchObservedRunningTime="2026-09-17 17:59:41.190793837 +0000 UTC m=+1556.519893525"
Sep 17 17:59:42.174615 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:42.174578    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"
Sep 17 17:59:42.176045 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:42.176006    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl" podUID="ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 17:59:43.179399 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:43.179357    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl" podUID="ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 17:59:46.176183 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:46.176150    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"
Sep 17 17:59:46.176765 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:46.176738    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj" podUID="5ba84ca0-1c91-4404-9df6-e4aa1e841401" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 17:59:48.184228 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:48.184201    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"
Sep 17 17:59:48.184713 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:48.184683    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl" podUID="ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 17:59:52.218765 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:52.218737    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:56.177193 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:56.177154    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj" podUID="5ba84ca0-1c91-4404-9df6-e4aa1e841401" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 17:59:58.185003 ip-10-0-142-4 kubenswrapper[2574]: I0917 17:59:58.184966    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl" podUID="ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 18:00:06.176678 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:00:06.176633    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj" podUID="5ba84ca0-1c91-4404-9df6-e4aa1e841401" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 18:00:08.184728 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:00:08.184692    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl" podUID="ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 18:00:16.176616 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:00:16.176576    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj" podUID="5ba84ca0-1c91-4404-9df6-e4aa1e841401" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 18:00:18.185621 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:00:18.185585    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl" podUID="ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 18:00:26.177690 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:00:26.177644    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj" podUID="5ba84ca0-1c91-4404-9df6-e4aa1e841401" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 18:00:28.184884 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:00:28.184844    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl" podUID="ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 18:00:36.177457 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:00:36.177415    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj" podUID="5ba84ca0-1c91-4404-9df6-e4aa1e841401" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 18:00:38.185246 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:00:38.185163    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl" podUID="ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 18:00:46.176694 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:00:46.176648    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj" podUID="5ba84ca0-1c91-4404-9df6-e4aa1e841401" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 18:00:48.185534 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:00:48.185499    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl" podUID="ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 18:00:50.110517 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:00:50.110482    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl" podUID="ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 18:00:56.177718 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:00:56.177690    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"
Sep 17 18:01:00.111241 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:00.111207    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"
Sep 17 18:01:04.240921 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:04.240881    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"]
Sep 17 18:01:04.241380 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:04.241192    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl" podUID="ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" containerName="kserve-container" containerID="cri-o://1af6ba855e9cf5b9ff77af5973640ecc1cd8601d5c1205a30cfa9d46c76e00f1" gracePeriod=30
Sep 17 18:01:04.241453 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:04.241419    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl" podUID="ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" containerName="kube-rbac-proxy" containerID="cri-o://9bc1dd20697033b1b4268f10089745f0607d2f5c29eb02d90aef6c7ab5c1d13e" gracePeriod=30
Sep 17 18:01:04.242661 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:04.242635    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"]
Sep 17 18:01:04.242840 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:04.242822    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj" podUID="5ba84ca0-1c91-4404-9df6-e4aa1e841401" containerName="kserve-container" containerID="cri-o://86d89a1eff5adf1821b6f5e535711a4f2256a62ef0898e59e492296f9497395d" gracePeriod=30
Sep 17 18:01:04.242956 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:04.242915    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj" podUID="5ba84ca0-1c91-4404-9df6-e4aa1e841401" containerName="kube-rbac-proxy" containerID="cri-o://b0d20e70de42ed6acb5971cff60724916e8e7d7d2be517c0c46124d69966fb18" gracePeriod=30
Sep 17 18:01:04.503677 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:04.503597    2574 generic.go:356] "Generic (PLEG): container finished" podID="5ba84ca0-1c91-4404-9df6-e4aa1e841401" containerID="b0d20e70de42ed6acb5971cff60724916e8e7d7d2be517c0c46124d69966fb18" exitCode=2
Sep 17 18:01:04.503802 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:04.503675    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj" event={"ID":"5ba84ca0-1c91-4404-9df6-e4aa1e841401","Type":"ContainerDied","Data":"b0d20e70de42ed6acb5971cff60724916e8e7d7d2be517c0c46124d69966fb18"}
Sep 17 18:01:04.505558 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:04.505536    2574 generic.go:356] "Generic (PLEG): container finished" podID="ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" containerID="9bc1dd20697033b1b4268f10089745f0607d2f5c29eb02d90aef6c7ab5c1d13e" exitCode=2
Sep 17 18:01:04.505649 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:04.505567    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl" event={"ID":"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e","Type":"ContainerDied","Data":"9bc1dd20697033b1b4268f10089745f0607d2f5c29eb02d90aef6c7ab5c1d13e"}
Sep 17 18:01:05.261983 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:05.261954    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:05.337272 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:05.337243    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:06.171793 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:06.171751    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj" podUID="5ba84ca0-1c91-4404-9df6-e4aa1e841401" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 17 18:01:06.177239 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:06.177206    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj" podUID="5ba84ca0-1c91-4404-9df6-e4aa1e841401" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 18:01:08.179612 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:08.179567    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl" podUID="ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 17 18:01:08.785795 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:08.785771    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"
Sep 17 18:01:08.882808 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:08.882786    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"
Sep 17 18:01:08.919221 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:08.919196    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ba84ca0-1c91-4404-9df6-e4aa1e841401-kserve-provision-location\") pod \"5ba84ca0-1c91-4404-9df6-e4aa1e841401\" (UID: \"5ba84ca0-1c91-4404-9df6-e4aa1e841401\") "
Sep 17 18:01:08.919351 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:08.919262    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhdbs\" (UniqueName: \"kubernetes.io/projected/5ba84ca0-1c91-4404-9df6-e4aa1e841401-kube-api-access-fhdbs\") pod \"5ba84ca0-1c91-4404-9df6-e4aa1e841401\" (UID: \"5ba84ca0-1c91-4404-9df6-e4aa1e841401\") "
Sep 17 18:01:08.919351 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:08.919296    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ba84ca0-1c91-4404-9df6-e4aa1e841401-proxy-tls\") pod \"5ba84ca0-1c91-4404-9df6-e4aa1e841401\" (UID: \"5ba84ca0-1c91-4404-9df6-e4aa1e841401\") "
Sep 17 18:01:08.919464 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:08.919376    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ba84ca0-1c91-4404-9df6-e4aa1e841401-kube-rbac-proxy-sar-config\") pod \"5ba84ca0-1c91-4404-9df6-e4aa1e841401\" (UID: \"5ba84ca0-1c91-4404-9df6-e4aa1e841401\") "
Sep 17 18:01:08.919464 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:08.919416    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ba84ca0-1c91-4404-9df6-e4aa1e841401-cabundle-cert\") pod \"5ba84ca0-1c91-4404-9df6-e4aa1e841401\" (UID: \"5ba84ca0-1c91-4404-9df6-e4aa1e841401\") "
Sep 17 18:01:08.919556 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:08.919480    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba84ca0-1c91-4404-9df6-e4aa1e841401-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5ba84ca0-1c91-4404-9df6-e4aa1e841401" (UID: "5ba84ca0-1c91-4404-9df6-e4aa1e841401"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:01:08.919756 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:08.919734    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ba84ca0-1c91-4404-9df6-e4aa1e841401-kserve-provision-location\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 18:01:08.919836 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:08.919728    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba84ca0-1c91-4404-9df6-e4aa1e841401-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5ba84ca0-1c91-4404-9df6-e4aa1e841401" (UID: "5ba84ca0-1c91-4404-9df6-e4aa1e841401"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:01:08.919836 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:08.919775    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba84ca0-1c91-4404-9df6-e4aa1e841401-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5ba84ca0-1c91-4404-9df6-e4aa1e841401" (UID: "5ba84ca0-1c91-4404-9df6-e4aa1e841401"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:01:08.921349 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:08.921325    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba84ca0-1c91-4404-9df6-e4aa1e841401-kube-api-access-fhdbs" (OuterVolumeSpecName: "kube-api-access-fhdbs") pod "5ba84ca0-1c91-4404-9df6-e4aa1e841401" (UID: "5ba84ca0-1c91-4404-9df6-e4aa1e841401"). InnerVolumeSpecName "kube-api-access-fhdbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:01:08.921349 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:08.921333    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ba84ca0-1c91-4404-9df6-e4aa1e841401-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5ba84ca0-1c91-4404-9df6-e4aa1e841401" (UID: "5ba84ca0-1c91-4404-9df6-e4aa1e841401"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:01:09.020998 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.020971    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-kserve-provision-location\") pod \"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e\" (UID: \"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e\") "
Sep 17 18:01:09.021247 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.021021    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-cabundle-cert\") pod \"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e\" (UID: \"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e\") "
Sep 17 18:01:09.021247 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.021084    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kdwz\" (UniqueName: \"kubernetes.io/projected/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-kube-api-access-7kdwz\") pod \"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e\" (UID: \"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e\") "
Sep 17 18:01:09.021247 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.021220    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-proxy-tls\") pod \"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e\" (UID: \"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e\") "
Sep 17 18:01:09.021425 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.021277    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-kube-rbac-proxy-sar-config\") pod \"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e\" (UID: \"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e\") "
Sep 17 18:01:09.021425 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.021248    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" (UID: "ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:01:09.021425 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.021418    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" (UID: "ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:01:09.021558 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.021541    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-kserve-provision-location\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 18:01:09.021604 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.021563    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhdbs\" (UniqueName: \"kubernetes.io/projected/5ba84ca0-1c91-4404-9df6-e4aa1e841401-kube-api-access-fhdbs\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 18:01:09.021604 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.021578    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ba84ca0-1c91-4404-9df6-e4aa1e841401-proxy-tls\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 18:01:09.021604 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.021593    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-cabundle-cert\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 18:01:09.021604 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.021601    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ba84ca0-1c91-4404-9df6-e4aa1e841401-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 18:01:09.021731 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.021609    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ba84ca0-1c91-4404-9df6-e4aa1e841401-cabundle-cert\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 18:01:09.021731 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.021650    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" (UID: "ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:01:09.022936 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.022915    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-kube-api-access-7kdwz" (OuterVolumeSpecName: "kube-api-access-7kdwz") pod "ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" (UID: "ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e"). InnerVolumeSpecName "kube-api-access-7kdwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:01:09.023087 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.023069    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" (UID: "ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:01:09.125558 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.125489    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kdwz\" (UniqueName: \"kubernetes.io/projected/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-kube-api-access-7kdwz\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 18:01:09.125558 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.125521    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-proxy-tls\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 18:01:09.125558 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.125538    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 18:01:09.388981 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.388911    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:09.457081 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.457038    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"]
Sep 17 18:01:09.457356 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.457344    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" containerName="kserve-container"
Sep 17 18:01:09.457401 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.457358    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" containerName="kserve-container"
Sep 17 18:01:09.457401 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.457368    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ba84ca0-1c91-4404-9df6-e4aa1e841401" containerName="kube-rbac-proxy"
Sep 17 18:01:09.457401 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.457374    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ba84ca0-1c91-4404-9df6-e4aa1e841401" containerName="kube-rbac-proxy"
Sep 17 18:01:09.457401 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.457383    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ba84ca0-1c91-4404-9df6-e4aa1e841401" containerName="kserve-container"
Sep 17 18:01:09.457401 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.457390    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ba84ca0-1c91-4404-9df6-e4aa1e841401" containerName="kserve-container"
Sep 17 18:01:09.457560 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.457416    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" containerName="kube-rbac-proxy"
Sep 17 18:01:09.457560 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.457422    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" containerName="kube-rbac-proxy"
Sep 17 18:01:09.457560 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.457429    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ba84ca0-1c91-4404-9df6-e4aa1e841401" containerName="storage-initializer"
Sep 17 18:01:09.457560 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.457435    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ba84ca0-1c91-4404-9df6-e4aa1e841401" containerName="storage-initializer"
Sep 17 18:01:09.457560 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.457440    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" containerName="storage-initializer"
Sep 17 18:01:09.457560 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.457445    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" containerName="storage-initializer"
Sep 17 18:01:09.457560 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.457502    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ba84ca0-1c91-4404-9df6-e4aa1e841401" containerName="kube-rbac-proxy"
Sep 17 18:01:09.457560 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.457510    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ba84ca0-1c91-4404-9df6-e4aa1e841401" containerName="kserve-container"
Sep 17 18:01:09.457560 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.457517    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" containerName="kube-rbac-proxy"
Sep 17 18:01:09.457560 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.457523    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" containerName="kserve-container"
Sep 17 18:01:09.461428 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.461411    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"
Sep 17 18:01:09.463803 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.463782    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 18:01:09.464475 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.464459    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 18:01:09.475367 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.475342    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"]
Sep 17 18:01:09.495804 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.495779    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:09.527960 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.527937    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7cf78f884f-vbt2p\" (UID: \"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"
Sep 17 18:01:09.528103 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.527992    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-vbt2p\" (UID: \"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"
Sep 17 18:01:09.528103 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.528089    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7cf78f884f-vbt2p\" (UID: \"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"
Sep 17 18:01:09.528207 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.528132    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-cabundle-cert\") pod \"isvc-canary-promote-predictor-7cf78f884f-vbt2p\" (UID: \"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"
Sep 17 18:01:09.528207 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.528188    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxdgh\" (UniqueName: \"kubernetes.io/projected/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-kube-api-access-bxdgh\") pod \"isvc-canary-promote-predictor-7cf78f884f-vbt2p\" (UID: \"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"
Sep 17 18:01:09.528671 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.528649    2574 generic.go:356] "Generic (PLEG): container finished" podID="ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" containerID="1af6ba855e9cf5b9ff77af5973640ecc1cd8601d5c1205a30cfa9d46c76e00f1" exitCode=0
Sep 17 18:01:09.528809 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.528712    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"
Sep 17 18:01:09.528809 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.528731    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl" event={"ID":"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e","Type":"ContainerDied","Data":"1af6ba855e9cf5b9ff77af5973640ecc1cd8601d5c1205a30cfa9d46c76e00f1"}
Sep 17 18:01:09.528809 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.528785    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl" event={"ID":"ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e","Type":"ContainerDied","Data":"c73af51cfa6e2d5729116a6f7157842b88fcd8fb72b953be731ace35dad2fb3f"}
Sep 17 18:01:09.528946 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.528813    2574 scope.go:117] "RemoveContainer" containerID="9bc1dd20697033b1b4268f10089745f0607d2f5c29eb02d90aef6c7ab5c1d13e"
Sep 17 18:01:09.531239 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.531217    2574 generic.go:356] "Generic (PLEG): container finished" podID="5ba84ca0-1c91-4404-9df6-e4aa1e841401" containerID="86d89a1eff5adf1821b6f5e535711a4f2256a62ef0898e59e492296f9497395d" exitCode=0
Sep 17 18:01:09.531336 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.531249    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj" event={"ID":"5ba84ca0-1c91-4404-9df6-e4aa1e841401","Type":"ContainerDied","Data":"86d89a1eff5adf1821b6f5e535711a4f2256a62ef0898e59e492296f9497395d"}
Sep 17 18:01:09.531336 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.531310    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj" event={"ID":"5ba84ca0-1c91-4404-9df6-e4aa1e841401","Type":"ContainerDied","Data":"77a00cc0db15def58fa0e5c665038e6926c8eb158b50acd50e5191075935365b"}
Sep 17 18:01:09.531410 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.531382    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"
Sep 17 18:01:09.537337 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.537320    2574 scope.go:117] "RemoveContainer" containerID="1af6ba855e9cf5b9ff77af5973640ecc1cd8601d5c1205a30cfa9d46c76e00f1"
Sep 17 18:01:09.544677 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.544663    2574 scope.go:117] "RemoveContainer" containerID="357be23dac09aa373188535e00187e23f554a0bb9020e233c5c60fd4c123aa16"
Sep 17 18:01:09.551632 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.551603    2574 scope.go:117] "RemoveContainer" containerID="9bc1dd20697033b1b4268f10089745f0607d2f5c29eb02d90aef6c7ab5c1d13e"
Sep 17 18:01:09.551889 ip-10-0-142-4 kubenswrapper[2574]: E0917 18:01:09.551872    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bc1dd20697033b1b4268f10089745f0607d2f5c29eb02d90aef6c7ab5c1d13e\": container with ID starting with 9bc1dd20697033b1b4268f10089745f0607d2f5c29eb02d90aef6c7ab5c1d13e not found: ID does not exist" containerID="9bc1dd20697033b1b4268f10089745f0607d2f5c29eb02d90aef6c7ab5c1d13e"
Sep 17 18:01:09.551948 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.551897    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bc1dd20697033b1b4268f10089745f0607d2f5c29eb02d90aef6c7ab5c1d13e"} err="failed to get container status \"9bc1dd20697033b1b4268f10089745f0607d2f5c29eb02d90aef6c7ab5c1d13e\": rpc error: code = NotFound desc = could not find container \"9bc1dd20697033b1b4268f10089745f0607d2f5c29eb02d90aef6c7ab5c1d13e\": container with ID starting with 9bc1dd20697033b1b4268f10089745f0607d2f5c29eb02d90aef6c7ab5c1d13e not found: ID does not exist"
Sep 17 18:01:09.551948 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.551914    2574 scope.go:117] "RemoveContainer" containerID="1af6ba855e9cf5b9ff77af5973640ecc1cd8601d5c1205a30cfa9d46c76e00f1"
Sep 17 18:01:09.552157 ip-10-0-142-4 kubenswrapper[2574]: E0917 18:01:09.552141    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1af6ba855e9cf5b9ff77af5973640ecc1cd8601d5c1205a30cfa9d46c76e00f1\": container with ID starting with 1af6ba855e9cf5b9ff77af5973640ecc1cd8601d5c1205a30cfa9d46c76e00f1 not found: ID does not exist" containerID="1af6ba855e9cf5b9ff77af5973640ecc1cd8601d5c1205a30cfa9d46c76e00f1"
Sep 17 18:01:09.552205 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.552163    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af6ba855e9cf5b9ff77af5973640ecc1cd8601d5c1205a30cfa9d46c76e00f1"} err="failed to get container status \"1af6ba855e9cf5b9ff77af5973640ecc1cd8601d5c1205a30cfa9d46c76e00f1\": rpc error: code = NotFound desc = could not find container \"1af6ba855e9cf5b9ff77af5973640ecc1cd8601d5c1205a30cfa9d46c76e00f1\": container with ID starting with 1af6ba855e9cf5b9ff77af5973640ecc1cd8601d5c1205a30cfa9d46c76e00f1 not found: ID does not exist"
Sep 17 18:01:09.552205 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.552183    2574 scope.go:117] "RemoveContainer" containerID="357be23dac09aa373188535e00187e23f554a0bb9020e233c5c60fd4c123aa16"
Sep 17 18:01:09.552420 ip-10-0-142-4 kubenswrapper[2574]: E0917 18:01:09.552404    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"357be23dac09aa373188535e00187e23f554a0bb9020e233c5c60fd4c123aa16\": container with ID starting with 357be23dac09aa373188535e00187e23f554a0bb9020e233c5c60fd4c123aa16 not found: ID does not exist" containerID="357be23dac09aa373188535e00187e23f554a0bb9020e233c5c60fd4c123aa16"
Sep 17 18:01:09.552462 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.552431    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"357be23dac09aa373188535e00187e23f554a0bb9020e233c5c60fd4c123aa16"} err="failed to get container status \"357be23dac09aa373188535e00187e23f554a0bb9020e233c5c60fd4c123aa16\": rpc error: code = NotFound desc = could not find container \"357be23dac09aa373188535e00187e23f554a0bb9020e233c5c60fd4c123aa16\": container with ID starting with 357be23dac09aa373188535e00187e23f554a0bb9020e233c5c60fd4c123aa16 not found: ID does not exist"
Sep 17 18:01:09.552462 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.552444    2574 scope.go:117] "RemoveContainer" containerID="b0d20e70de42ed6acb5971cff60724916e8e7d7d2be517c0c46124d69966fb18"
Sep 17 18:01:09.555829 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.555808    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"]
Sep 17 18:01:09.557849 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.557826    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-h8tpl"]
Sep 17 18:01:09.560086 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.560068    2574 scope.go:117] "RemoveContainer" containerID="86d89a1eff5adf1821b6f5e535711a4f2256a62ef0898e59e492296f9497395d"
Sep 17 18:01:09.566917 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.566899    2574 scope.go:117] "RemoveContainer" containerID="c37525df2c6ab41ea6f8ccf060c46cc6efa728974737b4bf104e4de2b0b89071"
Sep 17 18:01:09.572779 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.572758    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"]
Sep 17 18:01:09.574154 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.574121    2574 scope.go:117] "RemoveContainer" containerID="b0d20e70de42ed6acb5971cff60724916e8e7d7d2be517c0c46124d69966fb18"
Sep 17 18:01:09.574416 ip-10-0-142-4 kubenswrapper[2574]: E0917 18:01:09.574395    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0d20e70de42ed6acb5971cff60724916e8e7d7d2be517c0c46124d69966fb18\": container with ID starting with b0d20e70de42ed6acb5971cff60724916e8e7d7d2be517c0c46124d69966fb18 not found: ID does not exist" containerID="b0d20e70de42ed6acb5971cff60724916e8e7d7d2be517c0c46124d69966fb18"
Sep 17 18:01:09.574492 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.574421    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0d20e70de42ed6acb5971cff60724916e8e7d7d2be517c0c46124d69966fb18"} err="failed to get container status \"b0d20e70de42ed6acb5971cff60724916e8e7d7d2be517c0c46124d69966fb18\": rpc error: code = NotFound desc = could not find container \"b0d20e70de42ed6acb5971cff60724916e8e7d7d2be517c0c46124d69966fb18\": container with ID starting with b0d20e70de42ed6acb5971cff60724916e8e7d7d2be517c0c46124d69966fb18 not found: ID does not exist"
Sep 17 18:01:09.574492 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.574441    2574 scope.go:117] "RemoveContainer" containerID="86d89a1eff5adf1821b6f5e535711a4f2256a62ef0898e59e492296f9497395d"
Sep 17 18:01:09.574741 ip-10-0-142-4 kubenswrapper[2574]: E0917 18:01:09.574723    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86d89a1eff5adf1821b6f5e535711a4f2256a62ef0898e59e492296f9497395d\": container with ID starting with 86d89a1eff5adf1821b6f5e535711a4f2256a62ef0898e59e492296f9497395d not found: ID does not exist" containerID="86d89a1eff5adf1821b6f5e535711a4f2256a62ef0898e59e492296f9497395d"
Sep 17 18:01:09.574787 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.574748    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86d89a1eff5adf1821b6f5e535711a4f2256a62ef0898e59e492296f9497395d"} err="failed to get container status \"86d89a1eff5adf1821b6f5e535711a4f2256a62ef0898e59e492296f9497395d\": rpc error: code = NotFound desc = could not find container \"86d89a1eff5adf1821b6f5e535711a4f2256a62ef0898e59e492296f9497395d\": container with ID starting with 86d89a1eff5adf1821b6f5e535711a4f2256a62ef0898e59e492296f9497395d not found: ID does not exist"
Sep 17 18:01:09.574787 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.574763    2574 scope.go:117] "RemoveContainer" containerID="c37525df2c6ab41ea6f8ccf060c46cc6efa728974737b4bf104e4de2b0b89071"
Sep 17 18:01:09.575032 ip-10-0-142-4 kubenswrapper[2574]: E0917 18:01:09.575017    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c37525df2c6ab41ea6f8ccf060c46cc6efa728974737b4bf104e4de2b0b89071\": container with ID starting with c37525df2c6ab41ea6f8ccf060c46cc6efa728974737b4bf104e4de2b0b89071 not found: ID does not exist" containerID="c37525df2c6ab41ea6f8ccf060c46cc6efa728974737b4bf104e4de2b0b89071"
Sep 17 18:01:09.575122 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.575035    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37525df2c6ab41ea6f8ccf060c46cc6efa728974737b4bf104e4de2b0b89071"} err="failed to get container status \"c37525df2c6ab41ea6f8ccf060c46cc6efa728974737b4bf104e4de2b0b89071\": rpc error: code = NotFound desc = could not find container \"c37525df2c6ab41ea6f8ccf060c46cc6efa728974737b4bf104e4de2b0b89071\": container with ID starting with c37525df2c6ab41ea6f8ccf060c46cc6efa728974737b4bf104e4de2b0b89071 not found: ID does not exist"
Sep 17 18:01:09.575556 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.575539    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-6qfbj"]
Sep 17 18:01:09.579950 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.579933    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:09.629362 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.629333    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-vbt2p\" (UID: \"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"
Sep 17 18:01:09.629484 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.629379    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7cf78f884f-vbt2p\" (UID: \"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"
Sep 17 18:01:09.629484 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.629411    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-cabundle-cert\") pod \"isvc-canary-promote-predictor-7cf78f884f-vbt2p\" (UID: \"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"
Sep 17 18:01:09.629484 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.629439    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxdgh\" (UniqueName: \"kubernetes.io/projected/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-kube-api-access-bxdgh\") pod \"isvc-canary-promote-predictor-7cf78f884f-vbt2p\" (UID: \"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"
Sep 17 18:01:09.629654 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.629483    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7cf78f884f-vbt2p\" (UID: \"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"
Sep 17 18:01:09.629900 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.629880    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7cf78f884f-vbt2p\" (UID: \"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"
Sep 17 18:01:09.630209 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.630190    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-cabundle-cert\") pod \"isvc-canary-promote-predictor-7cf78f884f-vbt2p\" (UID: \"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"
Sep 17 18:01:09.630278 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.630207    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7cf78f884f-vbt2p\" (UID: \"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"
Sep 17 18:01:09.631712 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.631695    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-vbt2p\" (UID: \"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"
Sep 17 18:01:09.638589 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.638573    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxdgh\" (UniqueName: \"kubernetes.io/projected/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-kube-api-access-bxdgh\") pod \"isvc-canary-promote-predictor-7cf78f884f-vbt2p\" (UID: \"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"
Sep 17 18:01:09.658130 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.658079    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:09.771400 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:09.771363    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"
Sep 17 18:01:10.045026 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:10.045001    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"]
Sep 17 18:01:10.047736 ip-10-0-142-4 kubenswrapper[2574]: W0917 18:01:10.047707    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d6ae8ef_40a3_4cfe_a560_1bab52ff38bd.slice/crio-ba3a5b4b4be23748a6a2fca0fae63099ceb716098f9df55f3903ed743aa1ad7a WatchSource:0}: Error finding container ba3a5b4b4be23748a6a2fca0fae63099ceb716098f9df55f3903ed743aa1ad7a: Status 404 returned error can't find the container with id ba3a5b4b4be23748a6a2fca0fae63099ceb716098f9df55f3903ed743aa1ad7a
Sep 17 18:01:10.542178 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:10.542140    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p" event={"ID":"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd","Type":"ContainerStarted","Data":"f00f9f8e6de200250cc6292e50e51711a209c8efe4ca9707ddd3045b180c6417"}
Sep 17 18:01:10.542574 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:10.542204    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p" event={"ID":"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd","Type":"ContainerStarted","Data":"ba3a5b4b4be23748a6a2fca0fae63099ceb716098f9df55f3903ed743aa1ad7a"}
Sep 17 18:01:11.116182 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:11.116146    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ba84ca0-1c91-4404-9df6-e4aa1e841401" path="/var/lib/kubelet/pods/5ba84ca0-1c91-4404-9df6-e4aa1e841401/volumes"
Sep 17 18:01:11.116626 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:11.116611    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e" path="/var/lib/kubelet/pods/ce340a4c-3fdb-4bd2-8ffc-0b5919b3591e/volumes"
Sep 17 18:01:14.561319 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:14.561284    2574 generic.go:356] "Generic (PLEG): container finished" podID="3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" containerID="f00f9f8e6de200250cc6292e50e51711a209c8efe4ca9707ddd3045b180c6417" exitCode=0
Sep 17 18:01:14.561732 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:14.561354    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p" event={"ID":"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd","Type":"ContainerDied","Data":"f00f9f8e6de200250cc6292e50e51711a209c8efe4ca9707ddd3045b180c6417"}
Sep 17 18:01:15.567118 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:15.567081    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p" event={"ID":"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd","Type":"ContainerStarted","Data":"fdf9bacab98b0635d51dca73cdc8938298153e71c5c5f0ccefd56ec992e24a88"}
Sep 17 18:01:15.567534 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:15.567144    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p" event={"ID":"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd","Type":"ContainerStarted","Data":"b9e8a0b240f91ddebd92379019e61367c5e5e1cb256e981381ca2143b6bc3e9c"}
Sep 17 18:01:15.567534 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:15.567344    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"
Sep 17 18:01:15.590201 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:15.590154    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p" podStartSLOduration=2.8789244529999998 podStartE2EDuration="6.590138077s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.711213624s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:01:15.588422525 +0000 UTC m=+1650.917522149" watchObservedRunningTime="2026-09-17 18:01:15.590138077 +0000 UTC m=+1650.919237701"
Sep 17 18:01:16.570902 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:16.570872    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"
Sep 17 18:01:16.572270 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:16.572242    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p" podUID="3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:01:17.575147 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:17.575112    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p" podUID="3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:01:22.579599 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:22.579568    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"
Sep 17 18:01:22.580159 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:22.580124    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p" podUID="3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:01:32.579987 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:32.579946    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p" podUID="3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:01:42.580083 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:42.580027    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p" podUID="3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:01:52.580426 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:01:52.580385    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p" podUID="3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:02:02.580364 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:02.580322    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p" podUID="3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:02:09.799171 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:09.799084    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:09.861815 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:09.861779    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:12.580828 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:12.580787    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p" podUID="3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:02:22.580893 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:22.580844    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p" podUID="3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:02:32.111216 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:32.111186    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"
Sep 17 18:02:32.686946 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:32.686917    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"]
Sep 17 18:02:33.878017 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:33.877973    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p" podUID="3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" containerName="kserve-container" containerID="cri-o://b9e8a0b240f91ddebd92379019e61367c5e5e1cb256e981381ca2143b6bc3e9c" gracePeriod=30
Sep 17 18:02:33.878017 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:33.878010    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p" podUID="3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" containerName="kube-rbac-proxy" containerID="cri-o://fdf9bacab98b0635d51dca73cdc8938298153e71c5c5f0ccefd56ec992e24a88" gracePeriod=30
Sep 17 18:02:34.884558 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:34.884523    2574 generic.go:356] "Generic (PLEG): container finished" podID="3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" containerID="fdf9bacab98b0635d51dca73cdc8938298153e71c5c5f0ccefd56ec992e24a88" exitCode=2
Sep 17 18:02:34.884934 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:34.884596    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p" event={"ID":"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd","Type":"ContainerDied","Data":"fdf9bacab98b0635d51dca73cdc8938298153e71c5c5f0ccefd56ec992e24a88"}
Sep 17 18:02:37.575666 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:37.575622    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p" podUID="3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 17 18:02:38.020098 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.020076    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"
Sep 17 18:02:38.093903 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.093874    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-proxy-tls\") pod \"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd\" (UID: \"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd\") "
Sep 17 18:02:38.094048 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.093949    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-cabundle-cert\") pod \"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd\" (UID: \"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd\") "
Sep 17 18:02:38.094048 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.093987    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxdgh\" (UniqueName: \"kubernetes.io/projected/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-kube-api-access-bxdgh\") pod \"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd\" (UID: \"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd\") "
Sep 17 18:02:38.094199 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.094049    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-kube-rbac-proxy-sar-config\") pod \"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd\" (UID: \"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd\") "
Sep 17 18:02:38.094199 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.094102    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-kserve-provision-location\") pod \"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd\" (UID: \"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd\") "
Sep 17 18:02:38.094387 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.094360    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" (UID: "3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:38.094437 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.094399    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" (UID: "3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:38.094473 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.094443    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" (UID: "3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:38.095981 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.095960    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" (UID: "3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:02:38.096039 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.096005    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-kube-api-access-bxdgh" (OuterVolumeSpecName: "kube-api-access-bxdgh") pod "3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" (UID: "3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd"). InnerVolumeSpecName "kube-api-access-bxdgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:02:38.195701 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.195638    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxdgh\" (UniqueName: \"kubernetes.io/projected/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-kube-api-access-bxdgh\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 18:02:38.195701 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.195662    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 18:02:38.195701 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.195671    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-kserve-provision-location\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 18:02:38.195701 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.195680    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-proxy-tls\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 18:02:38.195701 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.195691    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd-cabundle-cert\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 18:02:38.902292 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.902258    2574 generic.go:356] "Generic (PLEG): container finished" podID="3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" containerID="b9e8a0b240f91ddebd92379019e61367c5e5e1cb256e981381ca2143b6bc3e9c" exitCode=0
Sep 17 18:02:38.902727 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.902311    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"
Sep 17 18:02:38.902727 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.902341    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p" event={"ID":"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd","Type":"ContainerDied","Data":"b9e8a0b240f91ddebd92379019e61367c5e5e1cb256e981381ca2143b6bc3e9c"}
Sep 17 18:02:38.902727 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.902399    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p" event={"ID":"3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd","Type":"ContainerDied","Data":"ba3a5b4b4be23748a6a2fca0fae63099ceb716098f9df55f3903ed743aa1ad7a"}
Sep 17 18:02:38.902727 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.902427    2574 scope.go:117] "RemoveContainer" containerID="fdf9bacab98b0635d51dca73cdc8938298153e71c5c5f0ccefd56ec992e24a88"
Sep 17 18:02:38.911357 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.911341    2574 scope.go:117] "RemoveContainer" containerID="b9e8a0b240f91ddebd92379019e61367c5e5e1cb256e981381ca2143b6bc3e9c"
Sep 17 18:02:38.918521 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.918506    2574 scope.go:117] "RemoveContainer" containerID="f00f9f8e6de200250cc6292e50e51711a209c8efe4ca9707ddd3045b180c6417"
Sep 17 18:02:38.926016 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.925932    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"]
Sep 17 18:02:38.926080 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.926039    2574 scope.go:117] "RemoveContainer" containerID="fdf9bacab98b0635d51dca73cdc8938298153e71c5c5f0ccefd56ec992e24a88"
Sep 17 18:02:38.926401 ip-10-0-142-4 kubenswrapper[2574]: E0917 18:02:38.926380    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdf9bacab98b0635d51dca73cdc8938298153e71c5c5f0ccefd56ec992e24a88\": container with ID starting with fdf9bacab98b0635d51dca73cdc8938298153e71c5c5f0ccefd56ec992e24a88 not found: ID does not exist" containerID="fdf9bacab98b0635d51dca73cdc8938298153e71c5c5f0ccefd56ec992e24a88"
Sep 17 18:02:38.926466 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.926407    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdf9bacab98b0635d51dca73cdc8938298153e71c5c5f0ccefd56ec992e24a88"} err="failed to get container status \"fdf9bacab98b0635d51dca73cdc8938298153e71c5c5f0ccefd56ec992e24a88\": rpc error: code = NotFound desc = could not find container \"fdf9bacab98b0635d51dca73cdc8938298153e71c5c5f0ccefd56ec992e24a88\": container with ID starting with fdf9bacab98b0635d51dca73cdc8938298153e71c5c5f0ccefd56ec992e24a88 not found: ID does not exist"
Sep 17 18:02:38.926466 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.926425    2574 scope.go:117] "RemoveContainer" containerID="b9e8a0b240f91ddebd92379019e61367c5e5e1cb256e981381ca2143b6bc3e9c"
Sep 17 18:02:38.926705 ip-10-0-142-4 kubenswrapper[2574]: E0917 18:02:38.926679    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9e8a0b240f91ddebd92379019e61367c5e5e1cb256e981381ca2143b6bc3e9c\": container with ID starting with b9e8a0b240f91ddebd92379019e61367c5e5e1cb256e981381ca2143b6bc3e9c not found: ID does not exist" containerID="b9e8a0b240f91ddebd92379019e61367c5e5e1cb256e981381ca2143b6bc3e9c"
Sep 17 18:02:38.926747 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.926711    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9e8a0b240f91ddebd92379019e61367c5e5e1cb256e981381ca2143b6bc3e9c"} err="failed to get container status \"b9e8a0b240f91ddebd92379019e61367c5e5e1cb256e981381ca2143b6bc3e9c\": rpc error: code = NotFound desc = could not find container \"b9e8a0b240f91ddebd92379019e61367c5e5e1cb256e981381ca2143b6bc3e9c\": container with ID starting with b9e8a0b240f91ddebd92379019e61367c5e5e1cb256e981381ca2143b6bc3e9c not found: ID does not exist"
Sep 17 18:02:38.926747 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.926726    2574 scope.go:117] "RemoveContainer" containerID="f00f9f8e6de200250cc6292e50e51711a209c8efe4ca9707ddd3045b180c6417"
Sep 17 18:02:38.926973 ip-10-0-142-4 kubenswrapper[2574]: E0917 18:02:38.926957    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f00f9f8e6de200250cc6292e50e51711a209c8efe4ca9707ddd3045b180c6417\": container with ID starting with f00f9f8e6de200250cc6292e50e51711a209c8efe4ca9707ddd3045b180c6417 not found: ID does not exist" containerID="f00f9f8e6de200250cc6292e50e51711a209c8efe4ca9707ddd3045b180c6417"
Sep 17 18:02:38.927020 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.926977    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f00f9f8e6de200250cc6292e50e51711a209c8efe4ca9707ddd3045b180c6417"} err="failed to get container status \"f00f9f8e6de200250cc6292e50e51711a209c8efe4ca9707ddd3045b180c6417\": rpc error: code = NotFound desc = could not find container \"f00f9f8e6de200250cc6292e50e51711a209c8efe4ca9707ddd3045b180c6417\": container with ID starting with f00f9f8e6de200250cc6292e50e51711a209c8efe4ca9707ddd3045b180c6417 not found: ID does not exist"
Sep 17 18:02:38.929631 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:38.929610    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-vbt2p"]
Sep 17 18:02:39.114940 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.114909    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" path="/var/lib/kubelet/pods/3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd/volumes"
Sep 17 18:02:39.599939 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.599912    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:39.615900 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.615875    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"]
Sep 17 18:02:39.616188 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.616175    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" containerName="kube-rbac-proxy"
Sep 17 18:02:39.616188 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.616189    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" containerName="kube-rbac-proxy"
Sep 17 18:02:39.616281 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.616201    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" containerName="storage-initializer"
Sep 17 18:02:39.616281 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.616206    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" containerName="storage-initializer"
Sep 17 18:02:39.616281 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.616216    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" containerName="kserve-container"
Sep 17 18:02:39.616281 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.616221    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" containerName="kserve-container"
Sep 17 18:02:39.616281 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.616276    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" containerName="kserve-container"
Sep 17 18:02:39.616281 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.616284    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d6ae8ef-40a3-4cfe-a560-1bab52ff38bd" containerName="kube-rbac-proxy"
Sep 17 18:02:39.620740 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.620723    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"
Sep 17 18:02:39.622950 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.622929    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 18:02:39.623067 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.622987    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-q5zmc\""
Sep 17 18:02:39.623225 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.623210    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:02:39.623782 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.623756    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:02:39.623871 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.623779    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:02:39.623871 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.623781    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:02:39.623871 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.623781    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 18:02:39.630850 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.630831    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"]
Sep 17 18:02:39.706413 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.706389    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a10c910-c03b-4208-a791-73b4d0210e83-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm\" (UID: \"6a10c910-c03b-4208-a791-73b4d0210e83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"
Sep 17 18:02:39.706523 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.706420    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a10c910-c03b-4208-a791-73b4d0210e83-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm\" (UID: \"6a10c910-c03b-4208-a791-73b4d0210e83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"
Sep 17 18:02:39.706523 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.706453    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvcvh\" (UniqueName: \"kubernetes.io/projected/6a10c910-c03b-4208-a791-73b4d0210e83-kube-api-access-kvcvh\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm\" (UID: \"6a10c910-c03b-4208-a791-73b4d0210e83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"
Sep 17 18:02:39.706602 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.706529    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a10c910-c03b-4208-a791-73b4d0210e83-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm\" (UID: \"6a10c910-c03b-4208-a791-73b4d0210e83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"
Sep 17 18:02:39.706602 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.706562    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a10c910-c03b-4208-a791-73b4d0210e83-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm\" (UID: \"6a10c910-c03b-4208-a791-73b4d0210e83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"
Sep 17 18:02:39.807394 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.807359    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a10c910-c03b-4208-a791-73b4d0210e83-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm\" (UID: \"6a10c910-c03b-4208-a791-73b4d0210e83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"
Sep 17 18:02:39.807394 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.807402    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a10c910-c03b-4208-a791-73b4d0210e83-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm\" (UID: \"6a10c910-c03b-4208-a791-73b4d0210e83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"
Sep 17 18:02:39.807544 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.807451    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvcvh\" (UniqueName: \"kubernetes.io/projected/6a10c910-c03b-4208-a791-73b4d0210e83-kube-api-access-kvcvh\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm\" (UID: \"6a10c910-c03b-4208-a791-73b4d0210e83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"
Sep 17 18:02:39.807544 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.807483    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a10c910-c03b-4208-a791-73b4d0210e83-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm\" (UID: \"6a10c910-c03b-4208-a791-73b4d0210e83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"
Sep 17 18:02:39.807544 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.807514    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a10c910-c03b-4208-a791-73b4d0210e83-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm\" (UID: \"6a10c910-c03b-4208-a791-73b4d0210e83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"
Sep 17 18:02:39.807904 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.807886    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a10c910-c03b-4208-a791-73b4d0210e83-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm\" (UID: \"6a10c910-c03b-4208-a791-73b4d0210e83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"
Sep 17 18:02:39.808225 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.808204    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a10c910-c03b-4208-a791-73b4d0210e83-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm\" (UID: \"6a10c910-c03b-4208-a791-73b4d0210e83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"
Sep 17 18:02:39.808275 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.808204    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a10c910-c03b-4208-a791-73b4d0210e83-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm\" (UID: \"6a10c910-c03b-4208-a791-73b4d0210e83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"
Sep 17 18:02:39.809904 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.809885    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a10c910-c03b-4208-a791-73b4d0210e83-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm\" (UID: \"6a10c910-c03b-4208-a791-73b4d0210e83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"
Sep 17 18:02:39.818783 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.818763    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvcvh\" (UniqueName: \"kubernetes.io/projected/6a10c910-c03b-4208-a791-73b4d0210e83-kube-api-access-kvcvh\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm\" (UID: \"6a10c910-c03b-4208-a791-73b4d0210e83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"
Sep 17 18:02:39.930752 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:39.930684    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"
Sep 17 18:02:40.207116 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:40.207085    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"]
Sep 17 18:02:40.209418 ip-10-0-142-4 kubenswrapper[2574]: W0917 18:02:40.209386    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a10c910_c03b_4208_a791_73b4d0210e83.slice/crio-63e4ab10b03089f6e1aba323befd772334faa676051819d7dacb53eb47f46935 WatchSource:0}: Error finding container 63e4ab10b03089f6e1aba323befd772334faa676051819d7dacb53eb47f46935: Status 404 returned error can't find the container with id 63e4ab10b03089f6e1aba323befd772334faa676051819d7dacb53eb47f46935
Sep 17 18:02:40.912917 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:40.912873    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm" event={"ID":"6a10c910-c03b-4208-a791-73b4d0210e83","Type":"ContainerStarted","Data":"4a78e08465687ae167a91157fbd8c7389738a4fd0bf5bc3411d4b9d40812a752"}
Sep 17 18:02:40.913184 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:40.912961    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm" event={"ID":"6a10c910-c03b-4208-a791-73b4d0210e83","Type":"ContainerStarted","Data":"63e4ab10b03089f6e1aba323befd772334faa676051819d7dacb53eb47f46935"}
Sep 17 18:02:44.931108 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:44.931068    2574 generic.go:356] "Generic (PLEG): container finished" podID="6a10c910-c03b-4208-a791-73b4d0210e83" containerID="4a78e08465687ae167a91157fbd8c7389738a4fd0bf5bc3411d4b9d40812a752" exitCode=0
Sep 17 18:02:44.931483 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:44.931120    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm" event={"ID":"6a10c910-c03b-4208-a791-73b4d0210e83","Type":"ContainerDied","Data":"4a78e08465687ae167a91157fbd8c7389738a4fd0bf5bc3411d4b9d40812a752"}
Sep 17 18:02:45.937115 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:45.937084    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm" event={"ID":"6a10c910-c03b-4208-a791-73b4d0210e83","Type":"ContainerStarted","Data":"2869a78c1b56f99cffa2139aaca481534a5ab03b1f216fb50d1c9accdab9cd54"}
Sep 17 18:02:45.937496 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:45.937144    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm" event={"ID":"6a10c910-c03b-4208-a791-73b4d0210e83","Type":"ContainerStarted","Data":"97502b3b2ee20f7efdb984fbc11d07c8adc435798931b9e6d035ce1252faef04"}
Sep 17 18:02:45.937496 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:45.937371    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"
Sep 17 18:02:45.961972 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:45.961924    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm" podStartSLOduration=3.195123846 podStartE2EDuration="6.961900535s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.766776689s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:02:45.960233313 +0000 UTC m=+1741.289332938" watchObservedRunningTime="2026-09-17 18:02:45.961900535 +0000 UTC m=+1741.291000158"
Sep 17 18:02:46.941026 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:46.940991    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"
Sep 17 18:02:46.942228 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:46.942200    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm" podUID="6a10c910-c03b-4208-a791-73b4d0210e83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:02:47.944734 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:47.944693    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm" podUID="6a10c910-c03b-4208-a791-73b4d0210e83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:02:52.949498 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:52.949470    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"
Sep 17 18:02:52.949961 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:02:52.949931    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm" podUID="6a10c910-c03b-4208-a791-73b4d0210e83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:03:02.950675 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:03:02.950631    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm" podUID="6a10c910-c03b-4208-a791-73b4d0210e83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:03:12.950883 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:03:12.950835    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm" podUID="6a10c910-c03b-4208-a791-73b4d0210e83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:03:22.950545 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:03:22.950498    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm" podUID="6a10c910-c03b-4208-a791-73b4d0210e83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:03:32.950036 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:03:32.949996    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm" podUID="6a10c910-c03b-4208-a791-73b4d0210e83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:03:42.950023 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:03:42.949918    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm" podUID="6a10c910-c03b-4208-a791-73b4d0210e83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:03:52.950966 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:03:52.950923    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm" podUID="6a10c910-c03b-4208-a791-73b4d0210e83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:04:02.951169 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:02.951139    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"
Sep 17 18:04:09.855843 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:09.855806    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"]
Sep 17 18:04:09.856319 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:09.856107    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm" podUID="6a10c910-c03b-4208-a791-73b4d0210e83" containerName="kserve-container" containerID="cri-o://97502b3b2ee20f7efdb984fbc11d07c8adc435798931b9e6d035ce1252faef04" gracePeriod=30
Sep 17 18:04:09.856319 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:09.856166    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm" podUID="6a10c910-c03b-4208-a791-73b4d0210e83" containerName="kube-rbac-proxy" containerID="cri-o://2869a78c1b56f99cffa2139aaca481534a5ab03b1f216fb50d1c9accdab9cd54" gracePeriod=30
Sep 17 18:04:09.927189 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:09.927160    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:10.260437 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:10.260353    2574 generic.go:356] "Generic (PLEG): container finished" podID="6a10c910-c03b-4208-a791-73b4d0210e83" containerID="2869a78c1b56f99cffa2139aaca481534a5ab03b1f216fb50d1c9accdab9cd54" exitCode=2
Sep 17 18:04:10.260583 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:10.260430    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm" event={"ID":"6a10c910-c03b-4208-a791-73b4d0210e83","Type":"ContainerDied","Data":"2869a78c1b56f99cffa2139aaca481534a5ab03b1f216fb50d1c9accdab9cd54"}
Sep 17 18:04:12.945501 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:12.945462    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm" podUID="6a10c910-c03b-4208-a791-73b4d0210e83" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 18:04:12.950810 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:12.950783    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm" podUID="6a10c910-c03b-4208-a791-73b4d0210e83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:04:14.282956 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:14.282922    2574 generic.go:356] "Generic (PLEG): container finished" podID="6a10c910-c03b-4208-a791-73b4d0210e83" containerID="97502b3b2ee20f7efdb984fbc11d07c8adc435798931b9e6d035ce1252faef04" exitCode=0
Sep 17 18:04:14.283286 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:14.282967    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm" event={"ID":"6a10c910-c03b-4208-a791-73b4d0210e83","Type":"ContainerDied","Data":"97502b3b2ee20f7efdb984fbc11d07c8adc435798931b9e6d035ce1252faef04"}
Sep 17 18:04:14.296669 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:14.296652    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"
Sep 17 18:04:14.330848 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:14.330821    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a10c910-c03b-4208-a791-73b4d0210e83-proxy-tls\") pod \"6a10c910-c03b-4208-a791-73b4d0210e83\" (UID: \"6a10c910-c03b-4208-a791-73b4d0210e83\") "
Sep 17 18:04:14.330957 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:14.330874    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a10c910-c03b-4208-a791-73b4d0210e83-cabundle-cert\") pod \"6a10c910-c03b-4208-a791-73b4d0210e83\" (UID: \"6a10c910-c03b-4208-a791-73b4d0210e83\") "
Sep 17 18:04:14.330957 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:14.330904    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a10c910-c03b-4208-a791-73b4d0210e83-kserve-provision-location\") pod \"6a10c910-c03b-4208-a791-73b4d0210e83\" (UID: \"6a10c910-c03b-4208-a791-73b4d0210e83\") "
Sep 17 18:04:14.330957 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:14.330952    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a10c910-c03b-4208-a791-73b4d0210e83-kube-rbac-proxy-sar-config\") pod \"6a10c910-c03b-4208-a791-73b4d0210e83\" (UID: \"6a10c910-c03b-4208-a791-73b4d0210e83\") "
Sep 17 18:04:14.331106 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:14.331011    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvcvh\" (UniqueName: \"kubernetes.io/projected/6a10c910-c03b-4208-a791-73b4d0210e83-kube-api-access-kvcvh\") pod \"6a10c910-c03b-4208-a791-73b4d0210e83\" (UID: \"6a10c910-c03b-4208-a791-73b4d0210e83\") "
Sep 17 18:04:14.331313 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:14.331282    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a10c910-c03b-4208-a791-73b4d0210e83-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6a10c910-c03b-4208-a791-73b4d0210e83" (UID: "6a10c910-c03b-4208-a791-73b4d0210e83"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:14.331429 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:14.331316    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a10c910-c03b-4208-a791-73b4d0210e83-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6a10c910-c03b-4208-a791-73b4d0210e83" (UID: "6a10c910-c03b-4208-a791-73b4d0210e83"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:14.331429 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:14.331348    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a10c910-c03b-4208-a791-73b4d0210e83-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6a10c910-c03b-4208-a791-73b4d0210e83" (UID: "6a10c910-c03b-4208-a791-73b4d0210e83"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:14.332872 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:14.332816    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a10c910-c03b-4208-a791-73b4d0210e83-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a10c910-c03b-4208-a791-73b4d0210e83" (UID: "6a10c910-c03b-4208-a791-73b4d0210e83"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:04:14.332991 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:14.332968    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a10c910-c03b-4208-a791-73b4d0210e83-kube-api-access-kvcvh" (OuterVolumeSpecName: "kube-api-access-kvcvh") pod "6a10c910-c03b-4208-a791-73b4d0210e83" (UID: "6a10c910-c03b-4208-a791-73b4d0210e83"). InnerVolumeSpecName "kube-api-access-kvcvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:04:14.432490 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:14.432454    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a10c910-c03b-4208-a791-73b4d0210e83-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 18:04:14.432490 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:14.432487    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvcvh\" (UniqueName: \"kubernetes.io/projected/6a10c910-c03b-4208-a791-73b4d0210e83-kube-api-access-kvcvh\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 18:04:14.432490 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:14.432496    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a10c910-c03b-4208-a791-73b4d0210e83-proxy-tls\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 18:04:14.432693 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:14.432504    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a10c910-c03b-4208-a791-73b4d0210e83-cabundle-cert\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 18:04:14.432693 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:14.432515    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a10c910-c03b-4208-a791-73b4d0210e83-kserve-provision-location\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 18:04:15.289599 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:15.289547    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm" event={"ID":"6a10c910-c03b-4208-a791-73b4d0210e83","Type":"ContainerDied","Data":"63e4ab10b03089f6e1aba323befd772334faa676051819d7dacb53eb47f46935"}
Sep 17 18:04:15.289599 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:15.289578    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"
Sep 17 18:04:15.290170 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:15.289629    2574 scope.go:117] "RemoveContainer" containerID="2869a78c1b56f99cffa2139aaca481534a5ab03b1f216fb50d1c9accdab9cd54"
Sep 17 18:04:15.297635 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:15.297617    2574 scope.go:117] "RemoveContainer" containerID="97502b3b2ee20f7efdb984fbc11d07c8adc435798931b9e6d035ce1252faef04"
Sep 17 18:04:15.304677 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:15.304659    2574 scope.go:117] "RemoveContainer" containerID="4a78e08465687ae167a91157fbd8c7389738a4fd0bf5bc3411d4b9d40812a752"
Sep 17 18:04:15.310712 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:15.310692    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"]
Sep 17 18:04:15.316786 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:15.316765    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-qmjsm"]
Sep 17 18:04:17.116447 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:17.116412    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a10c910-c03b-4208-a791-73b4d0210e83" path="/var/lib/kubelet/pods/6a10c910-c03b-4208-a791-73b4d0210e83/volumes"
Sep 17 18:04:24.115965 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:24.115919    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:24.215284 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:24.215256    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:34.175211 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:34.175178    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:34.428881 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:04:34.428794    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:03.259485 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:05:03.259444    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:19.460284 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:19.460250    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:19.562342 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:19.562310    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:20.261428 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:20.261397    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8hgkd/must-gather-7b99w"]
Sep 17 18:06:20.261695 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:20.261683    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a10c910-c03b-4208-a791-73b4d0210e83" containerName="kserve-container"
Sep 17 18:06:20.261745 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:20.261697    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a10c910-c03b-4208-a791-73b4d0210e83" containerName="kserve-container"
Sep 17 18:06:20.261745 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:20.261705    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a10c910-c03b-4208-a791-73b4d0210e83" containerName="kube-rbac-proxy"
Sep 17 18:06:20.261745 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:20.261710    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a10c910-c03b-4208-a791-73b4d0210e83" containerName="kube-rbac-proxy"
Sep 17 18:06:20.261745 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:20.261743    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a10c910-c03b-4208-a791-73b4d0210e83" containerName="storage-initializer"
Sep 17 18:06:20.261867 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:20.261749    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a10c910-c03b-4208-a791-73b4d0210e83" containerName="storage-initializer"
Sep 17 18:06:20.261867 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:20.261795    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a10c910-c03b-4208-a791-73b4d0210e83" containerName="kserve-container"
Sep 17 18:06:20.261867 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:20.261802    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a10c910-c03b-4208-a791-73b4d0210e83" containerName="kube-rbac-proxy"
Sep 17 18:06:20.264652 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:20.264635    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8hgkd/must-gather-7b99w"
Sep 17 18:06:20.271299 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:20.271277    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8hgkd\"/\"kube-root-ca.crt\""
Sep 17 18:06:20.271411 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:20.271300    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8hgkd\"/\"default-dockercfg-nxdrv\""
Sep 17 18:06:20.272115 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:20.272097    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8hgkd\"/\"openshift-service-ca.crt\""
Sep 17 18:06:20.293394 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:20.293369    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8hgkd/must-gather-7b99w"]
Sep 17 18:06:20.307898 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:20.307871    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:20.341692 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:20.341667    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6knz\" (UniqueName: \"kubernetes.io/projected/70bc618d-a525-42e1-8167-237462933cca-kube-api-access-j6knz\") pod \"must-gather-7b99w\" (UID: \"70bc618d-a525-42e1-8167-237462933cca\") " pod="openshift-must-gather-8hgkd/must-gather-7b99w"
Sep 17 18:06:20.341778 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:20.341723    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/70bc618d-a525-42e1-8167-237462933cca-must-gather-output\") pod \"must-gather-7b99w\" (UID: \"70bc618d-a525-42e1-8167-237462933cca\") " pod="openshift-must-gather-8hgkd/must-gather-7b99w"
Sep 17 18:06:20.442274 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:20.442247    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/70bc618d-a525-42e1-8167-237462933cca-must-gather-output\") pod \"must-gather-7b99w\" (UID: \"70bc618d-a525-42e1-8167-237462933cca\") " pod="openshift-must-gather-8hgkd/must-gather-7b99w"
Sep 17 18:06:20.442362 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:20.442295    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6knz\" (UniqueName: \"kubernetes.io/projected/70bc618d-a525-42e1-8167-237462933cca-kube-api-access-j6knz\") pod \"must-gather-7b99w\" (UID: \"70bc618d-a525-42e1-8167-237462933cca\") " pod="openshift-must-gather-8hgkd/must-gather-7b99w"
Sep 17 18:06:20.442641 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:20.442625    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/70bc618d-a525-42e1-8167-237462933cca-must-gather-output\") pod \"must-gather-7b99w\" (UID: \"70bc618d-a525-42e1-8167-237462933cca\") " pod="openshift-must-gather-8hgkd/must-gather-7b99w"
Sep 17 18:06:20.452626 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:20.452610    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:20.456998 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:20.456977    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6knz\" (UniqueName: \"kubernetes.io/projected/70bc618d-a525-42e1-8167-237462933cca-kube-api-access-j6knz\") pod \"must-gather-7b99w\" (UID: \"70bc618d-a525-42e1-8167-237462933cca\") " pod="openshift-must-gather-8hgkd/must-gather-7b99w"
Sep 17 18:06:20.588278 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:20.588247    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8hgkd/must-gather-7b99w"
Sep 17 18:06:20.858325 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:20.858302    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8hgkd/must-gather-7b99w"]
Sep 17 18:06:20.861023 ip-10-0-142-4 kubenswrapper[2574]: W0917 18:06:20.860991    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70bc618d_a525_42e1_8167_237462933cca.slice/crio-d09896a1c10db6df05776b0b1d10c3d349f9dd4f3c377612a001d93263eb8845 WatchSource:0}: Error finding container d09896a1c10db6df05776b0b1d10c3d349f9dd4f3c377612a001d93263eb8845: Status 404 returned error can't find the container with id d09896a1c10db6df05776b0b1d10c3d349f9dd4f3c377612a001d93263eb8845
Sep 17 18:06:20.862732 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:20.862715    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:06:21.773596 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:21.773557    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8hgkd/must-gather-7b99w" event={"ID":"70bc618d-a525-42e1-8167-237462933cca","Type":"ContainerStarted","Data":"d09896a1c10db6df05776b0b1d10c3d349f9dd4f3c377612a001d93263eb8845"}
Sep 17 18:06:27.808871 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:27.808826    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8hgkd/must-gather-7b99w" event={"ID":"70bc618d-a525-42e1-8167-237462933cca","Type":"ContainerStarted","Data":"15d4f56af92388da752d691920fbb7e750d0396c9b6d397c81c64f9a1bd17074"}
Sep 17 18:06:27.809354 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:27.808885    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8hgkd/must-gather-7b99w" event={"ID":"70bc618d-a525-42e1-8167-237462933cca","Type":"ContainerStarted","Data":"7e5d813a5eda88edc84eadef2b71df7b001fb7c79db6fb8b917f6cb9cd7ba368"}
Sep 17 18:06:50.910992 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:50.910953    2574 generic.go:356] "Generic (PLEG): container finished" podID="70bc618d-a525-42e1-8167-237462933cca" containerID="7e5d813a5eda88edc84eadef2b71df7b001fb7c79db6fb8b917f6cb9cd7ba368" exitCode=0
Sep 17 18:06:50.911434 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:50.911029    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8hgkd/must-gather-7b99w" event={"ID":"70bc618d-a525-42e1-8167-237462933cca","Type":"ContainerDied","Data":"7e5d813a5eda88edc84eadef2b71df7b001fb7c79db6fb8b917f6cb9cd7ba368"}
Sep 17 18:06:50.911434 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:50.911349    2574 scope.go:117] "RemoveContainer" containerID="7e5d813a5eda88edc84eadef2b71df7b001fb7c79db6fb8b917f6cb9cd7ba368"
Sep 17 18:06:51.174952 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:51.174874    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8hgkd_must-gather-7b99w_70bc618d-a525-42e1-8167-237462933cca/gather/0.log"
Sep 17 18:06:51.657170 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:51.657130    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:51.795545 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:51.795515    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wlfjm/must-gather-4jvr2"]
Sep 17 18:06:51.803929 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:51.803903    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wlfjm/must-gather-4jvr2"
Sep 17 18:06:51.806527 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:51.806505    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wlfjm\"/\"openshift-service-ca.crt\""
Sep 17 18:06:51.807387 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:51.807369    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wlfjm\"/\"default-dockercfg-v2w9r\""
Sep 17 18:06:51.807487 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:51.807409    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wlfjm\"/\"kube-root-ca.crt\""
Sep 17 18:06:51.810518 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:51.810496    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wlfjm/must-gather-4jvr2"]
Sep 17 18:06:51.810813 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:51.810789    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:51.911874 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:51.911794    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/31b5e2fb-3122-4cde-80dd-411587a17b99-must-gather-output\") pod \"must-gather-4jvr2\" (UID: \"31b5e2fb-3122-4cde-80dd-411587a17b99\") " pod="openshift-must-gather-wlfjm/must-gather-4jvr2"
Sep 17 18:06:51.911874 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:51.911837    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm9mz\" (UniqueName: \"kubernetes.io/projected/31b5e2fb-3122-4cde-80dd-411587a17b99-kube-api-access-jm9mz\") pod \"must-gather-4jvr2\" (UID: \"31b5e2fb-3122-4cde-80dd-411587a17b99\") " pod="openshift-must-gather-wlfjm/must-gather-4jvr2"
Sep 17 18:06:51.940591 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:51.940557    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:52.013218 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:52.013191    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm9mz\" (UniqueName: \"kubernetes.io/projected/31b5e2fb-3122-4cde-80dd-411587a17b99-kube-api-access-jm9mz\") pod \"must-gather-4jvr2\" (UID: \"31b5e2fb-3122-4cde-80dd-411587a17b99\") " pod="openshift-must-gather-wlfjm/must-gather-4jvr2"
Sep 17 18:06:52.013314 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:52.013257    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/31b5e2fb-3122-4cde-80dd-411587a17b99-must-gather-output\") pod \"must-gather-4jvr2\" (UID: \"31b5e2fb-3122-4cde-80dd-411587a17b99\") " pod="openshift-must-gather-wlfjm/must-gather-4jvr2"
Sep 17 18:06:52.013529 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:52.013513    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/31b5e2fb-3122-4cde-80dd-411587a17b99-must-gather-output\") pod \"must-gather-4jvr2\" (UID: \"31b5e2fb-3122-4cde-80dd-411587a17b99\") " pod="openshift-must-gather-wlfjm/must-gather-4jvr2"
Sep 17 18:06:52.024391 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:52.024366    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm9mz\" (UniqueName: \"kubernetes.io/projected/31b5e2fb-3122-4cde-80dd-411587a17b99-kube-api-access-jm9mz\") pod \"must-gather-4jvr2\" (UID: \"31b5e2fb-3122-4cde-80dd-411587a17b99\") " pod="openshift-must-gather-wlfjm/must-gather-4jvr2"
Sep 17 18:06:52.113677 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:52.113653    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wlfjm/must-gather-4jvr2"
Sep 17 18:06:52.385247 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:52.385213    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wlfjm/must-gather-4jvr2"]
Sep 17 18:06:52.388144 ip-10-0-142-4 kubenswrapper[2574]: W0917 18:06:52.388118    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31b5e2fb_3122_4cde_80dd_411587a17b99.slice/crio-92ac41beac96a9d2223f67d229937c9d3f379aee3891aab21b7ebd7031edba52 WatchSource:0}: Error finding container 92ac41beac96a9d2223f67d229937c9d3f379aee3891aab21b7ebd7031edba52: Status 404 returned error can't find the container with id 92ac41beac96a9d2223f67d229937c9d3f379aee3891aab21b7ebd7031edba52
Sep 17 18:06:52.919824 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:52.919790    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wlfjm/must-gather-4jvr2" event={"ID":"31b5e2fb-3122-4cde-80dd-411587a17b99","Type":"ContainerStarted","Data":"92ac41beac96a9d2223f67d229937c9d3f379aee3891aab21b7ebd7031edba52"}
Sep 17 18:06:53.928699 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:53.928110    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wlfjm/must-gather-4jvr2" event={"ID":"31b5e2fb-3122-4cde-80dd-411587a17b99","Type":"ContainerStarted","Data":"70f7c6d6a8d4f7a567ee47b5d77bfb12eceb9c42519aea4a3b7614d3d55fe31c"}
Sep 17 18:06:53.928699 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:53.928172    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wlfjm/must-gather-4jvr2" event={"ID":"31b5e2fb-3122-4cde-80dd-411587a17b99","Type":"ContainerStarted","Data":"4f6a077be73fb0700db82bb8388feb8b1c7f4a4841d7b7ca5545b5d3fd791433"}
Sep 17 18:06:53.948436 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:53.948132    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wlfjm/must-gather-4jvr2" podStartSLOduration=2.172652481 podStartE2EDuration="2.948109245s" totalImagesPullingTime="775.456764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:06:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:06:53.946437917 +0000 UTC m=+1989.275537542" watchObservedRunningTime="2026-09-17 18:06:53.948109245 +0000 UTC m=+1989.277208870"
Sep 17 18:06:54.306677 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:54.306594    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:54.843766 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:54.843733    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-n8p54_c58d2a73-a185-4e79-bbba-0b547390e9c7/global-pull-secret-syncer/0.log"
Sep 17 18:06:54.986096 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:54.986065    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log"
Sep 17 18:06:55.154830 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:55.154733    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-4.ec2.internal_3771e4c220b5a223d0f484f25cdc29e8/haproxy/0.log"
Sep 17 18:06:56.619471 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:56.619423    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-8hgkd/must-gather-7b99w"]
Sep 17 18:06:56.619930 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:56.619657    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-8hgkd/must-gather-7b99w" podUID="70bc618d-a525-42e1-8167-237462933cca" containerName="copy" containerID="cri-o://15d4f56af92388da752d691920fbb7e750d0396c9b6d397c81c64f9a1bd17074" gracePeriod=2
Sep 17 18:06:56.622844 ip-10-0-142-4 kubenswrapper[2574]: E0917 18:06:56.622803    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-7b99w\" is forbidden: User \"system:node:ip-10-0-142-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-8hgkd\": no relationship found between node 'ip-10-0-142-4.ec2.internal' and this object" podUID="70bc618d-a525-42e1-8167-237462933cca" pod="openshift-must-gather-8hgkd/must-gather-7b99w"
Sep 17 18:06:56.624352 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:56.624323    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-8hgkd/must-gather-7b99w"]
Sep 17 18:06:56.948893 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:56.948741    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8hgkd_must-gather-7b99w_70bc618d-a525-42e1-8167-237462933cca/copy/0.log"
Sep 17 18:06:56.949592 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:56.949479    2574 generic.go:356] "Generic (PLEG): container finished" podID="70bc618d-a525-42e1-8167-237462933cca" containerID="15d4f56af92388da752d691920fbb7e750d0396c9b6d397c81c64f9a1bd17074" exitCode=143
Sep 17 18:06:57.013383 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:57.009190    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8hgkd_must-gather-7b99w_70bc618d-a525-42e1-8167-237462933cca/copy/0.log"
Sep 17 18:06:57.014905 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:57.014884    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8hgkd/must-gather-7b99w"
Sep 17 18:06:57.021447 ip-10-0-142-4 kubenswrapper[2574]: E0917 18:06:57.020451    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-7b99w\" is forbidden: User \"system:node:ip-10-0-142-4.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-8hgkd\": no relationship found between node 'ip-10-0-142-4.ec2.internal' and this object" podUID="70bc618d-a525-42e1-8167-237462933cca" pod="openshift-must-gather-8hgkd/must-gather-7b99w"
Sep 17 18:06:57.061500 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:57.061232    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6knz\" (UniqueName: \"kubernetes.io/projected/70bc618d-a525-42e1-8167-237462933cca-kube-api-access-j6knz\") pod \"70bc618d-a525-42e1-8167-237462933cca\" (UID: \"70bc618d-a525-42e1-8167-237462933cca\") "
Sep 17 18:06:57.061500 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:57.061329    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/70bc618d-a525-42e1-8167-237462933cca-must-gather-output\") pod \"70bc618d-a525-42e1-8167-237462933cca\" (UID: \"70bc618d-a525-42e1-8167-237462933cca\") "
Sep 17 18:06:57.064628 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:57.064598    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bc618d-a525-42e1-8167-237462933cca-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "70bc618d-a525-42e1-8167-237462933cca" (UID: "70bc618d-a525-42e1-8167-237462933cca"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:57.071360 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:57.071328    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70bc618d-a525-42e1-8167-237462933cca-kube-api-access-j6knz" (OuterVolumeSpecName: "kube-api-access-j6knz") pod "70bc618d-a525-42e1-8167-237462933cca" (UID: "70bc618d-a525-42e1-8167-237462933cca"). InnerVolumeSpecName "kube-api-access-j6knz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:06:57.121235 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:57.120684    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70bc618d-a525-42e1-8167-237462933cca" path="/var/lib/kubelet/pods/70bc618d-a525-42e1-8167-237462933cca/volumes"
Sep 17 18:06:57.162464 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:57.162422    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6knz\" (UniqueName: \"kubernetes.io/projected/70bc618d-a525-42e1-8167-237462933cca-kube-api-access-j6knz\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 18:06:57.162679 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:57.162663    2574 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/70bc618d-a525-42e1-8167-237462933cca-must-gather-output\") on node \"ip-10-0-142-4.ec2.internal\" DevicePath \"\""
Sep 17 18:06:57.957034 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:57.956950    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8hgkd_must-gather-7b99w_70bc618d-a525-42e1-8167-237462933cca/copy/0.log"
Sep 17 18:06:57.957639 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:57.957545    2574 scope.go:117] "RemoveContainer" containerID="15d4f56af92388da752d691920fbb7e750d0396c9b6d397c81c64f9a1bd17074"
Sep 17 18:06:57.960987 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:57.957724    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8hgkd/must-gather-7b99w"
Sep 17 18:06:57.974398 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:57.974369    2574 scope.go:117] "RemoveContainer" containerID="7e5d813a5eda88edc84eadef2b71df7b001fb7c79db6fb8b917f6cb9cd7ba368"
Sep 17 18:06:58.382017 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:58.381976    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:58.471067 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:58.471015    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89m26_8146ff25-6272-4f4f-bc26-f165405e2610/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:59.365538 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:59.365487    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-nxp84_7027d734-65db-4433-8b0e-387cf90fa1b8/cluster-monitoring-operator/0.log"
Sep 17 18:06:59.397971 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:59.397924    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-95v8k_d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea/kube-state-metrics/0.log"
Sep 17 18:06:59.424410 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:59.424381    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-95v8k_d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea/kube-rbac-proxy-main/0.log"
Sep 17 18:06:59.458396 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:59.458366    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-95v8k_d0f5d75e-3ef8-4fe7-aa57-db82c43f48ea/kube-rbac-proxy-self/0.log"
Sep 17 18:06:59.670420 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:59.670339    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-js28h_cdaea3b1-196c-426f-8823-83e199793607/node-exporter/0.log"
Sep 17 18:06:59.702981 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:59.702935    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-js28h_cdaea3b1-196c-426f-8823-83e199793607/kube-rbac-proxy/0.log"
Sep 17 18:06:59.736700 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:06:59.736672    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-js28h_cdaea3b1-196c-426f-8823-83e199793607/init-textfile/0.log"
Sep 17 18:07:00.014740 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:00.014680    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0067e4a3-813f-4380-89cf-8bd5ca501766/prometheus/0.log"
Sep 17 18:07:00.041779 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:00.041746    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0067e4a3-813f-4380-89cf-8bd5ca501766/config-reloader/0.log"
Sep 17 18:07:00.086064 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:00.086019    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0067e4a3-813f-4380-89cf-8bd5ca501766/thanos-sidecar/0.log"
Sep 17 18:07:00.117700 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:00.117672    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0067e4a3-813f-4380-89cf-8bd5ca501766/kube-rbac-proxy-web/0.log"
Sep 17 18:07:00.144558 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:00.144527    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0067e4a3-813f-4380-89cf-8bd5ca501766/kube-rbac-proxy/0.log"
Sep 17 18:07:00.171794 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:00.171762    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0067e4a3-813f-4380-89cf-8bd5ca501766/kube-rbac-proxy-thanos/0.log"
Sep 17 18:07:00.212248 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:00.212170    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0067e4a3-813f-4380-89cf-8bd5ca501766/init-config-reloader/0.log"
Sep 17 18:07:00.253493 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:00.253454    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-4q5r7_c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9/prometheus-operator/0.log"
Sep 17 18:07:00.277271 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:00.277239    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-4q5r7_c67049c7-bfd5-4e2f-b68a-03e00ac9a5f9/kube-rbac-proxy/0.log"
Sep 17 18:07:00.309523 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:00.309489    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-r9x5k_5f71b6c1-e6be-4c65-a5de-e4a2a9e728e7/prometheus-operator-admission-webhook/0.log"
Sep 17 18:07:02.966827 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:02.966793    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-wt58c_efe4cd38-a36c-445f-aeb9-0f7167b306ec/download-server/0.log"
Sep 17 18:07:02.970758 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:02.970731    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wlfjm/perf-node-gather-daemonset-vln45"]
Sep 17 18:07:02.971139 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:02.971119    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70bc618d-a525-42e1-8167-237462933cca" containerName="gather"
Sep 17 18:07:02.971229 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:02.971141    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="70bc618d-a525-42e1-8167-237462933cca" containerName="gather"
Sep 17 18:07:02.971229 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:02.971166    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70bc618d-a525-42e1-8167-237462933cca" containerName="copy"
Sep 17 18:07:02.971229 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:02.971175    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="70bc618d-a525-42e1-8167-237462933cca" containerName="copy"
Sep 17 18:07:02.971347 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:02.971248    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="70bc618d-a525-42e1-8167-237462933cca" containerName="gather"
Sep 17 18:07:02.971347 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:02.971263    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="70bc618d-a525-42e1-8167-237462933cca" containerName="copy"
Sep 17 18:07:02.980439 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:02.980410    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-vln45"
Sep 17 18:07:02.985194 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:02.985170    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wlfjm/perf-node-gather-daemonset-vln45"]
Sep 17 18:07:03.023279 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:03.023244    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e88f9e8c-68f8-4f91-b439-77846c2bf1de-proc\") pod \"perf-node-gather-daemonset-vln45\" (UID: \"e88f9e8c-68f8-4f91-b439-77846c2bf1de\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-vln45"
Sep 17 18:07:03.023471 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:03.023306    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e88f9e8c-68f8-4f91-b439-77846c2bf1de-lib-modules\") pod \"perf-node-gather-daemonset-vln45\" (UID: \"e88f9e8c-68f8-4f91-b439-77846c2bf1de\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-vln45"
Sep 17 18:07:03.023471 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:03.023413    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e88f9e8c-68f8-4f91-b439-77846c2bf1de-podres\") pod \"perf-node-gather-daemonset-vln45\" (UID: \"e88f9e8c-68f8-4f91-b439-77846c2bf1de\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-vln45"
Sep 17 18:07:03.023471 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:03.023466    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kml76\" (UniqueName: \"kubernetes.io/projected/e88f9e8c-68f8-4f91-b439-77846c2bf1de-kube-api-access-kml76\") pod \"perf-node-gather-daemonset-vln45\" (UID: \"e88f9e8c-68f8-4f91-b439-77846c2bf1de\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-vln45"
Sep 17 18:07:03.023660 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:03.023537    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e88f9e8c-68f8-4f91-b439-77846c2bf1de-sys\") pod \"perf-node-gather-daemonset-vln45\" (UID: \"e88f9e8c-68f8-4f91-b439-77846c2bf1de\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-vln45"
Sep 17 18:07:03.124894 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:03.124852    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e88f9e8c-68f8-4f91-b439-77846c2bf1de-proc\") pod \"perf-node-gather-daemonset-vln45\" (UID: \"e88f9e8c-68f8-4f91-b439-77846c2bf1de\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-vln45"
Sep 17 18:07:03.125094 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:03.124921    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e88f9e8c-68f8-4f91-b439-77846c2bf1de-lib-modules\") pod \"perf-node-gather-daemonset-vln45\" (UID: \"e88f9e8c-68f8-4f91-b439-77846c2bf1de\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-vln45"
Sep 17 18:07:03.125094 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:03.124982    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e88f9e8c-68f8-4f91-b439-77846c2bf1de-podres\") pod \"perf-node-gather-daemonset-vln45\" (UID: \"e88f9e8c-68f8-4f91-b439-77846c2bf1de\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-vln45"
Sep 17 18:07:03.125094 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:03.124984    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e88f9e8c-68f8-4f91-b439-77846c2bf1de-proc\") pod \"perf-node-gather-daemonset-vln45\" (UID: \"e88f9e8c-68f8-4f91-b439-77846c2bf1de\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-vln45"
Sep 17 18:07:03.125272 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:03.125125    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e88f9e8c-68f8-4f91-b439-77846c2bf1de-lib-modules\") pod \"perf-node-gather-daemonset-vln45\" (UID: \"e88f9e8c-68f8-4f91-b439-77846c2bf1de\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-vln45"
Sep 17 18:07:03.125272 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:03.125160    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e88f9e8c-68f8-4f91-b439-77846c2bf1de-podres\") pod \"perf-node-gather-daemonset-vln45\" (UID: \"e88f9e8c-68f8-4f91-b439-77846c2bf1de\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-vln45"
Sep 17 18:07:03.125272 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:03.125181    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kml76\" (UniqueName: \"kubernetes.io/projected/e88f9e8c-68f8-4f91-b439-77846c2bf1de-kube-api-access-kml76\") pod \"perf-node-gather-daemonset-vln45\" (UID: \"e88f9e8c-68f8-4f91-b439-77846c2bf1de\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-vln45"
Sep 17 18:07:03.125272 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:03.125255    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e88f9e8c-68f8-4f91-b439-77846c2bf1de-sys\") pod \"perf-node-gather-daemonset-vln45\" (UID: \"e88f9e8c-68f8-4f91-b439-77846c2bf1de\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-vln45"
Sep 17 18:07:03.125460 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:03.125342    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e88f9e8c-68f8-4f91-b439-77846c2bf1de-sys\") pod \"perf-node-gather-daemonset-vln45\" (UID: \"e88f9e8c-68f8-4f91-b439-77846c2bf1de\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-vln45"
Sep 17 18:07:03.133728 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:03.133703    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kml76\" (UniqueName: \"kubernetes.io/projected/e88f9e8c-68f8-4f91-b439-77846c2bf1de-kube-api-access-kml76\") pod \"perf-node-gather-daemonset-vln45\" (UID: \"e88f9e8c-68f8-4f91-b439-77846c2bf1de\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-vln45"
Sep 17 18:07:03.293666 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:03.293572    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-vln45"
Sep 17 18:07:03.593434 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:03.593405    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wlfjm/perf-node-gather-daemonset-vln45"]
Sep 17 18:07:03.594410 ip-10-0-142-4 kubenswrapper[2574]: W0917 18:07:03.594382    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode88f9e8c_68f8_4f91_b439_77846c2bf1de.slice/crio-16132f5e4f5a0b91a10592494b9e43e73c511f5988bb7f2667bd890b6e59dfc8 WatchSource:0}: Error finding container 16132f5e4f5a0b91a10592494b9e43e73c511f5988bb7f2667bd890b6e59dfc8: Status 404 returned error can't find the container with id 16132f5e4f5a0b91a10592494b9e43e73c511f5988bb7f2667bd890b6e59dfc8
Sep 17 18:07:03.996266 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:03.996190    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-vln45" event={"ID":"e88f9e8c-68f8-4f91-b439-77846c2bf1de","Type":"ContainerStarted","Data":"b068ea843851c1c47338295a5f4c2a54e4b6a9f4313dbc63f64ceacf376d6cbe"}
Sep 17 18:07:03.996266 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:03.996244    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-vln45" event={"ID":"e88f9e8c-68f8-4f91-b439-77846c2bf1de","Type":"ContainerStarted","Data":"16132f5e4f5a0b91a10592494b9e43e73c511f5988bb7f2667bd890b6e59dfc8"}
Sep 17 18:07:03.996648 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:03.996308    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-vln45"
Sep 17 18:07:04.016281 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:04.016218    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-vln45" podStartSLOduration=2.016197533 podStartE2EDuration="2.016197533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:07:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:07:04.014204365 +0000 UTC m=+1999.343303990" watchObservedRunningTime="2026-09-17 18:07:04.016197533 +0000 UTC m=+1999.345297159"
Sep 17 18:07:04.838878 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:04.838851    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qpdzb_7656e584-bfe6-4f58-bdac-fe20192d5a44/dns/0.log"
Sep 17 18:07:04.879733 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:04.879700    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qpdzb_7656e584-bfe6-4f58-bdac-fe20192d5a44/kube-rbac-proxy/0.log"
Sep 17 18:07:05.009874 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:05.009846    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jvdsb_b5dd23d0-8b89-453c-99ee-dfd8a18c683c/dns-node-resolver/0.log"
Sep 17 18:07:05.631269 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:05.631235    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cbnqr_a811ebbe-f6a5-4bac-a036-e3d2f439f394/node-ca/0.log"
Sep 17 18:07:06.616594 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:06.616563    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-67f77bb47b-mlz49_999e96ab-504d-4201-9988-df670a74ee7e/router/0.log"
Sep 17 18:07:07.155806 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:07.155776    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lqq6s_ac78b330-46cb-49bd-970a-f7eaf7c4afb4/serve-healthcheck-canary/0.log"
Sep 17 18:07:07.664900 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:07.664871    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-dqbxl_496b2135-6189-4a33-b138-112cfd1a57d2/insights-operator/0.log"
Sep 17 18:07:07.665627 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:07.665603    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-dqbxl_496b2135-6189-4a33-b138-112cfd1a57d2/insights-operator/1.log"
Sep 17 18:07:07.707732 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:07.707712    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9jwbf_7137d714-8bad-4b65-ad47-49d4101a4237/kube-rbac-proxy/0.log"
Sep 17 18:07:07.760978 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:07.760959    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9jwbf_7137d714-8bad-4b65-ad47-49d4101a4237/exporter/0.log"
Sep 17 18:07:07.816380 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:07.816355    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9jwbf_7137d714-8bad-4b65-ad47-49d4101a4237/extractor/0.log"
Sep 17 18:07:10.011147 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:10.011119    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-vln45"
Sep 17 18:07:10.866852 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:10.866816    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-dnqsq_65f40148-a803-4220-8471-527fb92b57a0/s3-init/0.log"
Sep 17 18:07:10.890933 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:10.890909    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-dnqsq_65f40148-a803-4220-8471-527fb92b57a0/download-hf-model/0.log"
Sep 17 18:07:16.174684 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:16.174648    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-7f8r6_c2cebd92-d4c4-49d3-9fae-dadee9638035/migrator/0.log"
Sep 17 18:07:16.211205 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:16.211178    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-7f8r6_c2cebd92-d4c4-49d3-9fae-dadee9638035/graceful-termination/0.log"
Sep 17 18:07:17.759312 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:17.759271    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-96ncc_479fedb1-cbab-419a-a1b0-9c2a4c5eda27/kube-multus/0.log"
Sep 17 18:07:17.967676 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:17.967641    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qxvb8_0774cdca-c8f7-4cae-9037-633a199e0ff3/kube-multus-additional-cni-plugins/0.log"
Sep 17 18:07:17.993385 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:17.993357    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qxvb8_0774cdca-c8f7-4cae-9037-633a199e0ff3/egress-router-binary-copy/0.log"
Sep 17 18:07:18.019939 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:18.019866    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qxvb8_0774cdca-c8f7-4cae-9037-633a199e0ff3/cni-plugins/0.log"
Sep 17 18:07:18.049578 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:18.049529    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qxvb8_0774cdca-c8f7-4cae-9037-633a199e0ff3/bond-cni-plugin/0.log"
Sep 17 18:07:18.077800 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:18.077774    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qxvb8_0774cdca-c8f7-4cae-9037-633a199e0ff3/routeoverride-cni/0.log"
Sep 17 18:07:18.103365 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:18.103347    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qxvb8_0774cdca-c8f7-4cae-9037-633a199e0ff3/whereabouts-cni-bincopy/0.log"
Sep 17 18:07:18.127634 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:18.127612    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qxvb8_0774cdca-c8f7-4cae-9037-633a199e0ff3/whereabouts-cni/0.log"
Sep 17 18:07:18.517296 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:18.517267    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kcglc_c08424bd-0295-4dcb-9c57-3d5a5a6ba21d/network-metrics-daemon/0.log"
Sep 17 18:07:18.549080 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:18.549025    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kcglc_c08424bd-0295-4dcb-9c57-3d5a5a6ba21d/kube-rbac-proxy/0.log"
Sep 17 18:07:19.651449 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:19.651403    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c528h_9b120543-c1f5-49a0-9b46-8494758bc055/ovn-controller/0.log"
Sep 17 18:07:19.671388 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:19.671358    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c528h_9b120543-c1f5-49a0-9b46-8494758bc055/ovn-acl-logging/0.log"
Sep 17 18:07:19.703021 ip-10-0-142-4 kubenswrapper[2574]: I0917 18:07:19.703000    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c528h_9b120543-c1f5-49a0-9b46-8494758bc055/kube-rbac-proxy-node/0.log"