Sep 18 16:15:52.472560 ip-10-0-135-110 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 16:15:55.870910 ip-10-0-135-110 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:15:55.870910 ip-10-0-135-110 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 16:15:55.870910 ip-10-0-135-110 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:15:55.870910 ip-10-0-135-110 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 16:15:55.870910 ip-10-0-135-110 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:15:55.871646 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.870974    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 16:15:55.875684 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875657    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:15:55.875684 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875680    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:15:55.875684 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875686    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:15:55.875684 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875691    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:15:55.875944 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875695    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:15:55.875944 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875700    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:15:55.875944 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875704    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:15:55.875944 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875708    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:15:55.875944 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875711    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:15:55.875944 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875715    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:15:55.875944 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875719    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:15:55.875944 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875723    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:15:55.875944 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875726    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:15:55.875944 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875730    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:15:55.875944 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875733    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:15:55.875944 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875737    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:15:55.875944 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875742    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:15:55.875944 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875745    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:15:55.875944 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875749    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:15:55.875944 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875753    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:15:55.875944 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875757    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:15:55.875944 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875761    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:15:55.875944 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875765    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:15:55.876800 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875769    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:15:55.876800 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875773    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:15:55.876800 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875777    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:15:55.876800 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875782    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:15:55.876800 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875787    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:15:55.876800 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875790    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:15:55.876800 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875794    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:15:55.876800 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875798    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:15:55.876800 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875812    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:15:55.876800 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875816    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:15:55.876800 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875820    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:15:55.876800 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875825    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:15:55.876800 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875831    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:15:55.876800 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875836    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:15:55.876800 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875840    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:15:55.876800 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875845    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:15:55.876800 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875849    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:15:55.876800 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875854    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:15:55.876800 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875858    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:15:55.876800 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875862    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:15:55.877658 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875866    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:15:55.877658 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875871    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:15:55.877658 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875877    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:15:55.877658 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875884    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:15:55.877658 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875889    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:15:55.877658 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875893    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:15:55.877658 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875898    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:15:55.877658 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875903    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:15:55.877658 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875908    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:15:55.877658 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875915    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:15:55.877658 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875921    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:15:55.877658 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875926    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:15:55.877658 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875932    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:15:55.877658 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875937    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:15:55.877658 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875942    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:15:55.877658 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875946    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:15:55.877658 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875950    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:15:55.877658 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875954    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:15:55.877658 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875960    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:15:55.878455 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875965    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:15:55.878455 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875970    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:15:55.878455 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875974    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:15:55.878455 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875979    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:15:55.878455 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875984    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:15:55.878455 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875988    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:15:55.878455 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875993    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:15:55.878455 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.875997    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:15:55.878455 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876002    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:15:55.878455 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876006    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:15:55.878455 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876010    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:15:55.878455 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876015    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:15:55.878455 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876019    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:15:55.878455 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876024    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:15:55.878455 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876028    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:15:55.878455 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876031    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:15:55.878455 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876036    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:15:55.878455 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876040    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:15:55.878455 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876045    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:15:55.878455 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876049    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:15:55.879324 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876053    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:15:55.879324 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876059    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:15:55.879324 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876063    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:15:55.879324 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876068    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:15:55.879324 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876072    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:15:55.879324 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876076    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:15:55.879324 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876080    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:15:55.879324 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876084    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:15:55.879324 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876088    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:15:55.879324 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876092    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:15:55.879324 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876096    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:15:55.879324 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876101    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:15:55.879324 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876106    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:15:55.879324 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876110    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:15:55.879324 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876114    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:15:55.879324 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876118    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:15:55.879324 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.876122    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:15:55.879324 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877295    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:15:55.879324 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877324    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:15:55.879324 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877329    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:15:55.880173 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877333    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:15:55.880173 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877338    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:15:55.880173 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877343    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:15:55.880173 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877347    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:15:55.880173 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877351    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:15:55.880173 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877355    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:15:55.880173 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877360    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:15:55.880173 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877364    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:15:55.880173 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877369    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:15:55.880173 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877374    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:15:55.880173 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877378    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:15:55.880173 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877383    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:15:55.880173 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877387    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:15:55.880173 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877391    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:15:55.880173 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877396    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:15:55.880173 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877401    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:15:55.880173 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877405    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:15:55.880173 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877409    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:15:55.880173 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877414    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:15:55.880173 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877418    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:15:55.880862 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877422    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:15:55.880862 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877426    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:15:55.880862 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877430    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:15:55.880862 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877435    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:15:55.880862 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877439    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:15:55.880862 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877443    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:15:55.880862 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877448    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:15:55.880862 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877452    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:15:55.880862 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877456    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:15:55.880862 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877460    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:15:55.880862 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877466    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:15:55.880862 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877471    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:15:55.880862 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877478    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:15:55.880862 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877484    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:15:55.880862 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877488    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:15:55.880862 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877493    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:15:55.880862 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877497    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:15:55.880862 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877501    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:15:55.880862 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877505    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:15:55.880862 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877510    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:15:55.881482 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877514    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:15:55.881482 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877518    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:15:55.881482 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877522    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:15:55.881482 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877526    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:15:55.881482 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877530    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:15:55.881482 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877534    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:15:55.881482 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877539    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:15:55.881482 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877545    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:15:55.881482 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877550    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:15:55.881482 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877554    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:15:55.881482 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877558    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:15:55.881482 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877562    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:15:55.881482 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877567    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:15:55.881482 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877571    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:15:55.881482 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877575    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:15:55.881482 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877579    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:15:55.881482 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877583    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:15:55.881482 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877588    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:15:55.881482 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877592    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:15:55.881482 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877596    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:15:55.882005 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877600    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:15:55.882005 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877604    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:15:55.882005 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877608    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:15:55.882005 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877613    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:15:55.882005 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877617    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:15:55.882005 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877622    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:15:55.882005 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877626    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:15:55.882005 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877630    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:15:55.882005 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877635    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:15:55.882005 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877639    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:15:55.882005 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877644    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:15:55.882005 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877648    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:15:55.882005 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877652    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:15:55.882005 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877655    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:15:55.882005 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877659    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:15:55.882005 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877663    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:15:55.882005 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877668    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:15:55.882005 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877672    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:15:55.882005 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877676    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:15:55.882005 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877682    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:15:55.882765 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877687    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:15:55.882765 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877692    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:15:55.882765 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877696    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:15:55.882765 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877700    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:15:55.882765 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877705    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:15:55.882765 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877711    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:15:55.882765 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877717    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:15:55.882765 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877723    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:15:55.882765 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877728    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:15:55.882765 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877733    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:15:55.882765 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877738    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:15:55.882765 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877743    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:15:55.882765 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877748    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:15:55.882765 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877752    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:15:55.882765 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877756    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:15:55.882765 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.877763    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:15:55.882765 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878432    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:15:55.882765 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878441    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:15:55.882765 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878446    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:15:55.883280 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878451    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:15:55.883280 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878455    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:15:55.883280 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878460    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:15:55.883280 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878465    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:15:55.883280 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878469    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:15:55.883280 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878473    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:15:55.883280 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878477    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:15:55.883280 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878481    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:15:55.883280 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878486    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:15:55.883280 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878490    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:15:55.883280 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878497    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:15:55.883280 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878502    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:15:55.883280 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878508    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:15:55.883280 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878514    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:15:55.883280 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878518    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:15:55.883280 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878523    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:15:55.883280 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878527    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:15:55.883280 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878538    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:15:55.883280 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878543    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:15:55.883809 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878547    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:15:55.883809 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878551    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:15:55.883809 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878556    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:15:55.883809 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878560    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:15:55.883809 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878564    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:15:55.883809 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878569    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:15:55.883809 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878573    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:15:55.883809 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878577    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:15:55.883809 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878581    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:15:55.883809 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878586    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:15:55.883809 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878590    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:15:55.883809 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878594    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:15:55.883809 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878599    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:15:55.883809 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878603    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:15:55.883809 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878607    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:15:55.883809 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878611    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:15:55.883809 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878616    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:15:55.883809 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878620    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:15:55.883809 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878625    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:15:55.883809 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878629    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:15:55.884373 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878633    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:15:55.884373 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878637    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:15:55.884373 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878642    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:15:55.884373 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878646    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:15:55.884373 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878650    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:15:55.884373 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878654    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:15:55.884373 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878658    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:15:55.884373 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878663    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:15:55.884373 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878669    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:15:55.884373 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878673    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:15:55.884373 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878677    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:15:55.884373 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878682    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:15:55.884373 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878686    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:15:55.884373 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878690    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:15:55.884373 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878695    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:15:55.884373 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878699    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:15:55.884373 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878703    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:15:55.884373 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878708    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:15:55.884373 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878713    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:15:55.884373 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878717    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:15:55.884864 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878721    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:15:55.884864 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878726    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:15:55.884864 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878730    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:15:55.884864 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878734    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:15:55.884864 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878738    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:15:55.884864 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878743    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:15:55.884864 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878747    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:15:55.884864 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878752    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:15:55.884864 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878759    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:15:55.884864 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878764    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:15:55.884864 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878768    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:15:55.884864 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878772    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:15:55.884864 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878777    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:15:55.884864 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878781    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:15:55.884864 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878785    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:15:55.884864 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878789    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:15:55.884864 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878796    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:15:55.884864 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878801    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:15:55.884864 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878810    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:15:55.885346 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878816    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:15:55.885346 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878823    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:15:55.885346 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878827    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:15:55.885346 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878833    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:15:55.885346 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878838    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:15:55.885346 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878842    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:15:55.885346 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878847    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:15:55.885346 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878852    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:15:55.885346 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878856    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:15:55.885346 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878860    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:15:55.885346 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878864    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:15:55.885346 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878869    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:15:55.885346 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878873    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:15:55.885346 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878877    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:15:55.885346 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878883    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:15:55.885346 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878887    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:15:55.885346 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878891    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:15:55.885346 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.878896    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:15:55.885346 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879005    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 16:15:55.885346 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879060    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 16:15:55.885867 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879070    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 16:15:55.885867 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879077    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 16:15:55.885867 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879083    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 16:15:55.885867 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879088    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 16:15:55.885867 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879095    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 16:15:55.885867 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879102    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 16:15:55.885867 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879107    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 16:15:55.885867 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879117    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 16:15:55.885867 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879123    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 16:15:55.885867 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879130    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 16:15:55.885867 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879135    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 16:15:55.885867 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879142    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 18 16:15:55.885867 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879147    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 16:15:55.885867 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879152    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 18 16:15:55.885867 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879158    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 16:15:55.885867 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879163    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 16:15:55.885867 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879169    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 18 16:15:55.885867 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879174    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 16:15:55.885867 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879179    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 16:15:55.885867 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879184    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 16:15:55.885867 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879189    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 16:15:55.885867 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879195    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 16:15:55.885867 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879200    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 16:15:55.886486 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879205    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 16:15:55.886486 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879211    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 16:15:55.886486 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879216    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 16:15:55.886486 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879223    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 16:15:55.886486 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879228    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 16:15:55.886486 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879233    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 16:15:55.886486 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879238    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 16:15:55.886486 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879244    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 16:15:55.886486 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879249    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 16:15:55.886486 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879254    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 16:15:55.886486 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879258    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 16:15:55.886486 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879263    2580 flags.go:64] FLAG: --enable-server="true"
Sep 18 16:15:55.886486 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879268    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 16:15:55.886486 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879278    2580 flags.go:64] FLAG: --event-burst="100"
Sep 18 16:15:55.886486 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879284    2580 flags.go:64] FLAG: --event-qps="50"
Sep 18 16:15:55.886486 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879289    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 16:15:55.886486 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879294    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 16:15:55.886486 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879315    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 18 16:15:55.886486 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879321    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 16:15:55.886486 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879326    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 16:15:55.886486 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879332    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 16:15:55.886486 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879337    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 18 16:15:55.886486 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879342    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 16:15:55.886486 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879347    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 16:15:55.886486 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879352    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 16:15:55.887187 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879357    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 16:15:55.887187 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879362    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 16:15:55.887187 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879366    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 16:15:55.887187 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879371    2580 flags.go:64] FLAG: --feature-gates=""
Sep 18 16:15:55.887187 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879377    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 16:15:55.887187 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879382    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 16:15:55.887187 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879387    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 16:15:55.887187 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879393    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 16:15:55.887187 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879398    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 16:15:55.887187 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879403    2580 flags.go:64] FLAG: --help="false"
Sep 18 16:15:55.887187 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879411    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-135-110.ec2.internal"
Sep 18 16:15:55.887187 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879416    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 16:15:55.887187 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879421    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 16:15:55.887187 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879426    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 16:15:55.887187 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879432    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 16:15:55.887187 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879437    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 16:15:55.887187 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879442    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 16:15:55.887187 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879447    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 16:15:55.887187 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879451    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 16:15:55.887187 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879456    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 16:15:55.887187 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879461    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 16:15:55.887187 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879466    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 16:15:55.887187 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879471    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 18 16:15:55.887187 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879475    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 16:15:55.887885 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879480    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 16:15:55.887885 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879485    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 16:15:55.887885 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879490    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 16:15:55.887885 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879496    2580 flags.go:64] FLAG: --lock-file=""
Sep 18 16:15:55.887885 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879501    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 16:15:55.887885 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879506    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 16:15:55.887885 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879511    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 16:15:55.887885 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879520    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 16:15:55.887885 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879525    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 16:15:55.887885 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879530    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 16:15:55.887885 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879535    2580 flags.go:64] FLAG: --logging-format="text"
Sep 18 16:15:55.887885 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879540    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 16:15:55.887885 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879545    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 16:15:55.887885 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879550    2580 flags.go:64] FLAG: --manifest-url=""
Sep 18 16:15:55.887885 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879556    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 16:15:55.887885 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879563    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 16:15:55.887885 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879568    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 16:15:55.887885 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879575    2580 flags.go:64] FLAG: --max-pods="110"
Sep 18 16:15:55.887885 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879581    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 16:15:55.887885 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879586    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 16:15:55.887885 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879590    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 16:15:55.887885 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879595    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 16:15:55.887885 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879600    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 16:15:55.887885 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879604    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 16:15:55.887885 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879609    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 16:15:55.888523 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879621    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 16:15:55.888523 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879626    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 16:15:55.888523 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879631    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 16:15:55.888523 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879636    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 18 16:15:55.888523 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879640    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 16:15:55.888523 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879648    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 16:15:55.888523 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879653    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 16:15:55.888523 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879658    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 16:15:55.888523 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879662    2580 flags.go:64] FLAG: --port="10250"
Sep 18 16:15:55.888523 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879667    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 16:15:55.888523 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879674    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0976adfc09fa3e3aa"
Sep 18 16:15:55.888523 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879679    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 18 16:15:55.888523 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879684    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 16:15:55.888523 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879689    2580 flags.go:64] FLAG: --register-node="true"
Sep 18 16:15:55.888523 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879694    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 18 16:15:55.888523 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879700    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 18 16:15:55.888523 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879707    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 18 16:15:55.888523 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879711    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 16:15:55.888523 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879716    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 18 16:15:55.888523 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879722    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 16:15:55.888523 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879727    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 16:15:55.888523 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879732    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 16:15:55.888523 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879737    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 16:15:55.888523 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879742    2580 flags.go:64] FLAG: --runonce="false"
Sep 18 16:15:55.888523 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879747    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 16:15:55.889130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879752    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 16:15:55.889130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879759    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 16:15:55.889130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879763    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 16:15:55.889130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879768    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 16:15:55.889130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879773    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 16:15:55.889130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879778    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 16:15:55.889130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879784    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 16:15:55.889130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879788    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 16:15:55.889130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879793    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 16:15:55.889130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879798    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 16:15:55.889130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879803    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 16:15:55.889130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879807    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 16:15:55.889130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879812    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 18 16:15:55.889130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879817    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 16:15:55.889130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879826    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 16:15:55.889130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879830    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 16:15:55.889130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879835    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 16:15:55.889130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879844    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 18 16:15:55.889130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879849    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 16:15:55.889130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879853    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 16:15:55.889130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879858    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 16:15:55.889130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879863    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 16:15:55.889130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879868    2580 flags.go:64] FLAG: --v="2"
Sep 18 16:15:55.889130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879874    2580 flags.go:64] FLAG: --version="false"
Sep 18 16:15:55.889130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879881    2580 flags.go:64] FLAG: --vmodule=""
Sep 18 16:15:55.889796 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879887    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 16:15:55.889796 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.879892    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 16:15:55.889796 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880084    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:15:55.889796 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880092    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:15:55.889796 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880097    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:15:55.889796 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880102    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:15:55.889796 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880106    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:15:55.889796 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880111    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:15:55.889796 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880117    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:15:55.889796 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880121    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:15:55.889796 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880125    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:15:55.889796 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880130    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:15:55.889796 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880134    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:15:55.889796 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880138    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:15:55.889796 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880143    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:15:55.889796 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880148    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:15:55.889796 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880153    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:15:55.889796 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880157    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:15:55.889796 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880161    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:15:55.889796 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880166    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:15:55.890285 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880169    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:15:55.890285 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880173    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:15:55.890285 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880178    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:15:55.890285 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880182    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:15:55.890285 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880188    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:15:55.890285 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880192    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:15:55.890285 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880196    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:15:55.890285 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880200    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:15:55.890285 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880205    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:15:55.890285 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880209    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:15:55.890285 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880214    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:15:55.890285 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880218    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:15:55.890285 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880223    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:15:55.890285 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880227    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:15:55.890285 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880232    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:15:55.890285 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880236    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:15:55.890285 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880240    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:15:55.890285 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880244    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:15:55.890285 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880249    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:15:55.890285 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880253    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:15:55.890874 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880259    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:15:55.890874 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880263    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:15:55.890874 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880268    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:15:55.890874 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880272    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:15:55.890874 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880276    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:15:55.890874 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880280    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:15:55.890874 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880284    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:15:55.890874 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880288    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:15:55.890874 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880293    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:15:55.890874 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880313    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:15:55.890874 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880319    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:15:55.890874 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880323    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:15:55.890874 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880327    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:15:55.890874 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880332    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:15:55.890874 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880339    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:15:55.890874 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880344    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:15:55.890874 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880351    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:15:55.890874 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880355    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:15:55.890874 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880360    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:15:55.891369 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880364    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:15:55.891369 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880369    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:15:55.891369 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880373    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:15:55.891369 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880378    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:15:55.891369 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880382    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:15:55.891369 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880386    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:15:55.891369 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880394    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:15:55.891369 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880401    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:15:55.891369 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880406    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:15:55.891369 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880411    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:15:55.891369 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880416    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:15:55.891369 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880420    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:15:55.891369 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880424    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:15:55.891369 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880431    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:15:55.891369 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880436    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:15:55.891369 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880440    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:15:55.891369 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880445    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:15:55.891369 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880449    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:15:55.891369 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880453    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:15:55.891369 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880457    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:15:55.891861 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880461    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:15:55.891861 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880465    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:15:55.891861 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880470    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:15:55.891861 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880474    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:15:55.891861 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880478    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:15:55.891861 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880482    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:15:55.891861 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880487    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:15:55.891861 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880491    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:15:55.891861 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880495    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:15:55.891861 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880501    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:15:55.891861 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880507    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:15:55.891861 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880512    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:15:55.891861 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880516    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:15:55.891861 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880521    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:15:55.891861 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880525    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:15:55.891861 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880530    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:15:55.891861 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880534    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:15:55.891861 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880538    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:15:55.891861 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880543    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:15:55.892335 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880548    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:15:55.892335 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880552    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:15:55.892335 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.880556    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:15:55.892335 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.881027    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:15:55.892335 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.888460    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 16:15:55.892335 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.888479    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 16:15:55.892335 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.888499    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 16:15:55.892335 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.888505    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 16:15:55.892335 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888565    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:15:55.892335 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888571    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:15:55.892335 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888575    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:15:55.892335 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888578    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:15:55.892335 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888582    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:15:55.892335 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888585    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:15:55.892688 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888589    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:15:55.892688 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888593    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:15:55.892688 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888596    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:15:55.892688 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888600    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:15:55.892688 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888604    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:15:55.892688 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888607    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:15:55.892688 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888611    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:15:55.892688 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888613    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:15:55.892688 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888616    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:15:55.892688 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888619    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:15:55.892688 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888622    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:15:55.892688 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888625    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:15:55.892688 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888628    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:15:55.892688 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888630    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:15:55.892688 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888633    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:15:55.892688 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888637    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:15:55.892688 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888640    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:15:55.892688 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888643    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:15:55.893169 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888646    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:15:55.893169 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888649    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:15:55.893169 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888652    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:15:55.893169 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888655    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:15:55.893169 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888658    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:15:55.893169 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888667    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:15:55.893169 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888670    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:15:55.893169 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888673    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:15:55.893169 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888676    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:15:55.893169 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888679    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:15:55.893169 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888681    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:15:55.893169 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888684    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:15:55.893169 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888687    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:15:55.893169 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888690    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:15:55.893169 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888693    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:15:55.893169 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888695    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:15:55.893169 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888698    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:15:55.893169 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888700    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:15:55.893169 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888703    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:15:55.893169 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888705    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:15:55.893673 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888708    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:15:55.893673 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888711    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:15:55.893673 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888713    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:15:55.893673 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888716    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:15:55.893673 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888718    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:15:55.893673 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888721    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:15:55.893673 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888723    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:15:55.893673 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888726    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:15:55.893673 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888729    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:15:55.893673 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888731    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:15:55.893673 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888734    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:15:55.893673 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888737    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:15:55.893673 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888739    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:15:55.893673 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888743    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:15:55.893673 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888745    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:15:55.893673 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888748    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:15:55.893673 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888751    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:15:55.893673 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888753    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:15:55.893673 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888762    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:15:55.893673 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888764    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:15:55.894157 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888767    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:15:55.894157 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888770    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:15:55.894157 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888772    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:15:55.894157 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888775    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:15:55.894157 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888778    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:15:55.894157 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888780    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:15:55.894157 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888783    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:15:55.894157 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888785    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:15:55.894157 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888788    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:15:55.894157 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888790    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:15:55.894157 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888793    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:15:55.894157 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888795    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:15:55.894157 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888798    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:15:55.894157 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888800    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:15:55.894157 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888803    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:15:55.894157 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888806    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:15:55.894157 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888808    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:15:55.894157 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888811    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:15:55.894157 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888813    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:15:55.894157 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888816    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:15:55.894663 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888818    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:15:55.894663 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888821    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:15:55.894663 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888824    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:15:55.894663 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888826    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:15:55.894663 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888829    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:15:55.894663 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888831    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:15:55.894663 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888834    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:15:55.894663 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888837    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:15:55.894663 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888839    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:15:55.894663 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888842    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:15:55.894663 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888844    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:15:55.894663 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888852    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:15:55.894663 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888855    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:15:55.894663 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888858    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:15:55.894663 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.888860    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:15:55.895024 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.888865    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:15:55.895024 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889012    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:15:55.895024 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889016    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:15:55.895024 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889019    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:15:55.895024 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889022    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:15:55.895024 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889025    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:15:55.895024 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889028    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:15:55.895024 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889031    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:15:55.895024 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889033    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:15:55.895024 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889036    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:15:55.895024 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889039    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:15:55.895024 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889041    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:15:55.895024 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889044    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:15:55.895024 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889047    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:15:55.895024 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889049    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:15:55.895434 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889052    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:15:55.895434 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889055    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:15:55.895434 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889058    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:15:55.895434 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889061    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:15:55.895434 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889063    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:15:55.895434 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889066    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:15:55.895434 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889069    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:15:55.895434 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889071    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:15:55.895434 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889074    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:15:55.895434 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889077    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:15:55.895434 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889079    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:15:55.895434 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889082    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:15:55.895434 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889085    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:15:55.895434 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889088    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:15:55.895434 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889096    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:15:55.895434 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889099    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:15:55.895434 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889102    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:15:55.895434 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889104    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:15:55.895434 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889107    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:15:55.895434 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889110    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:15:55.895925 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889112    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:15:55.895925 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889115    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:15:55.895925 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889117    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:15:55.895925 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889121    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:15:55.895925 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889125    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:15:55.895925 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889128    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:15:55.895925 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889131    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:15:55.895925 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889133    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:15:55.895925 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889136    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:15:55.895925 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889139    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:15:55.895925 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889142    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:15:55.895925 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889146    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:15:55.895925 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889149    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:15:55.895925 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889151    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:15:55.895925 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889154    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:15:55.895925 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889156    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:15:55.895925 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889159    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:15:55.895925 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889162    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:15:55.895925 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889164    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:15:55.895925 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889167    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:15:55.896421 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889169    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:15:55.896421 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889172    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:15:55.896421 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889175    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:15:55.896421 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889177    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:15:55.896421 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889180    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:15:55.896421 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889183    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:15:55.896421 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889185    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:15:55.896421 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889193    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:15:55.896421 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889196    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:15:55.896421 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889199    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:15:55.896421 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889202    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:15:55.896421 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889204    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:15:55.896421 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889207    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:15:55.896421 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889210    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:15:55.896421 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889212    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:15:55.896421 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889215    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:15:55.896421 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889217    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:15:55.896421 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889220    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:15:55.896421 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889223    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:15:55.896421 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889225    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:15:55.896905 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889229    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:15:55.896905 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889232    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:15:55.896905 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889235    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:15:55.896905 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889238    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:15:55.896905 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889241    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:15:55.896905 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889244    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:15:55.896905 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889246    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:15:55.896905 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889249    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:15:55.896905 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889251    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:15:55.896905 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889254    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:15:55.896905 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889256    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:15:55.896905 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889259    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:15:55.896905 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889261    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:15:55.896905 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889264    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:15:55.896905 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889267    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:15:55.896905 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889269    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:15:55.896905 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889272    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:15:55.896905 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889275    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:15:55.896905 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889277    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:15:55.897378 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889280    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:15:55.897378 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889290    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:15:55.897378 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889293    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:15:55.897378 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889296    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:15:55.897378 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889314    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:15:55.897378 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:55.889317    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:15:55.897378 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.889322    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:15:55.897378 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.889451    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 16:15:55.897378 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.891707    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 16:15:55.897378 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.892597    2580 server.go:1019] "Starting client certificate rotation"
Sep 18 16:15:55.897378 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.892719    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:15:55.897378 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.892749    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:15:55.897378 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.895014    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 16:15:55.897708 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.896552    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 16:15:55.909940 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.909923    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 18 16:15:55.911542 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.911529    2580 log.go:25] "Validated CRI v1 image API"
Sep 18 16:15:55.912248 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.912234    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 16:15:55.914691 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.914673    2580 fs.go:135] Filesystem UUIDs: map[717f0bbe-6ccb-4598-ae67-7076dd41d542:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 97291d81-8bb8-4cc2-b693-b0ccb24a180f:/dev/nvme0n1p3]
Sep 18 16:15:55.914747 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.914690    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 16:15:55.918837 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.918727    2580 manager.go:216] Machine: {Timestamp:2026-09-18 16:15:55.918238397 +0000 UTC m=+2.006554028 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103978 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec258b90706b7fd48a665da9c331001e SystemUUID:ec258b90-706b-7fd4-8a66-5da9c331001e BootID:f27f70bb-a03d-4e8f-bba8-e7cc9569d4ba Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e7:99:bd:bd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e7:99:bd:bd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:e6:45:c8:05:f0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 16:15:55.918837 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.918832    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 16:15:55.918956 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.918944    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 16:15:55.920927 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.920901    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 16:15:55.921050 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.920929    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-110.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 16:15:55.922438 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.922428    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 16:15:55.922481 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.922441    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 16:15:55.922481 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.922449    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 16:15:55.922481 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.922460    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 16:15:55.922571 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.922527    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 16:15:55.925363 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.925352    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 16:15:55.925473 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.925463    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 16:15:55.925509 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.925501    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 16:15:55.925589 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.925513    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 16:15:55.925589 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.925528    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 16:15:55.925589 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.925536    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 18 16:15:55.925589 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.925546    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 16:15:55.925969 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.925952    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 16:15:55.930782 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.925979    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 16:15:55.930782 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.927874    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 16:15:55.930782 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.929915    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 16:15:55.930782 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.929943    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 16:15:55.930909 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.930783    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 16:15:55.930909 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.930850    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 16:15:55.930909 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.930865    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 16:15:55.930909 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.930871    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 16:15:55.930909 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.930877    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 16:15:55.930909 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.930886    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 16:15:55.930909 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.930892    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 16:15:55.930909 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.930911    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 16:15:55.931107 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.930917    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 16:15:55.931107 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.930924    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 16:15:55.931107 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.930929    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 16:15:55.931107 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.930938    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 16:15:55.931107 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.930943    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 16:15:55.931107 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.930968    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 16:15:55.931107 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.930973    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 16:15:55.934445 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.934433    2580 server.go:1268] "Started kubelet"
Sep 18 16:15:55.934543 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.934506    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 16:15:55.934596 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.934519    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 16:15:55.934648 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.934611    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 16:15:55.935278 ip-10-0-135-110 systemd[1]: Started Kubernetes Kubelet.
Sep 18 16:15:55.935851 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.935720    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 16:15:55.935970 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.935950    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 16:15:55.939798 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:55.939777    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 16:15:55.942194 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.942182    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 16:15:55.969569 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.969166    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 16:15:55.970960 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.970937    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 16:15:55.971972 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.971934    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 16:15:55.971972 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.971975    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 16:15:55.972453 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.972354    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 16:15:55.972453 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.972365    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 16:15:55.972453 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:55.972351    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-110.ec2.internal\" not found"
Sep 18 16:15:55.972623 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.972575    2580 factory.go:153] Registering CRI-O factory
Sep 18 16:15:55.972623 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.972596    2580 factory.go:223] Registration of the crio container factory successfully
Sep 18 16:15:55.972714 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.972657    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 16:15:55.972714 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.972668    2580 factory.go:55] Registering systemd factory
Sep 18 16:15:55.972714 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.972677    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 18 16:15:55.972714 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.972715    2580 factory.go:103] Registering Raw factory
Sep 18 16:15:55.972886 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.972734    2580 manager.go:1196] Started watching for new ooms in manager
Sep 18 16:15:55.972987 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:55.972952    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-110.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 16:15:55.973317 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.973275    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-110.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:15:55.973630 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:55.973592    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-110.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:15:55.973813 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:55.973521    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:15:55.974259 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.974246    2580 manager.go:349] Starting recovery of all containers
Sep 18 16:15:55.978988 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:55.978962    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 16:15:55.979890 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:55.978974    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-110.ec2.internal.18d676771fe17a8a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-110.ec2.internal,UID:ip-10-0-135-110.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-110.ec2.internal,},FirstTimestamp:2026-09-18 16:15:55.934407306 +0000 UTC m=+2.022722936,LastTimestamp:2026-09-18 16:15:55.934407306 +0000 UTC m=+2.022722936,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-110.ec2.internal,}"
Sep 18 16:15:55.984130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.984110    2580 manager.go:354] Recovery completed
Sep 18 16:15:55.988310 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.988285    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:15:55.989937 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.989911    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-110.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:15:55.989994 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.989950    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-110.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:15:55.989994 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.989960    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-110.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:15:55.990513 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.990493    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 16:15:55.990589 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.990515    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 16:15:55.990589 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.990539    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 16:15:55.990589 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.990497    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:15:55.991227 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.991214    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-110.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:15:55.991285 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.991236    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-110.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:15:55.991285 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.991246    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-110.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:15:55.992021 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:55.991960    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-110.ec2.internal.18d676772330cc8e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-110.ec2.internal,UID:ip-10-0-135-110.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-110.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-110.ec2.internal,},FirstTimestamp:2026-09-18 16:15:55.989937294 +0000 UTC m=+2.078252925,LastTimestamp:2026-09-18 16:15:55.989937294 +0000 UTC m=+2.078252925,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-110.ec2.internal,}"
Sep 18 16:15:55.992214 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.992202    2580 policy_none.go:49] "None policy: Start"
Sep 18 16:15:55.992246 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.992220    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 16:15:55.992246 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.992233    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 16:15:55.993278 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:55.993268    2580 policy_none.go:47] "Start"
Sep 18 16:15:56.001932 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:56.001863    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-110.ec2.internal.18d6767723310fe5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-110.ec2.internal,UID:ip-10-0-135-110.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-110.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-110.ec2.internal,},FirstTimestamp:2026-09-18 16:15:55.989954533 +0000 UTC m=+2.078270164,LastTimestamp:2026-09-18 16:15:55.989954533 +0000 UTC m=+2.078270164,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-110.ec2.internal,}"
Sep 18 16:15:56.011573 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:56.011501    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-110.ec2.internal.18d6767723313622  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-110.ec2.internal,UID:ip-10-0-135-110.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-110.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-110.ec2.internal,},FirstTimestamp:2026-09-18 16:15:55.989964322 +0000 UTC m=+2.078279953,LastTimestamp:2026-09-18 16:15:55.989964322 +0000 UTC m=+2.078279953,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-110.ec2.internal,}"
Sep 18 16:15:56.023032 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:56.022971    2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-110.ec2.internal.18d676772330cc8e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-110.ec2.internal.18d676772330cc8e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-110.ec2.internal,UID:ip-10-0-135-110.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-110.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-110.ec2.internal,},FirstTimestamp:2026-09-18 16:15:55.989937294 +0000 UTC m=+2.078252925,LastTimestamp:2026-09-18 16:15:55.99122573 +0000 UTC m=+2.079541360,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-110.ec2.internal,}"
Sep 18 16:15:56.036536 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.023963    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ztdmh"
Sep 18 16:15:56.036536 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.026878    2580 manager.go:337] "Starting Device Plugin manager"
Sep 18 16:15:56.036536 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:56.026916    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 16:15:56.036536 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.026930    2580 server.go:87] "Starting device plugin registration server"
Sep 18 16:15:56.036536 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.027151    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 16:15:56.036536 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.027173    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 16:15:56.036536 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.027277    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 16:15:56.036536 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.027388    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 16:15:56.036536 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.027424    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 16:15:56.036536 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:56.027929    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 16:15:56.036536 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:56.027981    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-110.ec2.internal\" not found"
Sep 18 16:15:56.036536 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.032839    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ztdmh"
Sep 18 16:15:56.038472 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:56.038412    2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-110.ec2.internal.18d6767723310fe5\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-110.ec2.internal.18d6767723310fe5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-110.ec2.internal,UID:ip-10-0-135-110.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-110.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-110.ec2.internal,},FirstTimestamp:2026-09-18 16:15:55.989954533 +0000 UTC m=+2.078270164,LastTimestamp:2026-09-18 16:15:55.99124097 +0000 UTC m=+2.079556600,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-110.ec2.internal,}"
Sep 18 16:15:56.114750 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.114719    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 16:15:56.115965 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.115946    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 16:15:56.116060 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.115992    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 16:15:56.116060 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.116014    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 16:15:56.116145 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:56.116055    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 16:15:56.118436 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.118420    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:15:56.128483 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.128442    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:15:56.129646 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.129632    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-110.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:15:56.129723 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.129663    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-110.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:15:56.129723 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.129679    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-110.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:15:56.129723 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.129707    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-110.ec2.internal"
Sep 18 16:15:56.137958 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.137945    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-110.ec2.internal"
Sep 18 16:15:56.138019 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:56.137963    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-110.ec2.internal\": node \"ip-10-0-135-110.ec2.internal\" not found"
Sep 18 16:15:56.156869 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:56.156850    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-110.ec2.internal\" not found"
Sep 18 16:15:56.216362 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.216326    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-110.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-110.ec2.internal"]
Sep 18 16:15:56.216457 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.216418    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:15:56.217283 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.217269    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-110.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:15:56.217434 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.217315    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-110.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:15:56.217434 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.217331    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-110.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:15:56.218724 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.218709    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:15:56.218838 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.218822    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-110.ec2.internal"
Sep 18 16:15:56.218886 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.218860    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:15:56.219471 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.219454    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-110.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:15:56.219554 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.219483    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-110.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:15:56.219554 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.219494    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-110.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:15:56.219554 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.219458    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-110.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:15:56.219692 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.219576    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-110.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:15:56.219692 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.219594    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-110.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:15:56.220832 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.220816    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-110.ec2.internal"
Sep 18 16:15:56.220909 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.220838    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:15:56.221954 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.221941    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-110.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:15:56.222044 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.221969    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-110.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:15:56.222044 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.221982    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-110.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:15:56.250844 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:56.250820    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-110.ec2.internal\" not found" node="ip-10-0-135-110.ec2.internal"
Sep 18 16:15:56.255015 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:56.255002    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-110.ec2.internal\" not found" node="ip-10-0-135-110.ec2.internal"
Sep 18 16:15:56.257049 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:56.257036    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-110.ec2.internal\" not found"
Sep 18 16:15:56.274558 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.274536    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/43c30dee52b2988542f60a71f377520c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-110.ec2.internal\" (UID: \"43c30dee52b2988542f60a71f377520c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-110.ec2.internal"
Sep 18 16:15:56.274634 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.274565    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43c30dee52b2988542f60a71f377520c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-110.ec2.internal\" (UID: \"43c30dee52b2988542f60a71f377520c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-110.ec2.internal"
Sep 18 16:15:56.274634 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.274590    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8e36c319ff90245c5f11900bd31f47e-config\") pod \"kube-apiserver-proxy-ip-10-0-135-110.ec2.internal\" (UID: \"b8e36c319ff90245c5f11900bd31f47e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-110.ec2.internal"
Sep 18 16:15:56.357538 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:56.357521    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-110.ec2.internal\" not found"
Sep 18 16:15:56.374848 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.374831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/43c30dee52b2988542f60a71f377520c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-110.ec2.internal\" (UID: \"43c30dee52b2988542f60a71f377520c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-110.ec2.internal"
Sep 18 16:15:56.374894 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.374854    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43c30dee52b2988542f60a71f377520c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-110.ec2.internal\" (UID: \"43c30dee52b2988542f60a71f377520c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-110.ec2.internal"
Sep 18 16:15:56.374894 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.374873    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8e36c319ff90245c5f11900bd31f47e-config\") pod \"kube-apiserver-proxy-ip-10-0-135-110.ec2.internal\" (UID: \"b8e36c319ff90245c5f11900bd31f47e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-110.ec2.internal"
Sep 18 16:15:56.374969 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.374918    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8e36c319ff90245c5f11900bd31f47e-config\") pod \"kube-apiserver-proxy-ip-10-0-135-110.ec2.internal\" (UID: \"b8e36c319ff90245c5f11900bd31f47e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-110.ec2.internal"
Sep 18 16:15:56.374969 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.374932    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/43c30dee52b2988542f60a71f377520c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-110.ec2.internal\" (UID: \"43c30dee52b2988542f60a71f377520c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-110.ec2.internal"
Sep 18 16:15:56.374969 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.374944    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43c30dee52b2988542f60a71f377520c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-110.ec2.internal\" (UID: \"43c30dee52b2988542f60a71f377520c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-110.ec2.internal"
Sep 18 16:15:56.458231 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:56.458209    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-110.ec2.internal\" not found"
Sep 18 16:15:56.553730 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.553702    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-110.ec2.internal"
Sep 18 16:15:56.558073 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.558031    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-110.ec2.internal"
Sep 18 16:15:56.559136 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:56.559108    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-110.ec2.internal\" not found"
Sep 18 16:15:56.660075 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:56.660043    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-110.ec2.internal\" not found"
Sep 18 16:15:56.760587 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:56.760515    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-110.ec2.internal\" not found"
Sep 18 16:15:56.860981 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:56.860950    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-110.ec2.internal\" not found"
Sep 18 16:15:56.892378 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.892357    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 16:15:56.892722 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:56.892489    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 16:15:56.961813 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:56.961789    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-110.ec2.internal\" not found"
Sep 18 16:15:57.037257 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.037189    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 16:10:56 +0000 UTC" deadline="2028-03-12 14:06:52.358188344 +0000 UTC"
Sep 18 16:15:57.037257 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.037221    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12981h50m55.3209706s"
Sep 18 16:15:57.062316 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:57.062289    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-110.ec2.internal\" not found"
Sep 18 16:15:57.064438 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.064425    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:15:57.066310 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.066281    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-110.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:15:57.066386 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.066327    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-110.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:15:57.066386 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.066342    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-110.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:15:57.163125 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:57.163099    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-110.ec2.internal\" not found"
Sep 18 16:15:57.195132 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.195111    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:15:57.263555 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:57.263526    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-110.ec2.internal\" not found"
Sep 18 16:15:57.317993 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:57.317935    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8e36c319ff90245c5f11900bd31f47e.slice/crio-6e32c0b300bd33cba0eafae0e4d21a5965b13ba769b267eb482d94ad1a7d62ba WatchSource:0}: Error finding container 6e32c0b300bd33cba0eafae0e4d21a5965b13ba769b267eb482d94ad1a7d62ba: Status 404 returned error can't find the container with id 6e32c0b300bd33cba0eafae0e4d21a5965b13ba769b267eb482d94ad1a7d62ba
Sep 18 16:15:57.318213 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:57.318191    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c30dee52b2988542f60a71f377520c.slice/crio-43a70dd30b4642f52575931e60494df4ddd16ddc8205b33cc7f3c933d942e5a4 WatchSource:0}: Error finding container 43a70dd30b4642f52575931e60494df4ddd16ddc8205b33cc7f3c933d942e5a4: Status 404 returned error can't find the container with id 43a70dd30b4642f52575931e60494df4ddd16ddc8205b33cc7f3c933d942e5a4
Sep 18 16:15:57.322291 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.322272    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:15:57.364084 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:57.364057    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-110.ec2.internal\" not found"
Sep 18 16:15:57.464946 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:57.464918    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-110.ec2.internal\" not found"
Sep 18 16:15:57.492903 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.492882    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:15:57.565979 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:57.565952    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-110.ec2.internal\" not found"
Sep 18 16:15:57.575940 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.575887    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:15:57.672151 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.672135    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-110.ec2.internal"
Sep 18 16:15:57.683697 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.683681    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 16:15:57.684438 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.684427    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-110.ec2.internal"
Sep 18 16:15:57.694152 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.694138    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 16:15:57.801233 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.801205    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:15:57.926852 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.926820    2580 apiserver.go:52] "Watching apiserver"
Sep 18 16:15:57.932326 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.932289    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 16:15:57.932814 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.932789    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-h6tt8","openshift-ovn-kubernetes/ovnkube-node-gw4fg","kube-system/kube-apiserver-proxy-ip-10-0-135-110.ec2.internal","openshift-cluster-node-tuning-operator/tuned-96w8b","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-110.ec2.internal","openshift-multus/multus-8tkmt","openshift-multus/multus-additional-cni-plugins-ls6gx","openshift-network-diagnostics/network-check-target-fs4cj","kube-system/global-pull-secret-syncer-cmscw","kube-system/konnectivity-agent-k6gdw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf","openshift-dns/node-resolver-krchj","openshift-image-registry/node-ca-4vmq6","openshift-multus/network-metrics-daemon-tnb7c"]
Sep 18 16:15:57.935027 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.934513    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ls6gx"
Sep 18 16:15:57.936332 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.936055    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf"
Sep 18 16:15:57.937117 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.937095    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 16:15:57.937481 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.937460    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fllbm\""
Sep 18 16:15:57.937626 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.937593    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 16:15:57.937740 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.937714    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 16:15:57.939526 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.937815    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 16:15:57.939526 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.937901    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 16:15:57.939526 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.937917    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:15:57.939526 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:57.937980    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tnb7c" podUID="c861f27f-910e-48aa-b4bc-3c20bcd911ef"
Sep 18 16:15:57.939526 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.938482    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 16:15:57.939526 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.938578    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 16:15:57.939526 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.938683    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wl54c\""
Sep 18 16:15:57.939526 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.938800    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 16:15:57.939526 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.939060    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:57.941100 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.941078    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:15:57.941191 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:57.941172    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs4cj" podUID="8b313705-71ee-4c64-a650-b308f048dc2a"
Sep 18 16:15:57.942197 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.942176    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:15:57.942268 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.942218    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 16:15:57.942925 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.942899    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jrhnt\""
Sep 18 16:15:57.946475 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.946455    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:57.947828 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.947655    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:15:57.947917 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:57.947877    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmscw" podUID="97bef555-d008-4f5b-a974-91fbafd1eeb1"
Sep 18 16:15:57.949323 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.949156    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 16:15:57.949323 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.949242    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vj79p\""
Sep 18 16:15:57.949964 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.949481    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4vmq6"
Sep 18 16:15:57.950832 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.950817    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:57.952258 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.952238    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 16:15:57.952488 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.952469    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 16:15:57.952859 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.952840    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-trwtz\""
Sep 18 16:15:57.953898 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.953109    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 16:15:57.953898 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.953516    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 16:15:57.954083 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.954063    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 16:15:57.954327 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.954263    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 16:15:57.954597 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.954573    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-krchj"
Sep 18 16:15:57.954826 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.954773    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 16:15:57.954963 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.954945    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k6gdw"
Sep 18 16:15:57.955069 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.955049    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 16:15:57.955684 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.955226    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h6tt8"
Sep 18 16:15:57.955684 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.955610    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hwhdt\""
Sep 18 16:15:57.955846 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.955803    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 16:15:57.957028 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.956888    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 16:15:57.957215 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.957199    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 16:15:57.957658 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.957588    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:15:57.959003 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.957865    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 16:15:57.959003 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.958356    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tkfs8\""
Sep 18 16:15:57.959003 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.958628    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-55dsj\""
Sep 18 16:15:57.959003 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.958835    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 16:15:57.959259 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.959003    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 16:15:57.959259 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.959004    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 16:15:57.959432 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.959406    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6kklg\""
Sep 18 16:15:57.970330 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.970294    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 16:15:57.972221 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.972206    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 16:15:57.984337 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.984314    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-etc-kubernetes\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:57.984426 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.984348    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 16:15:57.984426 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.984352    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/752ec166-f680-4caf-b569-5ca2d59307dc-tmp\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:57.984426 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.984397    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-systemd-units\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:57.984426 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.984420    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-run-openvswitch\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:57.984672 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.984446    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/68f204b6-4215-48a9-973e-d1e84ae4fed5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ls6gx\" (UID: \"68f204b6-4215-48a9-973e-d1e84ae4fed5\") " pod="openshift-multus/multus-additional-cni-plugins-ls6gx"
Sep 18 16:15:57.984672 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.984471    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-hostroot\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:57.984672 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.984492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-run-systemd\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:57.984672 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.984522    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-log-socket\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:57.984672 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.984549    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/68f204b6-4215-48a9-973e-d1e84ae4fed5-cnibin\") pod \"multus-additional-cni-plugins-ls6gx\" (UID: \"68f204b6-4215-48a9-973e-d1e84ae4fed5\") " pod="openshift-multus/multus-additional-cni-plugins-ls6gx"
Sep 18 16:15:57.984672 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.984610    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/022894e1-03ce-4883-91de-722dc423fc87-etc-selinux\") pod \"aws-ebs-csi-driver-node-pv5cf\" (UID: \"022894e1-03ce-4883-91de-722dc423fc87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf"
Sep 18 16:15:57.984672 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.984638    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-host-run-k8s-cni-cncf-io\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:57.985003 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.984664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr29z\" (UniqueName: \"kubernetes.io/projected/cfa7efc4-7409-41f5-b891-7ce9e5b6982f-kube-api-access-rr29z\") pod \"node-ca-4vmq6\" (UID: \"cfa7efc4-7409-41f5-b891-7ce9e5b6982f\") " pod="openshift-image-registry/node-ca-4vmq6"
Sep 18 16:15:57.985003 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.984702    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-etc-openvswitch\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:57.985003 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.984752    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzt9g\" (UniqueName: \"kubernetes.io/projected/8b313705-71ee-4c64-a650-b308f048dc2a-kube-api-access-kzt9g\") pod \"network-check-target-fs4cj\" (UID: \"8b313705-71ee-4c64-a650-b308f048dc2a\") " pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:15:57.985003 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.984776    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e80a48a0-bdd3-4128-999a-cc2574ced499-multus-daemon-config\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:57.985003 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.984799    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-etc-sysctl-d\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:57.985003 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.984821    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-etc-systemd\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:57.985003 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.984844    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-run\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:57.985003 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.984871    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-var-lib-openvswitch\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:57.985003 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.984894    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-host-run-ovn-kubernetes\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:57.985003 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.984916    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c861f27f-910e-48aa-b4bc-3c20bcd911ef-metrics-certs\") pod \"network-metrics-daemon-tnb7c\" (UID: \"c861f27f-910e-48aa-b4bc-3c20bcd911ef\") " pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:15:57.985003 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.984974    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-multus-conf-dir\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:57.985003 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.985003    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97bef555-d008-4f5b-a974-91fbafd1eeb1-dbus\") pod \"global-pull-secret-syncer-cmscw\" (UID: \"97bef555-d008-4f5b-a974-91fbafd1eeb1\") " pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:15:57.985621 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.985026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97bef555-d008-4f5b-a974-91fbafd1eeb1-original-pull-secret\") pod \"global-pull-secret-syncer-cmscw\" (UID: \"97bef555-d008-4f5b-a974-91fbafd1eeb1\") " pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:15:57.985621 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.985055    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cfa7efc4-7409-41f5-b891-7ce9e5b6982f-host\") pod \"node-ca-4vmq6\" (UID: \"cfa7efc4-7409-41f5-b891-7ce9e5b6982f\") " pod="openshift-image-registry/node-ca-4vmq6"
Sep 18 16:15:57.985621 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.985076    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-host-var-lib-cni-bin\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:57.985621 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.985098    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/752ec166-f680-4caf-b569-5ca2d59307dc-etc-tuned\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:57.985621 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.985121    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rsqq\" (UniqueName: \"kubernetes.io/projected/11366fd3-c5b8-4bf4-8ba6-217fd3a1b658-kube-api-access-9rsqq\") pod \"iptables-alerter-h6tt8\" (UID: \"11366fd3-c5b8-4bf4-8ba6-217fd3a1b658\") " pod="openshift-network-operator/iptables-alerter-h6tt8"
Sep 18 16:15:57.985621 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.985144    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/68f204b6-4215-48a9-973e-d1e84ae4fed5-system-cni-dir\") pod \"multus-additional-cni-plugins-ls6gx\" (UID: \"68f204b6-4215-48a9-973e-d1e84ae4fed5\") " pod="openshift-multus/multus-additional-cni-plugins-ls6gx"
Sep 18 16:15:57.985621 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.985166    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/022894e1-03ce-4883-91de-722dc423fc87-device-dir\") pod \"aws-ebs-csi-driver-node-pv5cf\" (UID: \"022894e1-03ce-4883-91de-722dc423fc87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf"
Sep 18 16:15:57.985621 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.985195    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-multus-cni-dir\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:57.985621 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.985218    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-etc-modprobe-d\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:57.985621 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.985240    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-host-kubelet\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:57.985621 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.985273    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-host-run-netns\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:57.985621 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.985327    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-ovnkube-script-lib\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:57.985621 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.985351    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/68f204b6-4215-48a9-973e-d1e84ae4fed5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ls6gx\" (UID: \"68f204b6-4215-48a9-973e-d1e84ae4fed5\") " pod="openshift-multus/multus-additional-cni-plugins-ls6gx"
Sep 18 16:15:57.985621 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.985406    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/68f204b6-4215-48a9-973e-d1e84ae4fed5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ls6gx\" (UID: \"68f204b6-4215-48a9-973e-d1e84ae4fed5\") " pod="openshift-multus/multus-additional-cni-plugins-ls6gx"
Sep 18 16:15:57.985621 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.985468    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-host-run-netns\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:57.985621 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.985563    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-etc-sysconfig\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:57.986417 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.985603    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-etc-kubernetes\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:57.986417 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.985628    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-host-cni-netd\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:57.986417 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.985673    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/022894e1-03ce-4883-91de-722dc423fc87-registration-dir\") pod \"aws-ebs-csi-driver-node-pv5cf\" (UID: \"022894e1-03ce-4883-91de-722dc423fc87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf"
Sep 18 16:15:57.986417 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.985714    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e80a48a0-bdd3-4128-999a-cc2574ced499-cni-binary-copy\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:57.986417 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.985747    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-run-ovn\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:57.986417 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.985817    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11366fd3-c5b8-4bf4-8ba6-217fd3a1b658-host-slash\") pod \"iptables-alerter-h6tt8\" (UID: \"11366fd3-c5b8-4bf4-8ba6-217fd3a1b658\") " pod="openshift-network-operator/iptables-alerter-h6tt8"
Sep 18 16:15:57.986417 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.985860    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/68f204b6-4215-48a9-973e-d1e84ae4fed5-cni-binary-copy\") pod \"multus-additional-cni-plugins-ls6gx\" (UID: \"68f204b6-4215-48a9-973e-d1e84ae4fed5\") " pod="openshift-multus/multus-additional-cni-plugins-ls6gx"
Sep 18 16:15:57.986417 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.985889    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-cnibin\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:57.986417 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.986244    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnnn9\" (UniqueName: \"kubernetes.io/projected/e80a48a0-bdd3-4128-999a-cc2574ced499-kube-api-access-pnnn9\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:57.986417 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.986276    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-env-overrides\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:57.986417 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.986290    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97bef555-d008-4f5b-a974-91fbafd1eeb1-kubelet-config\") pod \"global-pull-secret-syncer-cmscw\" (UID: \"97bef555-d008-4f5b-a974-91fbafd1eeb1\") " pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:15:57.986417 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.986325    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-etc-sysctl-conf\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:57.986417 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.986356    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8c726ad7-c84f-4629-9797-52cfa3327996-agent-certs\") pod \"konnectivity-agent-k6gdw\" (UID: \"8c726ad7-c84f-4629-9797-52cfa3327996\") " pod="kube-system/konnectivity-agent-k6gdw"
Sep 18 16:15:57.986417 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.986400    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8c726ad7-c84f-4629-9797-52cfa3327996-konnectivity-ca\") pod \"konnectivity-agent-k6gdw\" (UID: \"8c726ad7-c84f-4629-9797-52cfa3327996\") " pod="kube-system/konnectivity-agent-k6gdw"
Sep 18 16:15:57.987079 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.986432    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-host\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:57.987079 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.986467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/022894e1-03ce-4883-91de-722dc423fc87-sys-fs\") pod \"aws-ebs-csi-driver-node-pv5cf\" (UID: \"022894e1-03ce-4883-91de-722dc423fc87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf"
Sep 18 16:15:57.987079 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.986515    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-host-var-lib-cni-multus\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:57.987079 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.986557    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-var-lib-kubelet\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:57.987079 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.986573    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zvkf\" (UniqueName: \"kubernetes.io/projected/c861f27f-910e-48aa-b4bc-3c20bcd911ef-kube-api-access-4zvkf\") pod \"network-metrics-daemon-tnb7c\" (UID: \"c861f27f-910e-48aa-b4bc-3c20bcd911ef\") " pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:15:57.987079 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.986593    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-system-cni-dir\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:57.987079 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.986624    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cfa7efc4-7409-41f5-b891-7ce9e5b6982f-serviceca\") pod \"node-ca-4vmq6\" (UID: \"cfa7efc4-7409-41f5-b891-7ce9e5b6982f\") " pod="openshift-image-registry/node-ca-4vmq6"
Sep 18 16:15:57.987079 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.986673    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-lib-modules\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:57.987079 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.986699    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkf2q\" (UniqueName: \"kubernetes.io/projected/752ec166-f680-4caf-b569-5ca2d59307dc-kube-api-access-hkf2q\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:57.987079 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.986732    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-host-slash\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:57.987079 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.986754    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-node-log\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:57.987079 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.986781    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-ovnkube-config\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:57.987079 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.986804    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e945222a-603b-4489-a528-2f607d157b14-tmp-dir\") pod \"node-resolver-krchj\" (UID: \"e945222a-603b-4489-a528-2f607d157b14\") " pod="openshift-dns/node-resolver-krchj"
Sep 18 16:15:57.987079 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.986829    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/022894e1-03ce-4883-91de-722dc423fc87-socket-dir\") pod \"aws-ebs-csi-driver-node-pv5cf\" (UID: \"022894e1-03ce-4883-91de-722dc423fc87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf"
Sep 18 16:15:57.987079 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.986851    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-sys\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:57.987079 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.986875    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:57.987897 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.986907    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11366fd3-c5b8-4bf4-8ba6-217fd3a1b658-iptables-alerter-script\") pod \"iptables-alerter-h6tt8\" (UID: \"11366fd3-c5b8-4bf4-8ba6-217fd3a1b658\") " pod="openshift-network-operator/iptables-alerter-h6tt8"
Sep 18 16:15:57.987897 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.986948    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bg7f\" (UniqueName: \"kubernetes.io/projected/68f204b6-4215-48a9-973e-d1e84ae4fed5-kube-api-access-7bg7f\") pod \"multus-additional-cni-plugins-ls6gx\" (UID: \"68f204b6-4215-48a9-973e-d1e84ae4fed5\") " pod="openshift-multus/multus-additional-cni-plugins-ls6gx"
Sep 18 16:15:57.987897 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.986978    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-multus-socket-dir-parent\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:57.987897 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.987024    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-host-cni-bin\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:57.987897 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.987051    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-ovn-node-metrics-cert\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:57.987897 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.987090    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmctj\" (UniqueName: \"kubernetes.io/projected/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-kube-api-access-nmctj\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:57.987897 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.987115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e945222a-603b-4489-a528-2f607d157b14-hosts-file\") pod \"node-resolver-krchj\" (UID: \"e945222a-603b-4489-a528-2f607d157b14\") " pod="openshift-dns/node-resolver-krchj"
Sep 18 16:15:57.987897 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.987149    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/022894e1-03ce-4883-91de-722dc423fc87-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pv5cf\" (UID: \"022894e1-03ce-4883-91de-722dc423fc87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf"
Sep 18 16:15:57.987897 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.987169    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9jmh\" (UniqueName: \"kubernetes.io/projected/022894e1-03ce-4883-91de-722dc423fc87-kube-api-access-q9jmh\") pod \"aws-ebs-csi-driver-node-pv5cf\" (UID: \"022894e1-03ce-4883-91de-722dc423fc87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf"
Sep 18 16:15:57.987897 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.987201    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xbcv\" (UniqueName: \"kubernetes.io/projected/e945222a-603b-4489-a528-2f607d157b14-kube-api-access-9xbcv\") pod \"node-resolver-krchj\" (UID: \"e945222a-603b-4489-a528-2f607d157b14\") " pod="openshift-dns/node-resolver-krchj"
Sep 18 16:15:57.987897 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.987227    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-host-var-lib-kubelet\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:57.987897 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:57.987260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-host-run-multus-certs\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.011417 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.011348    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-z8t9l"
Sep 18 16:15:58.019480 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.019461    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-z8t9l"
Sep 18 16:15:58.088089 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088055    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/68f204b6-4215-48a9-973e-d1e84ae4fed5-system-cni-dir\") pod \"multus-additional-cni-plugins-ls6gx\" (UID: \"68f204b6-4215-48a9-973e-d1e84ae4fed5\") " pod="openshift-multus/multus-additional-cni-plugins-ls6gx"
Sep 18 16:15:58.088273 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088128    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/68f204b6-4215-48a9-973e-d1e84ae4fed5-system-cni-dir\") pod \"multus-additional-cni-plugins-ls6gx\" (UID: \"68f204b6-4215-48a9-973e-d1e84ae4fed5\") " pod="openshift-multus/multus-additional-cni-plugins-ls6gx"
Sep 18 16:15:58.088273 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088175    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/022894e1-03ce-4883-91de-722dc423fc87-device-dir\") pod \"aws-ebs-csi-driver-node-pv5cf\" (UID: \"022894e1-03ce-4883-91de-722dc423fc87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf"
Sep 18 16:15:58.088273 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088202    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-multus-cni-dir\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.088273 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088227    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-etc-modprobe-d\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.088273 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088252    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-host-kubelet\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.088273 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088274    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-host-run-netns\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.088642 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088311    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-ovnkube-script-lib\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.088642 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088314    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/022894e1-03ce-4883-91de-722dc423fc87-device-dir\") pod \"aws-ebs-csi-driver-node-pv5cf\" (UID: \"022894e1-03ce-4883-91de-722dc423fc87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf"
Sep 18 16:15:58.088642 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088336    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/68f204b6-4215-48a9-973e-d1e84ae4fed5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ls6gx\" (UID: \"68f204b6-4215-48a9-973e-d1e84ae4fed5\") " pod="openshift-multus/multus-additional-cni-plugins-ls6gx"
Sep 18 16:15:58.088642 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-host-kubelet\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.088642 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088373    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-host-run-netns\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.088642 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088380    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-etc-modprobe-d\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.088642 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088391    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-multus-cni-dir\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.088642 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088411    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/68f204b6-4215-48a9-973e-d1e84ae4fed5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ls6gx\" (UID: \"68f204b6-4215-48a9-973e-d1e84ae4fed5\") " pod="openshift-multus/multus-additional-cni-plugins-ls6gx"
Sep 18 16:15:58.088642 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088434    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-host-run-netns\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.088642 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088449    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-etc-sysconfig\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.088642 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088470    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-etc-kubernetes\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.088642 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088495    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-host-cni-netd\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.088642 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088512    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-etc-sysconfig\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.088642 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088520    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/022894e1-03ce-4883-91de-722dc423fc87-registration-dir\") pod \"aws-ebs-csi-driver-node-pv5cf\" (UID: \"022894e1-03ce-4883-91de-722dc423fc87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf"
Sep 18 16:15:58.088642 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088513    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/68f204b6-4215-48a9-973e-d1e84ae4fed5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ls6gx\" (UID: \"68f204b6-4215-48a9-973e-d1e84ae4fed5\") " pod="openshift-multus/multus-additional-cni-plugins-ls6gx"
Sep 18 16:15:58.088642 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088543    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e80a48a0-bdd3-4128-999a-cc2574ced499-cni-binary-copy\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.088642 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088494    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-host-run-netns\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.089157 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088560    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-etc-kubernetes\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.089157 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088578    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-run-ovn\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.089157 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088596    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/022894e1-03ce-4883-91de-722dc423fc87-registration-dir\") pod \"aws-ebs-csi-driver-node-pv5cf\" (UID: \"022894e1-03ce-4883-91de-722dc423fc87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf"
Sep 18 16:15:58.089157 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088604    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11366fd3-c5b8-4bf4-8ba6-217fd3a1b658-host-slash\") pod \"iptables-alerter-h6tt8\" (UID: \"11366fd3-c5b8-4bf4-8ba6-217fd3a1b658\") " pod="openshift-network-operator/iptables-alerter-h6tt8"
Sep 18 16:15:58.089157 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088629    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/68f204b6-4215-48a9-973e-d1e84ae4fed5-cni-binary-copy\") pod \"multus-additional-cni-plugins-ls6gx\" (UID: \"68f204b6-4215-48a9-973e-d1e84ae4fed5\") " pod="openshift-multus/multus-additional-cni-plugins-ls6gx"
Sep 18 16:15:58.089157 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088635    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-run-ovn\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.089157 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088651    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-cnibin\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.089157 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088673    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnnn9\" (UniqueName: \"kubernetes.io/projected/e80a48a0-bdd3-4128-999a-cc2574ced499-kube-api-access-pnnn9\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.089157 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-env-overrides\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.089157 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088724    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97bef555-d008-4f5b-a974-91fbafd1eeb1-kubelet-config\") pod \"global-pull-secret-syncer-cmscw\" (UID: \"97bef555-d008-4f5b-a974-91fbafd1eeb1\") " pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:15:58.089157 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088762    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-etc-sysctl-conf\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.089157 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8c726ad7-c84f-4629-9797-52cfa3327996-agent-certs\") pod \"konnectivity-agent-k6gdw\" (UID: \"8c726ad7-c84f-4629-9797-52cfa3327996\") " pod="kube-system/konnectivity-agent-k6gdw"
Sep 18 16:15:58.089157 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088805    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8c726ad7-c84f-4629-9797-52cfa3327996-konnectivity-ca\") pod \"konnectivity-agent-k6gdw\" (UID: \"8c726ad7-c84f-4629-9797-52cfa3327996\") " pod="kube-system/konnectivity-agent-k6gdw"
Sep 18 16:15:58.089157 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088825    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-host\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.089157 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/022894e1-03ce-4883-91de-722dc423fc87-sys-fs\") pod \"aws-ebs-csi-driver-node-pv5cf\" (UID: \"022894e1-03ce-4883-91de-722dc423fc87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf"
Sep 18 16:15:58.089157 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-host-var-lib-cni-multus\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.089157 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088894    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-var-lib-kubelet\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.089736 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088919    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zvkf\" (UniqueName: \"kubernetes.io/projected/c861f27f-910e-48aa-b4bc-3c20bcd911ef-kube-api-access-4zvkf\") pod \"network-metrics-daemon-tnb7c\" (UID: \"c861f27f-910e-48aa-b4bc-3c20bcd911ef\") " pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:15:58.089736 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088942    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-system-cni-dir\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.089736 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088963    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cfa7efc4-7409-41f5-b891-7ce9e5b6982f-serviceca\") pod \"node-ca-4vmq6\" (UID: \"cfa7efc4-7409-41f5-b891-7ce9e5b6982f\") " pod="openshift-image-registry/node-ca-4vmq6"
Sep 18 16:15:58.089736 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-lib-modules\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.089736 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089009    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkf2q\" (UniqueName: \"kubernetes.io/projected/752ec166-f680-4caf-b569-5ca2d59307dc-kube-api-access-hkf2q\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.089736 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089032    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-host-slash\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.089736 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089046    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-ovnkube-script-lib\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.089736 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089055    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-node-log\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.089736 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089073    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/68f204b6-4215-48a9-973e-d1e84ae4fed5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ls6gx\" (UID: \"68f204b6-4215-48a9-973e-d1e84ae4fed5\") " pod="openshift-multus/multus-additional-cni-plugins-ls6gx"
Sep 18 16:15:58.089736 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e80a48a0-bdd3-4128-999a-cc2574ced499-cni-binary-copy\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.089736 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089122    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/022894e1-03ce-4883-91de-722dc423fc87-sys-fs\") pod \"aws-ebs-csi-driver-node-pv5cf\" (UID: \"022894e1-03ce-4883-91de-722dc423fc87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf"
Sep 18 16:15:58.089736 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089138    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-host-var-lib-cni-multus\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.089736 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089151    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/68f204b6-4215-48a9-973e-d1e84ae4fed5-cni-binary-copy\") pod \"multus-additional-cni-plugins-ls6gx\" (UID: \"68f204b6-4215-48a9-973e-d1e84ae4fed5\") " pod="openshift-multus/multus-additional-cni-plugins-ls6gx"
Sep 18 16:15:58.089736 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089161    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-host-cni-netd\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.089736 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089172    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-cnibin\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.089736 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089186    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-var-lib-kubelet\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.089736 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.088674    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11366fd3-c5b8-4bf4-8ba6-217fd3a1b658-host-slash\") pod \"iptables-alerter-h6tt8\" (UID: \"11366fd3-c5b8-4bf4-8ba6-217fd3a1b658\") " pod="openshift-network-operator/iptables-alerter-h6tt8"
Sep 18 16:15:58.089736 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089079    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-ovnkube-config\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.090511 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089247    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e945222a-603b-4489-a528-2f607d157b14-tmp-dir\") pod \"node-resolver-krchj\" (UID: \"e945222a-603b-4489-a528-2f607d157b14\") " pod="openshift-dns/node-resolver-krchj"
Sep 18 16:15:58.090511 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089261    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97bef555-d008-4f5b-a974-91fbafd1eeb1-kubelet-config\") pod \"global-pull-secret-syncer-cmscw\" (UID: \"97bef555-d008-4f5b-a974-91fbafd1eeb1\") " pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:15:58.090511 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089270    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/022894e1-03ce-4883-91de-722dc423fc87-socket-dir\") pod \"aws-ebs-csi-driver-node-pv5cf\" (UID: \"022894e1-03ce-4883-91de-722dc423fc87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf"
Sep 18 16:15:58.090511 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089294    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-sys\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.090511 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089336    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.090511 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089363    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11366fd3-c5b8-4bf4-8ba6-217fd3a1b658-iptables-alerter-script\") pod \"iptables-alerter-h6tt8\" (UID: \"11366fd3-c5b8-4bf4-8ba6-217fd3a1b658\") " pod="openshift-network-operator/iptables-alerter-h6tt8"
Sep 18 16:15:58.090511 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089389    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bg7f\" (UniqueName: \"kubernetes.io/projected/68f204b6-4215-48a9-973e-d1e84ae4fed5-kube-api-access-7bg7f\") pod \"multus-additional-cni-plugins-ls6gx\" (UID: \"68f204b6-4215-48a9-973e-d1e84ae4fed5\") " pod="openshift-multus/multus-additional-cni-plugins-ls6gx"
Sep 18 16:15:58.090511 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-multus-socket-dir-parent\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.090511 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089437    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-host-cni-bin\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.090511 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089461    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-ovn-node-metrics-cert\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.090511 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089483    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmctj\" (UniqueName: \"kubernetes.io/projected/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-kube-api-access-nmctj\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.090511 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089507    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e945222a-603b-4489-a528-2f607d157b14-hosts-file\") pod \"node-resolver-krchj\" (UID: \"e945222a-603b-4489-a528-2f607d157b14\") " pod="openshift-dns/node-resolver-krchj"
Sep 18 16:15:58.090511 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089535    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-ovnkube-config\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.090511 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089557    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 16:15:58.090511 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089598    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-host-cni-bin\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.090511 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089601    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/022894e1-03ce-4883-91de-722dc423fc87-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pv5cf\" (UID: \"022894e1-03ce-4883-91de-722dc423fc87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf"
Sep 18 16:15:58.090511 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089647    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-node-log\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.091569 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089694    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-multus-socket-dir-parent\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.091569 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089706    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8c726ad7-c84f-4629-9797-52cfa3327996-konnectivity-ca\") pod \"konnectivity-agent-k6gdw\" (UID: \"8c726ad7-c84f-4629-9797-52cfa3327996\") " pod="kube-system/konnectivity-agent-k6gdw"
Sep 18 16:15:58.091569 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089780    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-lib-modules\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.091569 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089543    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/022894e1-03ce-4883-91de-722dc423fc87-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pv5cf\" (UID: \"022894e1-03ce-4883-91de-722dc423fc87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf"
Sep 18 16:15:58.091569 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9jmh\" (UniqueName: \"kubernetes.io/projected/022894e1-03ce-4883-91de-722dc423fc87-kube-api-access-q9jmh\") pod \"aws-ebs-csi-driver-node-pv5cf\" (UID: \"022894e1-03ce-4883-91de-722dc423fc87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf"
Sep 18 16:15:58.091569 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089838    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-system-cni-dir\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.091569 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089855    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xbcv\" (UniqueName: \"kubernetes.io/projected/e945222a-603b-4489-a528-2f607d157b14-kube-api-access-9xbcv\") pod \"node-resolver-krchj\" (UID: \"e945222a-603b-4489-a528-2f607d157b14\") " pod="openshift-dns/node-resolver-krchj"
Sep 18 16:15:58.091569 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089873    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11366fd3-c5b8-4bf4-8ba6-217fd3a1b658-iptables-alerter-script\") pod \"iptables-alerter-h6tt8\" (UID: \"11366fd3-c5b8-4bf4-8ba6-217fd3a1b658\") " pod="openshift-network-operator/iptables-alerter-h6tt8"
Sep 18 16:15:58.091569 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-host-var-lib-kubelet\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.091569 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089908    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-host-run-multus-certs\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.091569 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089916    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-host-var-lib-kubelet\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.091569 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089933    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-etc-kubernetes\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.091569 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089954    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-host-slash\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.091569 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089964    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/752ec166-f680-4caf-b569-5ca2d59307dc-tmp\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.091569 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.089989    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-systemd-units\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.091569 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090017    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-run-openvswitch\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.091569 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090056    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/68f204b6-4215-48a9-973e-d1e84ae4fed5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ls6gx\" (UID: \"68f204b6-4215-48a9-973e-d1e84ae4fed5\") " pod="openshift-multus/multus-additional-cni-plugins-ls6gx"
Sep 18 16:15:58.092412 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090083    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-env-overrides\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.092412 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090128    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-hostroot\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.092412 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090084    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-hostroot\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.092412 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-run-systemd\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.092412 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090193    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/022894e1-03ce-4883-91de-722dc423fc87-socket-dir\") pod \"aws-ebs-csi-driver-node-pv5cf\" (UID: \"022894e1-03ce-4883-91de-722dc423fc87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf"
Sep 18 16:15:58.092412 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090211    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-log-socket\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.092412 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090227    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-host\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.092412 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090236    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/68f204b6-4215-48a9-973e-d1e84ae4fed5-cnibin\") pod \"multus-additional-cni-plugins-ls6gx\" (UID: \"68f204b6-4215-48a9-973e-d1e84ae4fed5\") " pod="openshift-multus/multus-additional-cni-plugins-ls6gx"
Sep 18 16:15:58.092412 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090244    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-sys\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.092412 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090269    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/022894e1-03ce-4883-91de-722dc423fc87-etc-selinux\") pod \"aws-ebs-csi-driver-node-pv5cf\" (UID: \"022894e1-03ce-4883-91de-722dc423fc87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf"
Sep 18 16:15:58.092412 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090296    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-run-systemd\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.092412 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-host-run-k8s-cni-cncf-io\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.092412 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090343    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-host-run-k8s-cni-cncf-io\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.092412 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090367    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr29z\" (UniqueName: \"kubernetes.io/projected/cfa7efc4-7409-41f5-b891-7ce9e5b6982f-kube-api-access-rr29z\") pod \"node-ca-4vmq6\" (UID: \"cfa7efc4-7409-41f5-b891-7ce9e5b6982f\") " pod="openshift-image-registry/node-ca-4vmq6"
Sep 18 16:15:58.092412 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090395    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-etc-openvswitch\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.092412 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090401    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-systemd-units\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.092412 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090421    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/022894e1-03ce-4883-91de-722dc423fc87-etc-selinux\") pod \"aws-ebs-csi-driver-node-pv5cf\" (UID: \"022894e1-03ce-4883-91de-722dc423fc87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf"
Sep 18 16:15:58.092412 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090433    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzt9g\" (UniqueName: \"kubernetes.io/projected/8b313705-71ee-4c64-a650-b308f048dc2a-kube-api-access-kzt9g\") pod \"network-check-target-fs4cj\" (UID: \"8b313705-71ee-4c64-a650-b308f048dc2a\") " pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:15:58.093225 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090439    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-etc-openvswitch\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.093225 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090462    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-run-openvswitch\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.093225 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090482    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-log-socket\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.093225 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090489    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e80a48a0-bdd3-4128-999a-cc2574ced499-multus-daemon-config\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.093225 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090516    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-etc-sysctl-d\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.093225 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090761    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-etc-systemd\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.093225 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-run\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.093225 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090812    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-var-lib-openvswitch\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.093225 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090849    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-var-lib-openvswitch\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.093225 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090885    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/68f204b6-4215-48a9-973e-d1e84ae4fed5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ls6gx\" (UID: \"68f204b6-4215-48a9-973e-d1e84ae4fed5\") " pod="openshift-multus/multus-additional-cni-plugins-ls6gx"
Sep 18 16:15:58.093225 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090889    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-host-run-ovn-kubernetes\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.093225 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090951    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e945222a-603b-4489-a528-2f607d157b14-tmp-dir\") pod \"node-resolver-krchj\" (UID: \"e945222a-603b-4489-a528-2f607d157b14\") " pod="openshift-dns/node-resolver-krchj"
Sep 18 16:15:58.093225 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090997    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-etc-sysctl-d\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.093225 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.091028    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-etc-systemd\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.093225 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.091042    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.093225 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.091073    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-run\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.093225 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.090848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-host-run-ovn-kubernetes\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.094057 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.091089    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-host-run-multus-certs\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.094057 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.091105    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-etc-kubernetes\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.094057 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.091109    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c861f27f-910e-48aa-b4bc-3c20bcd911ef-metrics-certs\") pod \"network-metrics-daemon-tnb7c\" (UID: \"c861f27f-910e-48aa-b4bc-3c20bcd911ef\") " pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:15:58.094057 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.091135    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/68f204b6-4215-48a9-973e-d1e84ae4fed5-cnibin\") pod \"multus-additional-cni-plugins-ls6gx\" (UID: \"68f204b6-4215-48a9-973e-d1e84ae4fed5\") " pod="openshift-multus/multus-additional-cni-plugins-ls6gx"
Sep 18 16:15:58.094057 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.091146    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e945222a-603b-4489-a528-2f607d157b14-hosts-file\") pod \"node-resolver-krchj\" (UID: \"e945222a-603b-4489-a528-2f607d157b14\") " pod="openshift-dns/node-resolver-krchj"
Sep 18 16:15:58.094057 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.091149    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-multus-conf-dir\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.094057 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.091178    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97bef555-d008-4f5b-a974-91fbafd1eeb1-dbus\") pod \"global-pull-secret-syncer-cmscw\" (UID: \"97bef555-d008-4f5b-a974-91fbafd1eeb1\") " pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:15:58.094057 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.091205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97bef555-d008-4f5b-a974-91fbafd1eeb1-original-pull-secret\") pod \"global-pull-secret-syncer-cmscw\" (UID: \"97bef555-d008-4f5b-a974-91fbafd1eeb1\") " pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:15:58.094057 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.091231    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cfa7efc4-7409-41f5-b891-7ce9e5b6982f-host\") pod \"node-ca-4vmq6\" (UID: \"cfa7efc4-7409-41f5-b891-7ce9e5b6982f\") " pod="openshift-image-registry/node-ca-4vmq6"
Sep 18 16:15:58.094057 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.091235    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97bef555-d008-4f5b-a974-91fbafd1eeb1-dbus\") pod \"global-pull-secret-syncer-cmscw\" (UID: \"97bef555-d008-4f5b-a974-91fbafd1eeb1\") " pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:15:58.094057 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.091245    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/752ec166-f680-4caf-b569-5ca2d59307dc-etc-sysctl-conf\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.094057 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.091254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-host-var-lib-cni-bin\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.094057 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.091277    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/752ec166-f680-4caf-b569-5ca2d59307dc-etc-tuned\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.094057 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.091287    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cfa7efc4-7409-41f5-b891-7ce9e5b6982f-host\") pod \"node-ca-4vmq6\" (UID: \"cfa7efc4-7409-41f5-b891-7ce9e5b6982f\") " pod="openshift-image-registry/node-ca-4vmq6"
Sep 18 16:15:58.094057 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.091182    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-multus-conf-dir\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.094057 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.091322    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rsqq\" (UniqueName: \"kubernetes.io/projected/11366fd3-c5b8-4bf4-8ba6-217fd3a1b658-kube-api-access-9rsqq\") pod \"iptables-alerter-h6tt8\" (UID: \"11366fd3-c5b8-4bf4-8ba6-217fd3a1b658\") " pod="openshift-network-operator/iptables-alerter-h6tt8"
Sep 18 16:15:58.094057 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.091352    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e80a48a0-bdd3-4128-999a-cc2574ced499-host-var-lib-cni-bin\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.094057 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:58.091324    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:15:58.094960 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:58.091434    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97bef555-d008-4f5b-a974-91fbafd1eeb1-original-pull-secret podName:97bef555-d008-4f5b-a974-91fbafd1eeb1 nodeName:}" failed. No retries permitted until 2026-09-18 16:15:58.591413236 +0000 UTC m=+4.679728874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97bef555-d008-4f5b-a974-91fbafd1eeb1-original-pull-secret") pod "global-pull-secret-syncer-cmscw" (UID: "97bef555-d008-4f5b-a974-91fbafd1eeb1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:15:58.094960 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.091458    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e80a48a0-bdd3-4128-999a-cc2574ced499-multus-daemon-config\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.094960 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.091538    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cfa7efc4-7409-41f5-b891-7ce9e5b6982f-serviceca\") pod \"node-ca-4vmq6\" (UID: \"cfa7efc4-7409-41f5-b891-7ce9e5b6982f\") " pod="openshift-image-registry/node-ca-4vmq6"
Sep 18 16:15:58.094960 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:58.091551    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:15:58.094960 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:58.091610    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c861f27f-910e-48aa-b4bc-3c20bcd911ef-metrics-certs podName:c861f27f-910e-48aa-b4bc-3c20bcd911ef nodeName:}" failed. No retries permitted until 2026-09-18 16:15:58.591587785 +0000 UTC m=+4.679903410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c861f27f-910e-48aa-b4bc-3c20bcd911ef-metrics-certs") pod "network-metrics-daemon-tnb7c" (UID: "c861f27f-910e-48aa-b4bc-3c20bcd911ef") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:15:58.094960 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.093758    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/752ec166-f680-4caf-b569-5ca2d59307dc-etc-tuned\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.094960 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.093814    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/752ec166-f680-4caf-b569-5ca2d59307dc-tmp\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.094960 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.093941    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-ovn-node-metrics-cert\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.094960 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.094571    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8c726ad7-c84f-4629-9797-52cfa3327996-agent-certs\") pod \"konnectivity-agent-k6gdw\" (UID: \"8c726ad7-c84f-4629-9797-52cfa3327996\") " pod="kube-system/konnectivity-agent-k6gdw"
Sep 18 16:15:58.101882 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:58.101716    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:15:58.101882 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:58.101747    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:15:58.101882 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:58.101761    2580 projected.go:196] Error preparing data for projected volume kube-api-access-kzt9g for pod openshift-network-diagnostics/network-check-target-fs4cj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:15:58.101882 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:58.101824    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b313705-71ee-4c64-a650-b308f048dc2a-kube-api-access-kzt9g podName:8b313705-71ee-4c64-a650-b308f048dc2a nodeName:}" failed. No retries permitted until 2026-09-18 16:15:58.601807246 +0000 UTC m=+4.690122885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kzt9g" (UniqueName: "kubernetes.io/projected/8b313705-71ee-4c64-a650-b308f048dc2a-kube-api-access-kzt9g") pod "network-check-target-fs4cj" (UID: "8b313705-71ee-4c64-a650-b308f048dc2a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:15:58.104724 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.104695    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9jmh\" (UniqueName: \"kubernetes.io/projected/022894e1-03ce-4883-91de-722dc423fc87-kube-api-access-q9jmh\") pod \"aws-ebs-csi-driver-node-pv5cf\" (UID: \"022894e1-03ce-4883-91de-722dc423fc87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf"
Sep 18 16:15:58.104813 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.104711    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkf2q\" (UniqueName: \"kubernetes.io/projected/752ec166-f680-4caf-b569-5ca2d59307dc-kube-api-access-hkf2q\") pod \"tuned-96w8b\" (UID: \"752ec166-f680-4caf-b569-5ca2d59307dc\") " pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.104911 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.104861    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zvkf\" (UniqueName: \"kubernetes.io/projected/c861f27f-910e-48aa-b4bc-3c20bcd911ef-kube-api-access-4zvkf\") pod \"network-metrics-daemon-tnb7c\" (UID: \"c861f27f-910e-48aa-b4bc-3c20bcd911ef\") " pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:15:58.105270 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.105247    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xbcv\" (UniqueName: \"kubernetes.io/projected/e945222a-603b-4489-a528-2f607d157b14-kube-api-access-9xbcv\") pod \"node-resolver-krchj\" (UID: \"e945222a-603b-4489-a528-2f607d157b14\") " pod="openshift-dns/node-resolver-krchj"
Sep 18 16:15:58.105784 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.105734    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bg7f\" (UniqueName: \"kubernetes.io/projected/68f204b6-4215-48a9-973e-d1e84ae4fed5-kube-api-access-7bg7f\") pod \"multus-additional-cni-plugins-ls6gx\" (UID: \"68f204b6-4215-48a9-973e-d1e84ae4fed5\") " pod="openshift-multus/multus-additional-cni-plugins-ls6gx"
Sep 18 16:15:58.106713 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.106521    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnnn9\" (UniqueName: \"kubernetes.io/projected/e80a48a0-bdd3-4128-999a-cc2574ced499-kube-api-access-pnnn9\") pod \"multus-8tkmt\" (UID: \"e80a48a0-bdd3-4128-999a-cc2574ced499\") " pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.106815 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.106750    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr29z\" (UniqueName: \"kubernetes.io/projected/cfa7efc4-7409-41f5-b891-7ce9e5b6982f-kube-api-access-rr29z\") pod \"node-ca-4vmq6\" (UID: \"cfa7efc4-7409-41f5-b891-7ce9e5b6982f\") " pod="openshift-image-registry/node-ca-4vmq6"
Sep 18 16:15:58.107123 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.107103    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rsqq\" (UniqueName: \"kubernetes.io/projected/11366fd3-c5b8-4bf4-8ba6-217fd3a1b658-kube-api-access-9rsqq\") pod \"iptables-alerter-h6tt8\" (UID: \"11366fd3-c5b8-4bf4-8ba6-217fd3a1b658\") " pod="openshift-network-operator/iptables-alerter-h6tt8"
Sep 18 16:15:58.107500 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.107482    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmctj\" (UniqueName: \"kubernetes.io/projected/96f857f8-22b6-4d0b-b6fd-826b9b507ef8-kube-api-access-nmctj\") pod \"ovnkube-node-gw4fg\" (UID: \"96f857f8-22b6-4d0b-b6fd-826b9b507ef8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.121016 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.120927    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-110.ec2.internal" event={"ID":"43c30dee52b2988542f60a71f377520c","Type":"ContainerStarted","Data":"43a70dd30b4642f52575931e60494df4ddd16ddc8205b33cc7f3c933d942e5a4"}
Sep 18 16:15:58.122250 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.122222    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-110.ec2.internal" event={"ID":"b8e36c319ff90245c5f11900bd31f47e","Type":"ContainerStarted","Data":"6e32c0b300bd33cba0eafae0e4d21a5965b13ba769b267eb482d94ad1a7d62ba"}
Sep 18 16:15:58.251871 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.251802    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ls6gx"
Sep 18 16:15:58.259882 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.259855    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf"
Sep 18 16:15:58.269955 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.269929    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-96w8b"
Sep 18 16:15:58.276755 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.276735    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8tkmt"
Sep 18 16:15:58.286364 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.286341    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4vmq6"
Sep 18 16:15:58.293041 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.293021    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:15:58.301567 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.301546    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-krchj"
Sep 18 16:15:58.310044 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.310026    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k6gdw"
Sep 18 16:15:58.316592 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.316573    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h6tt8"
Sep 18 16:15:58.594573 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.594486    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c861f27f-910e-48aa-b4bc-3c20bcd911ef-metrics-certs\") pod \"network-metrics-daemon-tnb7c\" (UID: \"c861f27f-910e-48aa-b4bc-3c20bcd911ef\") " pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:15:58.594573 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.594538    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97bef555-d008-4f5b-a974-91fbafd1eeb1-original-pull-secret\") pod \"global-pull-secret-syncer-cmscw\" (UID: \"97bef555-d008-4f5b-a974-91fbafd1eeb1\") " pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:15:58.594792 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:58.594600    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:15:58.594792 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:58.594658    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:15:58.594792 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:58.594674    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c861f27f-910e-48aa-b4bc-3c20bcd911ef-metrics-certs podName:c861f27f-910e-48aa-b4bc-3c20bcd911ef nodeName:}" failed. No retries permitted until 2026-09-18 16:15:59.594653168 +0000 UTC m=+5.682968805 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c861f27f-910e-48aa-b4bc-3c20bcd911ef-metrics-certs") pod "network-metrics-daemon-tnb7c" (UID: "c861f27f-910e-48aa-b4bc-3c20bcd911ef") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:15:58.594792 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:58.594719    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97bef555-d008-4f5b-a974-91fbafd1eeb1-original-pull-secret podName:97bef555-d008-4f5b-a974-91fbafd1eeb1 nodeName:}" failed. No retries permitted until 2026-09-18 16:15:59.594699657 +0000 UTC m=+5.683015279 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97bef555-d008-4f5b-a974-91fbafd1eeb1-original-pull-secret") pod "global-pull-secret-syncer-cmscw" (UID: "97bef555-d008-4f5b-a974-91fbafd1eeb1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:15:58.695100 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:58.695064    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzt9g\" (UniqueName: \"kubernetes.io/projected/8b313705-71ee-4c64-a650-b308f048dc2a-kube-api-access-kzt9g\") pod \"network-check-target-fs4cj\" (UID: \"8b313705-71ee-4c64-a650-b308f048dc2a\") " pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:15:58.695281 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:58.695243    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:15:58.695281 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:58.695272    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:15:58.695407 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:58.695285    2580 projected.go:196] Error preparing data for projected volume kube-api-access-kzt9g for pod openshift-network-diagnostics/network-check-target-fs4cj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:15:58.695407 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:58.695357    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b313705-71ee-4c64-a650-b308f048dc2a-kube-api-access-kzt9g podName:8b313705-71ee-4c64-a650-b308f048dc2a nodeName:}" failed. No retries permitted until 2026-09-18 16:15:59.695343117 +0000 UTC m=+5.783658747 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kzt9g" (UniqueName: "kubernetes.io/projected/8b313705-71ee-4c64-a650-b308f048dc2a-kube-api-access-kzt9g") pod "network-check-target-fs4cj" (UID: "8b313705-71ee-4c64-a650-b308f048dc2a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:15:59.022075 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:59.022005    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 16:10:58 +0000 UTC" deadline="2028-05-22 15:34:20.205429024 +0000 UTC"
Sep 18 16:15:59.022075 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:59.022071    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14687h18m21.183362294s"
Sep 18 16:15:59.116892 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:59.116855    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:15:59.117068 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:59.116909    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:15:59.117068 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:59.116990    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs4cj" podUID="8b313705-71ee-4c64-a650-b308f048dc2a"
Sep 18 16:15:59.117174 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:59.117131    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tnb7c" podUID="c861f27f-910e-48aa-b4bc-3c20bcd911ef"
Sep 18 16:15:59.178325 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:59.178274    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f204b6_4215_48a9_973e_d1e84ae4fed5.slice/crio-18d61f1ece109ba20ac95897220445b476e8a0797d522e3e2d31e983f6214c8e WatchSource:0}: Error finding container 18d61f1ece109ba20ac95897220445b476e8a0797d522e3e2d31e983f6214c8e: Status 404 returned error can't find the container with id 18d61f1ece109ba20ac95897220445b476e8a0797d522e3e2d31e983f6214c8e
Sep 18 16:15:59.179577 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:59.179554    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode945222a_603b_4489_a528_2f607d157b14.slice/crio-62484fb4f3af61df8ee7e3d316f14cf8f4e25a32235415d565d1b82d15bea53b WatchSource:0}: Error finding container 62484fb4f3af61df8ee7e3d316f14cf8f4e25a32235415d565d1b82d15bea53b: Status 404 returned error can't find the container with id 62484fb4f3af61df8ee7e3d316f14cf8f4e25a32235415d565d1b82d15bea53b
Sep 18 16:15:59.180562 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:59.180537    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c726ad7_c84f_4629_9797_52cfa3327996.slice/crio-b6e0ab4ef4d2134bc3c222863d18bf17d48fd010081a622c212fb84319122919 WatchSource:0}: Error finding container b6e0ab4ef4d2134bc3c222863d18bf17d48fd010081a622c212fb84319122919: Status 404 returned error can't find the container with id b6e0ab4ef4d2134bc3c222863d18bf17d48fd010081a622c212fb84319122919
Sep 18 16:15:59.184743 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:59.184724    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa7efc4_7409_41f5_b891_7ce9e5b6982f.slice/crio-a5032f3013a55a398051a5eeee7f4a9f2f900e203622cec02ee90510114243ec WatchSource:0}: Error finding container a5032f3013a55a398051a5eeee7f4a9f2f900e203622cec02ee90510114243ec: Status 404 returned error can't find the container with id a5032f3013a55a398051a5eeee7f4a9f2f900e203622cec02ee90510114243ec
Sep 18 16:15:59.185727 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:59.185706    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752ec166_f680_4caf_b569_5ca2d59307dc.slice/crio-4d1de70d997f5f30973673bccd34c67a758ebd7654debe8fe431e53ab593bdb0 WatchSource:0}: Error finding container 4d1de70d997f5f30973673bccd34c67a758ebd7654debe8fe431e53ab593bdb0: Status 404 returned error can't find the container with id 4d1de70d997f5f30973673bccd34c67a758ebd7654debe8fe431e53ab593bdb0
Sep 18 16:15:59.186762 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:59.186732    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11366fd3_c5b8_4bf4_8ba6_217fd3a1b658.slice/crio-db26aa5067c3e80d89a4171ef698ea9e3397e409d882754039c2aadea3ff68ec WatchSource:0}: Error finding container db26aa5067c3e80d89a4171ef698ea9e3397e409d882754039c2aadea3ff68ec: Status 404 returned error can't find the container with id db26aa5067c3e80d89a4171ef698ea9e3397e409d882754039c2aadea3ff68ec
Sep 18 16:15:59.188102 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:59.187847    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f857f8_22b6_4d0b_b6fd_826b9b507ef8.slice/crio-4d698080fd297ab6df7520b344b3d73bb075fd72151c27528f0f12814647bdb5 WatchSource:0}: Error finding container 4d698080fd297ab6df7520b344b3d73bb075fd72151c27528f0f12814647bdb5: Status 404 returned error can't find the container with id 4d698080fd297ab6df7520b344b3d73bb075fd72151c27528f0f12814647bdb5
Sep 18 16:15:59.188704 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:59.188682    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod022894e1_03ce_4883_91de_722dc423fc87.slice/crio-917e781666ddd9cbca4dada84666950ab8b937381b4d91ea0d5dff9abe967663 WatchSource:0}: Error finding container 917e781666ddd9cbca4dada84666950ab8b937381b4d91ea0d5dff9abe967663: Status 404 returned error can't find the container with id 917e781666ddd9cbca4dada84666950ab8b937381b4d91ea0d5dff9abe967663
Sep 18 16:15:59.190608 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:15:59.190590    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode80a48a0_bdd3_4128_999a_cc2574ced499.slice/crio-3c6057fe22330c2e817b9accbbcb4a8d029bbf55547ffbeaaf11c101ee2b6a72 WatchSource:0}: Error finding container 3c6057fe22330c2e817b9accbbcb4a8d029bbf55547ffbeaaf11c101ee2b6a72: Status 404 returned error can't find the container with id 3c6057fe22330c2e817b9accbbcb4a8d029bbf55547ffbeaaf11c101ee2b6a72
Sep 18 16:15:59.601997 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:59.601756    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c861f27f-910e-48aa-b4bc-3c20bcd911ef-metrics-certs\") pod \"network-metrics-daemon-tnb7c\" (UID: \"c861f27f-910e-48aa-b4bc-3c20bcd911ef\") " pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:15:59.602170 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:59.602020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97bef555-d008-4f5b-a974-91fbafd1eeb1-original-pull-secret\") pod \"global-pull-secret-syncer-cmscw\" (UID: \"97bef555-d008-4f5b-a974-91fbafd1eeb1\") " pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:15:59.602170 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:59.601872    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:15:59.602170 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:59.602124    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c861f27f-910e-48aa-b4bc-3c20bcd911ef-metrics-certs podName:c861f27f-910e-48aa-b4bc-3c20bcd911ef nodeName:}" failed. No retries permitted until 2026-09-18 16:16:01.602097899 +0000 UTC m=+7.690413532 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c861f27f-910e-48aa-b4bc-3c20bcd911ef-metrics-certs") pod "network-metrics-daemon-tnb7c" (UID: "c861f27f-910e-48aa-b4bc-3c20bcd911ef") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:15:59.602170 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:59.602140    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:15:59.602417 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:59.602194    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97bef555-d008-4f5b-a974-91fbafd1eeb1-original-pull-secret podName:97bef555-d008-4f5b-a974-91fbafd1eeb1 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:01.602178665 +0000 UTC m=+7.690494289 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97bef555-d008-4f5b-a974-91fbafd1eeb1-original-pull-secret") pod "global-pull-secret-syncer-cmscw" (UID: "97bef555-d008-4f5b-a974-91fbafd1eeb1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:15:59.703363 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:15:59.703325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzt9g\" (UniqueName: \"kubernetes.io/projected/8b313705-71ee-4c64-a650-b308f048dc2a-kube-api-access-kzt9g\") pod \"network-check-target-fs4cj\" (UID: \"8b313705-71ee-4c64-a650-b308f048dc2a\") " pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:15:59.703564 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:59.703547    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:15:59.703654 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:59.703572    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:15:59.703654 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:59.703584    2580 projected.go:196] Error preparing data for projected volume kube-api-access-kzt9g for pod openshift-network-diagnostics/network-check-target-fs4cj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:15:59.703654 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:15:59.703643    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b313705-71ee-4c64-a650-b308f048dc2a-kube-api-access-kzt9g podName:8b313705-71ee-4c64-a650-b308f048dc2a nodeName:}" failed. No retries permitted until 2026-09-18 16:16:01.70362643 +0000 UTC m=+7.791942062 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kzt9g" (UniqueName: "kubernetes.io/projected/8b313705-71ee-4c64-a650-b308f048dc2a-kube-api-access-kzt9g") pod "network-check-target-fs4cj" (UID: "8b313705-71ee-4c64-a650-b308f048dc2a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:00.022917 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:00.022799    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 16:10:58 +0000 UTC" deadline="2028-06-29 06:52:01.089181828 +0000 UTC"
Sep 18 16:16:00.022917 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:00.022842    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15590h36m1.066344623s"
Sep 18 16:16:00.117761 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:00.117250    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:00.117761 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:00.117401    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmscw" podUID="97bef555-d008-4f5b-a974-91fbafd1eeb1"
Sep 18 16:16:00.137006 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:00.136968    2580 generic.go:356] "Generic (PLEG): container finished" podID="43c30dee52b2988542f60a71f377520c" containerID="5e5802625679422b8de1ecd04fe341589ca3d1dbb9cf16ca68cb493240a5d06b" exitCode=0
Sep 18 16:16:00.137190 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:00.137163    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-110.ec2.internal" event={"ID":"43c30dee52b2988542f60a71f377520c","Type":"ContainerDied","Data":"5e5802625679422b8de1ecd04fe341589ca3d1dbb9cf16ca68cb493240a5d06b"}
Sep 18 16:16:00.151616 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:00.151044    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-110.ec2.internal" event={"ID":"b8e36c319ff90245c5f11900bd31f47e","Type":"ContainerStarted","Data":"e73ba25fcbca667bb3149362dd314c3c69de157746a599bdb5645406fbf7e909"}
Sep 18 16:16:00.155861 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:00.155188    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf" event={"ID":"022894e1-03ce-4883-91de-722dc423fc87","Type":"ContainerStarted","Data":"917e781666ddd9cbca4dada84666950ab8b937381b4d91ea0d5dff9abe967663"}
Sep 18 16:16:00.167101 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:00.167070    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg" event={"ID":"96f857f8-22b6-4d0b-b6fd-826b9b507ef8","Type":"ContainerStarted","Data":"4d698080fd297ab6df7520b344b3d73bb075fd72151c27528f0f12814647bdb5"}
Sep 18 16:16:00.177368 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:00.176937    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-110.ec2.internal" podStartSLOduration=3.176918181 podStartE2EDuration="3.176918181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:00.176001857 +0000 UTC m=+6.264317499" watchObservedRunningTime="2026-09-18 16:16:00.176918181 +0000 UTC m=+6.265233822"
Sep 18 16:16:00.178909 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:00.178855    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h6tt8" event={"ID":"11366fd3-c5b8-4bf4-8ba6-217fd3a1b658","Type":"ContainerStarted","Data":"db26aa5067c3e80d89a4171ef698ea9e3397e409d882754039c2aadea3ff68ec"}
Sep 18 16:16:00.185169 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:00.185106    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k6gdw" event={"ID":"8c726ad7-c84f-4629-9797-52cfa3327996","Type":"ContainerStarted","Data":"b6e0ab4ef4d2134bc3c222863d18bf17d48fd010081a622c212fb84319122919"}
Sep 18 16:16:00.197709 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:00.197628    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ls6gx" event={"ID":"68f204b6-4215-48a9-973e-d1e84ae4fed5","Type":"ContainerStarted","Data":"18d61f1ece109ba20ac95897220445b476e8a0797d522e3e2d31e983f6214c8e"}
Sep 18 16:16:00.203576 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:00.203538    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8tkmt" event={"ID":"e80a48a0-bdd3-4128-999a-cc2574ced499","Type":"ContainerStarted","Data":"3c6057fe22330c2e817b9accbbcb4a8d029bbf55547ffbeaaf11c101ee2b6a72"}
Sep 18 16:16:00.225476 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:00.225435    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-96w8b" event={"ID":"752ec166-f680-4caf-b569-5ca2d59307dc","Type":"ContainerStarted","Data":"4d1de70d997f5f30973673bccd34c67a758ebd7654debe8fe431e53ab593bdb0"}
Sep 18 16:16:00.238472 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:00.238442    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4vmq6" event={"ID":"cfa7efc4-7409-41f5-b891-7ce9e5b6982f","Type":"ContainerStarted","Data":"a5032f3013a55a398051a5eeee7f4a9f2f900e203622cec02ee90510114243ec"}
Sep 18 16:16:00.253417 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:00.253384    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-krchj" event={"ID":"e945222a-603b-4489-a528-2f607d157b14","Type":"ContainerStarted","Data":"62484fb4f3af61df8ee7e3d316f14cf8f4e25a32235415d565d1b82d15bea53b"}
Sep 18 16:16:01.116648 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:01.116617    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:01.117101 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:01.116740    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs4cj" podUID="8b313705-71ee-4c64-a650-b308f048dc2a"
Sep 18 16:16:01.117171 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:01.117130    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:01.117330 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:01.117237    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tnb7c" podUID="c861f27f-910e-48aa-b4bc-3c20bcd911ef"
Sep 18 16:16:01.281760 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:01.281726    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-110.ec2.internal" event={"ID":"43c30dee52b2988542f60a71f377520c","Type":"ContainerStarted","Data":"6bb77417808cd1c27dcb1b5e49fd812b35837cef3b0b365e7ad67cf736bde7ee"}
Sep 18 16:16:01.619194 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:01.618797    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c861f27f-910e-48aa-b4bc-3c20bcd911ef-metrics-certs\") pod \"network-metrics-daemon-tnb7c\" (UID: \"c861f27f-910e-48aa-b4bc-3c20bcd911ef\") " pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:01.619194 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:01.618847    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97bef555-d008-4f5b-a974-91fbafd1eeb1-original-pull-secret\") pod \"global-pull-secret-syncer-cmscw\" (UID: \"97bef555-d008-4f5b-a974-91fbafd1eeb1\") " pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:01.619194 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:01.618974    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:01.619194 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:01.618974    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:01.619194 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:01.619037    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97bef555-d008-4f5b-a974-91fbafd1eeb1-original-pull-secret podName:97bef555-d008-4f5b-a974-91fbafd1eeb1 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:05.619019819 +0000 UTC m=+11.707335439 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97bef555-d008-4f5b-a974-91fbafd1eeb1-original-pull-secret") pod "global-pull-secret-syncer-cmscw" (UID: "97bef555-d008-4f5b-a974-91fbafd1eeb1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:01.619194 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:01.619056    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c861f27f-910e-48aa-b4bc-3c20bcd911ef-metrics-certs podName:c861f27f-910e-48aa-b4bc-3c20bcd911ef nodeName:}" failed. No retries permitted until 2026-09-18 16:16:05.619046382 +0000 UTC m=+11.707362008 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c861f27f-910e-48aa-b4bc-3c20bcd911ef-metrics-certs") pod "network-metrics-daemon-tnb7c" (UID: "c861f27f-910e-48aa-b4bc-3c20bcd911ef") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:01.720397 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:01.720358    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzt9g\" (UniqueName: \"kubernetes.io/projected/8b313705-71ee-4c64-a650-b308f048dc2a-kube-api-access-kzt9g\") pod \"network-check-target-fs4cj\" (UID: \"8b313705-71ee-4c64-a650-b308f048dc2a\") " pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:01.721096 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:01.720580    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:01.721096 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:01.720603    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:01.721096 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:01.720616    2580 projected.go:196] Error preparing data for projected volume kube-api-access-kzt9g for pod openshift-network-diagnostics/network-check-target-fs4cj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:01.721096 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:01.720672    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b313705-71ee-4c64-a650-b308f048dc2a-kube-api-access-kzt9g podName:8b313705-71ee-4c64-a650-b308f048dc2a nodeName:}" failed. No retries permitted until 2026-09-18 16:16:05.720655252 +0000 UTC m=+11.808970875 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kzt9g" (UniqueName: "kubernetes.io/projected/8b313705-71ee-4c64-a650-b308f048dc2a-kube-api-access-kzt9g") pod "network-check-target-fs4cj" (UID: "8b313705-71ee-4c64-a650-b308f048dc2a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:02.116515 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:02.116407    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:02.116691 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:02.116547    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmscw" podUID="97bef555-d008-4f5b-a974-91fbafd1eeb1"
Sep 18 16:16:03.117260 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:03.117223    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:03.117750 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:03.117388    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs4cj" podUID="8b313705-71ee-4c64-a650-b308f048dc2a"
Sep 18 16:16:03.117750 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:03.117437    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:03.117750 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:03.117564    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tnb7c" podUID="c861f27f-910e-48aa-b4bc-3c20bcd911ef"
Sep 18 16:16:04.117152 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:04.117119    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:04.117358 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:04.117254    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmscw" podUID="97bef555-d008-4f5b-a974-91fbafd1eeb1"
Sep 18 16:16:05.117147 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:05.116681    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:05.117147 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:05.116693    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:05.117147 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:05.116807    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs4cj" podUID="8b313705-71ee-4c64-a650-b308f048dc2a"
Sep 18 16:16:05.117147 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:05.116852    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tnb7c" podUID="c861f27f-910e-48aa-b4bc-3c20bcd911ef"
Sep 18 16:16:05.655203 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:05.654972    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c861f27f-910e-48aa-b4bc-3c20bcd911ef-metrics-certs\") pod \"network-metrics-daemon-tnb7c\" (UID: \"c861f27f-910e-48aa-b4bc-3c20bcd911ef\") " pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:05.655203 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:05.655030    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97bef555-d008-4f5b-a974-91fbafd1eeb1-original-pull-secret\") pod \"global-pull-secret-syncer-cmscw\" (UID: \"97bef555-d008-4f5b-a974-91fbafd1eeb1\") " pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:05.655203 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:05.655123    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:05.655203 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:05.655145    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:05.655203 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:05.655204    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c861f27f-910e-48aa-b4bc-3c20bcd911ef-metrics-certs podName:c861f27f-910e-48aa-b4bc-3c20bcd911ef nodeName:}" failed. No retries permitted until 2026-09-18 16:16:13.655183254 +0000 UTC m=+19.743498890 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c861f27f-910e-48aa-b4bc-3c20bcd911ef-metrics-certs") pod "network-metrics-daemon-tnb7c" (UID: "c861f27f-910e-48aa-b4bc-3c20bcd911ef") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:05.655616 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:05.655224    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97bef555-d008-4f5b-a974-91fbafd1eeb1-original-pull-secret podName:97bef555-d008-4f5b-a974-91fbafd1eeb1 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:13.65521382 +0000 UTC m=+19.743529441 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97bef555-d008-4f5b-a974-91fbafd1eeb1-original-pull-secret") pod "global-pull-secret-syncer-cmscw" (UID: "97bef555-d008-4f5b-a974-91fbafd1eeb1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:05.756272 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:05.756230    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzt9g\" (UniqueName: \"kubernetes.io/projected/8b313705-71ee-4c64-a650-b308f048dc2a-kube-api-access-kzt9g\") pod \"network-check-target-fs4cj\" (UID: \"8b313705-71ee-4c64-a650-b308f048dc2a\") " pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:05.756496 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:05.756477    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:05.756551 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:05.756506    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:05.756551 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:05.756519    2580 projected.go:196] Error preparing data for projected volume kube-api-access-kzt9g for pod openshift-network-diagnostics/network-check-target-fs4cj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:05.756641 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:05.756589    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b313705-71ee-4c64-a650-b308f048dc2a-kube-api-access-kzt9g podName:8b313705-71ee-4c64-a650-b308f048dc2a nodeName:}" failed. No retries permitted until 2026-09-18 16:16:13.756566641 +0000 UTC m=+19.844882282 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kzt9g" (UniqueName: "kubernetes.io/projected/8b313705-71ee-4c64-a650-b308f048dc2a-kube-api-access-kzt9g") pod "network-check-target-fs4cj" (UID: "8b313705-71ee-4c64-a650-b308f048dc2a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:06.118102 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:06.118065    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:06.118629 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:06.118196    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmscw" podUID="97bef555-d008-4f5b-a974-91fbafd1eeb1"
Sep 18 16:16:07.116575 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:07.116540    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:07.116731 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:07.116669    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs4cj" podUID="8b313705-71ee-4c64-a650-b308f048dc2a"
Sep 18 16:16:07.116804 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:07.116540    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:07.116967 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:07.116925    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tnb7c" podUID="c861f27f-910e-48aa-b4bc-3c20bcd911ef"
Sep 18 16:16:08.116988 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:08.116928    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:08.117494 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:08.117073    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmscw" podUID="97bef555-d008-4f5b-a974-91fbafd1eeb1"
Sep 18 16:16:09.116612 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:09.116581    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:09.116818 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:09.116682    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs4cj" podUID="8b313705-71ee-4c64-a650-b308f048dc2a"
Sep 18 16:16:09.116818 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:09.116736    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:09.116887 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:09.116826    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tnb7c" podUID="c861f27f-910e-48aa-b4bc-3c20bcd911ef"
Sep 18 16:16:10.116551 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:10.116522    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:10.116969 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:10.116651    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmscw" podUID="97bef555-d008-4f5b-a974-91fbafd1eeb1"
Sep 18 16:16:11.116696 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:11.116663    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:11.117109 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:11.116785    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs4cj" podUID="8b313705-71ee-4c64-a650-b308f048dc2a"
Sep 18 16:16:11.117109 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:11.116846    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:11.117109 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:11.116947    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tnb7c" podUID="c861f27f-910e-48aa-b4bc-3c20bcd911ef"
Sep 18 16:16:12.116626 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:12.116580    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:12.116814 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:12.116709    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmscw" podUID="97bef555-d008-4f5b-a974-91fbafd1eeb1"
Sep 18 16:16:13.117268 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:13.117236    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:13.117268 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:13.117261    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:13.117763 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:13.117365    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs4cj" podUID="8b313705-71ee-4c64-a650-b308f048dc2a"
Sep 18 16:16:13.117763 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:13.117491    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tnb7c" podUID="c861f27f-910e-48aa-b4bc-3c20bcd911ef"
Sep 18 16:16:13.713348 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:13.713311    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c861f27f-910e-48aa-b4bc-3c20bcd911ef-metrics-certs\") pod \"network-metrics-daemon-tnb7c\" (UID: \"c861f27f-910e-48aa-b4bc-3c20bcd911ef\") " pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:13.713525 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:13.713357    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97bef555-d008-4f5b-a974-91fbafd1eeb1-original-pull-secret\") pod \"global-pull-secret-syncer-cmscw\" (UID: \"97bef555-d008-4f5b-a974-91fbafd1eeb1\") " pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:13.713525 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:13.713457    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:13.713525 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:13.713469    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:13.713641 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:13.713530    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c861f27f-910e-48aa-b4bc-3c20bcd911ef-metrics-certs podName:c861f27f-910e-48aa-b4bc-3c20bcd911ef nodeName:}" failed. No retries permitted until 2026-09-18 16:16:29.713510197 +0000 UTC m=+35.801825835 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c861f27f-910e-48aa-b4bc-3c20bcd911ef-metrics-certs") pod "network-metrics-daemon-tnb7c" (UID: "c861f27f-910e-48aa-b4bc-3c20bcd911ef") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:13.713641 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:13.713550    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97bef555-d008-4f5b-a974-91fbafd1eeb1-original-pull-secret podName:97bef555-d008-4f5b-a974-91fbafd1eeb1 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:29.713541087 +0000 UTC m=+35.801856709 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97bef555-d008-4f5b-a974-91fbafd1eeb1-original-pull-secret") pod "global-pull-secret-syncer-cmscw" (UID: "97bef555-d008-4f5b-a974-91fbafd1eeb1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:13.814163 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:13.814132    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzt9g\" (UniqueName: \"kubernetes.io/projected/8b313705-71ee-4c64-a650-b308f048dc2a-kube-api-access-kzt9g\") pod \"network-check-target-fs4cj\" (UID: \"8b313705-71ee-4c64-a650-b308f048dc2a\") " pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:13.814341 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:13.814309    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:13.814341 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:13.814329    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:13.814341 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:13.814339    2580 projected.go:196] Error preparing data for projected volume kube-api-access-kzt9g for pod openshift-network-diagnostics/network-check-target-fs4cj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:13.814509 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:13.814385    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b313705-71ee-4c64-a650-b308f048dc2a-kube-api-access-kzt9g podName:8b313705-71ee-4c64-a650-b308f048dc2a nodeName:}" failed. No retries permitted until 2026-09-18 16:16:29.81437169 +0000 UTC m=+35.902687321 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kzt9g" (UniqueName: "kubernetes.io/projected/8b313705-71ee-4c64-a650-b308f048dc2a-kube-api-access-kzt9g") pod "network-check-target-fs4cj" (UID: "8b313705-71ee-4c64-a650-b308f048dc2a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:14.117041 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:14.116970    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:14.117188 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:14.117095    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmscw" podUID="97bef555-d008-4f5b-a974-91fbafd1eeb1"
Sep 18 16:16:15.116895 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:15.116858    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:15.117334 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:15.116858    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:15.117334 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:15.116994    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tnb7c" podUID="c861f27f-910e-48aa-b4bc-3c20bcd911ef"
Sep 18 16:16:15.117334 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:15.117133    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs4cj" podUID="8b313705-71ee-4c64-a650-b308f048dc2a"
Sep 18 16:16:16.116983 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:16.116953    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:16.117346 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:16.117035    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmscw" podUID="97bef555-d008-4f5b-a974-91fbafd1eeb1"
Sep 18 16:16:17.116406 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:17.116213    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:17.116534 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:17.116271    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:17.116534 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:17.116474    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs4cj" podUID="8b313705-71ee-4c64-a650-b308f048dc2a"
Sep 18 16:16:17.116673 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:17.116544    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tnb7c" podUID="c861f27f-910e-48aa-b4bc-3c20bcd911ef"
Sep 18 16:16:17.312220 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:17.312034    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf" event={"ID":"022894e1-03ce-4883-91de-722dc423fc87","Type":"ContainerStarted","Data":"4cb9955442f42a21434057b7facc35c617087d0389b53b5b408ef4141af32de1"}
Sep 18 16:16:17.315659 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:17.315627    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg" event={"ID":"96f857f8-22b6-4d0b-b6fd-826b9b507ef8","Type":"ContainerStarted","Data":"2a35cf5755d4ff165d032eda27f90b9cf4093bc45c3beaae5c39d4b10cf27ea5"}
Sep 18 16:16:17.315777 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:17.315708    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg" event={"ID":"96f857f8-22b6-4d0b-b6fd-826b9b507ef8","Type":"ContainerStarted","Data":"da660a9b3bc2cb19fa09ec198557905812690e809757b0bb8cff81831326dc00"}
Sep 18 16:16:17.315777 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:17.315754    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg" event={"ID":"96f857f8-22b6-4d0b-b6fd-826b9b507ef8","Type":"ContainerStarted","Data":"3b19c6ce715bd6bc5790d3b0614170a1baa409a8d94547e73215f29c43fa8c7b"}
Sep 18 16:16:17.315908 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:17.315798    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg" event={"ID":"96f857f8-22b6-4d0b-b6fd-826b9b507ef8","Type":"ContainerStarted","Data":"0e567b50dff8fbb0ebaff7f66f81fbf27ef4dee1a712310bf69073c9187149cf"}
Sep 18 16:16:17.315908 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:17.315846    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg" event={"ID":"96f857f8-22b6-4d0b-b6fd-826b9b507ef8","Type":"ContainerStarted","Data":"d67d7cf7a14b08120c3b73835c849d950fcf9e660b25e6c407e172f8d1b2f6d9"}
Sep 18 16:16:17.315908 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:17.315893    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg" event={"ID":"96f857f8-22b6-4d0b-b6fd-826b9b507ef8","Type":"ContainerStarted","Data":"a0977bb188760f01d1d0ac54964613ff864cba869886581dd67eec94925f56ad"}
Sep 18 16:16:17.317278 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:17.317250    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k6gdw" event={"ID":"8c726ad7-c84f-4629-9797-52cfa3327996","Type":"ContainerStarted","Data":"8de442d86734a0b30d6151c70047cdcca06408ea77bb2c5351b946b392ab1fcd"}
Sep 18 16:16:17.319154 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:17.319129    2580 generic.go:356] "Generic (PLEG): container finished" podID="68f204b6-4215-48a9-973e-d1e84ae4fed5" containerID="e50ca62609ee2cbdc6fa1b4ecf5671df80dbb154851c1679bd24489b6484c229" exitCode=0
Sep 18 16:16:17.319270 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:17.319212    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ls6gx" event={"ID":"68f204b6-4215-48a9-973e-d1e84ae4fed5","Type":"ContainerDied","Data":"e50ca62609ee2cbdc6fa1b4ecf5671df80dbb154851c1679bd24489b6484c229"}
Sep 18 16:16:17.320972 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:17.320947    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8tkmt" event={"ID":"e80a48a0-bdd3-4128-999a-cc2574ced499","Type":"ContainerStarted","Data":"bee0c04830caa82a35c2af95cbfa9f09c03650943d8afb412a1298c78a3a4dbd"}
Sep 18 16:16:17.322751 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:17.322729    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-96w8b" event={"ID":"752ec166-f680-4caf-b569-5ca2d59307dc","Type":"ContainerStarted","Data":"4a3405a2ee4a90a52025562d11c3e23790c13fa05ed3499ba5e5766bb6097916"}
Sep 18 16:16:17.324337 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:17.324282    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4vmq6" event={"ID":"cfa7efc4-7409-41f5-b891-7ce9e5b6982f","Type":"ContainerStarted","Data":"1b416b3cea88cbceeba4494b682521ef199818932cf8bff5a1947c55a7a04c9a"}
Sep 18 16:16:17.325859 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:17.325837    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-krchj" event={"ID":"e945222a-603b-4489-a528-2f607d157b14","Type":"ContainerStarted","Data":"a5b7e9b40a9cb8578b8441629e0f4c28ac5c83cbbe07ad7997868be7d40cd5b8"}
Sep 18 16:16:17.337001 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:17.336967    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-k6gdw" podStartSLOduration=11.376848904 podStartE2EDuration="21.336956716s" totalImagesPullingTime="9.960107812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:17.335742573 +0000 UTC m=+23.424058212" watchObservedRunningTime="2026-09-18 16:16:17.336956716 +0000 UTC m=+23.425272356"
Sep 18 16:16:17.337520 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:17.337493    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-110.ec2.internal" podStartSLOduration=20.337483544 podStartE2EDuration="20.337483544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:01.298079547 +0000 UTC m=+7.386395188" watchObservedRunningTime="2026-09-18 16:16:17.337483544 +0000 UTC m=+23.425799184"
Sep 18 16:16:17.367067 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:17.367025    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-96w8b" podStartSLOduration=4.141211618 podStartE2EDuration="21.367011959s" totalImagesPullingTime="17.225800341s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:17.36656488 +0000 UTC m=+23.454880520" watchObservedRunningTime="2026-09-18 16:16:17.367011959 +0000 UTC m=+23.455327598"
Sep 18 16:16:17.388659 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:17.388622    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8tkmt" podStartSLOduration=4.10434117 podStartE2EDuration="21.388608803s" totalImagesPullingTime="17.284267633s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:17.388327349 +0000 UTC m=+23.476642983" watchObservedRunningTime="2026-09-18 16:16:17.388608803 +0000 UTC m=+23.476924440"
Sep 18 16:16:17.405034 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:17.404995    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4vmq6" podStartSLOduration=4.179066615 podStartE2EDuration="21.40498317s" totalImagesPullingTime="17.225916555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:17.404792075 +0000 UTC m=+23.493107714" watchObservedRunningTime="2026-09-18 16:16:17.40498317 +0000 UTC m=+23.493298809"
Sep 18 16:16:17.423968 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:17.423925    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-krchj" podStartSLOduration=4.206557075 podStartE2EDuration="21.423909366s" totalImagesPullingTime="17.217352291s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:17.423691615 +0000 UTC m=+23.512007261" watchObservedRunningTime="2026-09-18 16:16:17.423909366 +0000 UTC m=+23.512225006"
Sep 18 16:16:17.495184 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:17.495148    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:17.578835 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:17.578811    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:17.682706 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:17.682679    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:17.820120 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:17.820049    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 16:16:17.995651 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:17.995607    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:18.040915 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:18.040817    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T16:16:17.820079141Z","UUID":"0f83f9ff-9a4c-4374-9b75-9fad65de4fcd","Handler":null,"Name":"","Endpoint":""}
Sep 18 16:16:18.042340 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:18.042314    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 16:16:18.042479 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:18.042348    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 16:16:18.096834 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:18.096755    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:18.116551 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:18.116527    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:18.116710 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:18.116684    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmscw" podUID="97bef555-d008-4f5b-a974-91fbafd1eeb1"
Sep 18 16:16:18.194869 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:18.194846    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-k6gdw"
Sep 18 16:16:18.195591 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:18.195519    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-k6gdw"
Sep 18 16:16:18.331468 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:18.331429    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf" event={"ID":"022894e1-03ce-4883-91de-722dc423fc87","Type":"ContainerStarted","Data":"bf62e8d036ad3b6fc2fc27cf1b77248813c9a3ed7081add351071ebc5e291dd4"}
Sep 18 16:16:18.333450 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:18.333261    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h6tt8" event={"ID":"11366fd3-c5b8-4bf4-8ba6-217fd3a1b658","Type":"ContainerStarted","Data":"8dbdb13e277c67653ba2fa175dc39e27f03a2f9f973a93c2093535239e7eed91"}
Sep 18 16:16:18.350289 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:18.350054    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-h6tt8" podStartSLOduration=5.127433359 podStartE2EDuration="22.350038402s" totalImagesPullingTime="17.222605043s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:18.349418301 +0000 UTC m=+24.437733941" watchObservedRunningTime="2026-09-18 16:16:18.350038402 +0000 UTC m=+24.438354044"
Sep 18 16:16:18.478350 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:18.478315    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:18.559058 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:18.559030    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:18.785275 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:18.785220    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:18.856780 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:18.856746    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:19.040837 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:19.040799    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:19.116232 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:19.116184    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:19.116422 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:19.116294    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs4cj" podUID="8b313705-71ee-4c64-a650-b308f048dc2a"
Sep 18 16:16:19.116422 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:19.116354    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:19.116539 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:19.116479    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tnb7c" podUID="c861f27f-910e-48aa-b4bc-3c20bcd911ef"
Sep 18 16:16:19.121384 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:19.121287    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:19.236733 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:19.236659    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:19.337929 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:19.337893    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf" event={"ID":"022894e1-03ce-4883-91de-722dc423fc87","Type":"ContainerStarted","Data":"dae11b0dfc2355dc9679f420594da81d41d9ae17fbcc43f6285967e06c000d68"}
Sep 18 16:16:19.339663 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:19.339634    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:19.341495 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:19.341471    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 16:16:19.341593 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:19.341496    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg" event={"ID":"96f857f8-22b6-4d0b-b6fd-826b9b507ef8","Type":"ContainerStarted","Data":"89326b5562789cb007b3848e316b010279142762a01ee246d5f5f213efb1bdba"}
Sep 18 16:16:19.366198 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:19.366151    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv5cf" podStartSLOduration=4.369910068 podStartE2EDuration="23.366136412s" totalImagesPullingTime="18.996226344s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:19.365103976 +0000 UTC m=+25.453419613" watchObservedRunningTime="2026-09-18 16:16:19.366136412 +0000 UTC m=+25.454452050"
Sep 18 16:16:19.429477 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:19.429442    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:19.529180 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:19.529093    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:19.629601 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:19.629551    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:20.116634 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:20.116585    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:20.116794 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:20.116699    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmscw" podUID="97bef555-d008-4f5b-a974-91fbafd1eeb1"
Sep 18 16:16:20.344296 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:20.344259    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:20.403640 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:20.403564    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:20.553224 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:20.553195    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-krchj_e945222a-603b-4489-a528-2f607d157b14/dns-node-resolver/0.log"
Sep 18 16:16:20.575521 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:20.575500    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-k6gdw"
Sep 18 16:16:20.575640 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:20.575627    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 16:16:20.575974 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:20.575957    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k6gdw"
Sep 18 16:16:20.655084 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:20.655000    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:21.117176 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:21.116914    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:21.117403 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:21.116981    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:21.117403 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:21.117226    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs4cj" podUID="8b313705-71ee-4c64-a650-b308f048dc2a"
Sep 18 16:16:21.117403 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:21.117323    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tnb7c" podUID="c861f27f-910e-48aa-b4bc-3c20bcd911ef"
Sep 18 16:16:21.140917 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:21.140778    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:21.228373 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:21.228336    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:21.350860 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:21.350735    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg" event={"ID":"96f857f8-22b6-4d0b-b6fd-826b9b507ef8","Type":"ContainerStarted","Data":"409adc5cbfbeee5b263830a1948df903a2bf531aadbbb2a2ebddf575b5790313"}
Sep 18 16:16:21.353105 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:21.353083    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4vmq6_cfa7efc4-7409-41f5-b891-7ce9e5b6982f/node-ca/0.log"
Sep 18 16:16:21.380349 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:21.380280    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg" podStartSLOduration=8.116454127 podStartE2EDuration="25.380259361s" totalImagesPullingTime="17.263805234s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:21.379125071 +0000 UTC m=+27.467440736" watchObservedRunningTime="2026-09-18 16:16:21.380259361 +0000 UTC m=+27.468575003"
Sep 18 16:16:22.116689 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:22.116653    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:22.116851 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:22.116765    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmscw" podUID="97bef555-d008-4f5b-a974-91fbafd1eeb1"
Sep 18 16:16:22.303617 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:22.303545    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:22.355827 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:22.354223    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 16:16:22.355827 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:22.355534    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:16:22.355827 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:22.355707    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:16:22.380927 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:22.380576    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:16:22.385898 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:22.385860    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:16:22.400586 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:22.400556    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:23.014747 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:23.014714    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:23.115483 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:23.115445    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:23.116248 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:23.116224    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:23.116408 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:23.116224    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:23.116408 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:23.116378    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tnb7c" podUID="c861f27f-910e-48aa-b4bc-3c20bcd911ef"
Sep 18 16:16:23.116508 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:23.116420    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs4cj" podUID="8b313705-71ee-4c64-a650-b308f048dc2a"
Sep 18 16:16:23.352286 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:23.352054    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cmscw"]
Sep 18 16:16:23.352437 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:23.352338    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:23.352485 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:23.352449    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmscw" podUID="97bef555-d008-4f5b-a974-91fbafd1eeb1"
Sep 18 16:16:23.356219 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:23.356136    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 16:16:23.357247 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:23.357226    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fs4cj"]
Sep 18 16:16:23.357367 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:23.357316    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:23.357425 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:23.357404    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs4cj" podUID="8b313705-71ee-4c64-a650-b308f048dc2a"
Sep 18 16:16:23.357634 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:23.357614    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tnb7c"]
Sep 18 16:16:23.357726 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:23.357698    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:23.357823 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:23.357801    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tnb7c" podUID="c861f27f-910e-48aa-b4bc-3c20bcd911ef"
Sep 18 16:16:23.489288 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:23.489264    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:24.360384 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:24.360347    2580 generic.go:356] "Generic (PLEG): container finished" podID="68f204b6-4215-48a9-973e-d1e84ae4fed5" containerID="55c5af8104e704f7406926d55c429c3ca86ab2d89ad260b6090109e5660888af" exitCode=0
Sep 18 16:16:24.360750 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:24.360434    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ls6gx" event={"ID":"68f204b6-4215-48a9-973e-d1e84ae4fed5","Type":"ContainerDied","Data":"55c5af8104e704f7406926d55c429c3ca86ab2d89ad260b6090109e5660888af"}
Sep 18 16:16:24.360750 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:24.360651    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 16:16:24.394158 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:24.394130    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:25.116733 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:25.116702    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:25.116733 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:25.116722    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:25.116902 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:25.116722    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:25.116902 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:25.116840    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmscw" podUID="97bef555-d008-4f5b-a974-91fbafd1eeb1"
Sep 18 16:16:25.116998 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:25.116956    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tnb7c" podUID="c861f27f-910e-48aa-b4bc-3c20bcd911ef"
Sep 18 16:16:25.117052 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:25.117029    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs4cj" podUID="8b313705-71ee-4c64-a650-b308f048dc2a"
Sep 18 16:16:26.065507 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:26.065467    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:26.173098 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:26.173073    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:26.240186 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:26.240121    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:26.333150 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:26.333125    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:26.366628 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:26.366595    2580 generic.go:356] "Generic (PLEG): container finished" podID="68f204b6-4215-48a9-973e-d1e84ae4fed5" containerID="e73c8612f4674c8c0cdc2f5062e38dad3f03a80d2b9aa71466b10f1999ed5e0e" exitCode=0
Sep 18 16:16:26.366746 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:26.366660    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ls6gx" event={"ID":"68f204b6-4215-48a9-973e-d1e84ae4fed5","Type":"ContainerDied","Data":"e73c8612f4674c8c0cdc2f5062e38dad3f03a80d2b9aa71466b10f1999ed5e0e"}
Sep 18 16:16:26.423252 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:26.423203    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:26.568330 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:26.568236    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:26.668078 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:26.668050    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:26.844359 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:26.844282    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:27.116850 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:27.116823    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:27.117188 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:27.116854    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:27.117188 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:27.116859    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:27.117188 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:27.116948    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmscw" podUID="97bef555-d008-4f5b-a974-91fbafd1eeb1"
Sep 18 16:16:27.117188 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:27.117005    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tnb7c" podUID="c861f27f-910e-48aa-b4bc-3c20bcd911ef"
Sep 18 16:16:27.117188 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:27.117105    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs4cj" podUID="8b313705-71ee-4c64-a650-b308f048dc2a"
Sep 18 16:16:27.569156 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:27.569126    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:27.731758 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:27.731700    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:27.806033 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:27.806005    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:28.059994 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:28.059788    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:16:28.060128 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:28.060115    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 16:16:28.078754 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:28.078726    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gw4fg"
Sep 18 16:16:28.165284 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:28.165255    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:28.372379 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:28.372320    2580 generic.go:356] "Generic (PLEG): container finished" podID="68f204b6-4215-48a9-973e-d1e84ae4fed5" containerID="11276add50955250d746fa9fe6f6dc8eb84cfcab79d0d2274b8c72713a355332" exitCode=0
Sep 18 16:16:28.372379 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:28.372367    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ls6gx" event={"ID":"68f204b6-4215-48a9-973e-d1e84ae4fed5","Type":"ContainerDied","Data":"11276add50955250d746fa9fe6f6dc8eb84cfcab79d0d2274b8c72713a355332"}
Sep 18 16:16:28.373805 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:28.373753    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:28.524916 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:28.524872    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:28.625208 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:28.625143    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:28.761125 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:28.761100    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:28.968333 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:28.968292    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:29.116830 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:29.116807    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:29.116830 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:29.116822    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:29.116998 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:29.116825    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:29.116998 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:29.116925    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs4cj" podUID="8b313705-71ee-4c64-a650-b308f048dc2a"
Sep 18 16:16:29.117089 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:29.117003    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tnb7c" podUID="c861f27f-910e-48aa-b4bc-3c20bcd911ef"
Sep 18 16:16:29.117129 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:29.117088    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmscw" podUID="97bef555-d008-4f5b-a974-91fbafd1eeb1"
Sep 18 16:16:29.162169 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:29.162141    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:29.358373 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:29.358241    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:29.731251 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:29.731217    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c861f27f-910e-48aa-b4bc-3c20bcd911ef-metrics-certs\") pod \"network-metrics-daemon-tnb7c\" (UID: \"c861f27f-910e-48aa-b4bc-3c20bcd911ef\") " pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:29.731437 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:29.731261    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97bef555-d008-4f5b-a974-91fbafd1eeb1-original-pull-secret\") pod \"global-pull-secret-syncer-cmscw\" (UID: \"97bef555-d008-4f5b-a974-91fbafd1eeb1\") " pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:29.731437 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:29.731404    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:29.731557 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:29.731461    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97bef555-d008-4f5b-a974-91fbafd1eeb1-original-pull-secret podName:97bef555-d008-4f5b-a974-91fbafd1eeb1 nodeName:}" failed. No retries permitted until 2026-09-18 16:17:01.73144201 +0000 UTC m=+67.819757632 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97bef555-d008-4f5b-a974-91fbafd1eeb1-original-pull-secret") pod "global-pull-secret-syncer-cmscw" (UID: "97bef555-d008-4f5b-a974-91fbafd1eeb1") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:29.731557 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:29.731402    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:29.731557 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:29.731538    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c861f27f-910e-48aa-b4bc-3c20bcd911ef-metrics-certs podName:c861f27f-910e-48aa-b4bc-3c20bcd911ef nodeName:}" failed. No retries permitted until 2026-09-18 16:17:01.731524008 +0000 UTC m=+67.819839655 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c861f27f-910e-48aa-b4bc-3c20bcd911ef-metrics-certs") pod "network-metrics-daemon-tnb7c" (UID: "c861f27f-910e-48aa-b4bc-3c20bcd911ef") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:29.832479 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:29.832448    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzt9g\" (UniqueName: \"kubernetes.io/projected/8b313705-71ee-4c64-a650-b308f048dc2a-kube-api-access-kzt9g\") pod \"network-check-target-fs4cj\" (UID: \"8b313705-71ee-4c64-a650-b308f048dc2a\") " pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:29.832652 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:29.832622    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:29.832652 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:29.832643    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:29.832785 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:29.832656    2580 projected.go:196] Error preparing data for projected volume kube-api-access-kzt9g for pod openshift-network-diagnostics/network-check-target-fs4cj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:29.832785 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:29.832716    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b313705-71ee-4c64-a650-b308f048dc2a-kube-api-access-kzt9g podName:8b313705-71ee-4c64-a650-b308f048dc2a nodeName:}" failed. No retries permitted until 2026-09-18 16:17:01.832698926 +0000 UTC m=+67.921014560 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kzt9g" (UniqueName: "kubernetes.io/projected/8b313705-71ee-4c64-a650-b308f048dc2a-kube-api-access-kzt9g") pod "network-check-target-fs4cj" (UID: "8b313705-71ee-4c64-a650-b308f048dc2a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:30.259787 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:30.259756    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:30.378429 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:30.378398    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:30.578441 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:30.578368    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:30.773589 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:30.773553    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:30.970020 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:30.969952    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:31.117180 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:31.117154    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:31.117346 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:31.117154    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:31.117346 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:31.117247    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs4cj" podUID="8b313705-71ee-4c64-a650-b308f048dc2a"
Sep 18 16:16:31.117346 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:31.117260    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:31.117346 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:31.117330    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmscw" podUID="97bef555-d008-4f5b-a974-91fbafd1eeb1"
Sep 18 16:16:31.117476 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:31.117390    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tnb7c" podUID="c861f27f-910e-48aa-b4bc-3c20bcd911ef"
Sep 18 16:16:31.168016 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:31.167911    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:31.267941 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:31.267857    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:32.167978 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:32.167950    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:32.564179 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:32.564149    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:32.665152 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:32.665118    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:32.765822 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:32.765786    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:32.966807 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:32.966772    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:33.116993 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:33.116963    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:33.116993 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:33.116992    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:33.117152 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:33.117078    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs4cj" podUID="8b313705-71ee-4c64-a650-b308f048dc2a"
Sep 18 16:16:33.117152 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:33.117128    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:33.117232 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:33.117201    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmscw" podUID="97bef555-d008-4f5b-a974-91fbafd1eeb1"
Sep 18 16:16:33.117326 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:33.117291    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tnb7c" podUID="c861f27f-910e-48aa-b4bc-3c20bcd911ef"
Sep 18 16:16:33.170489 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:33.170463    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:33.385743 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:33.385691    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ls6gx" event={"ID":"68f204b6-4215-48a9-973e-d1e84ae4fed5","Type":"ContainerStarted","Data":"65a2da67083763b94ac9bfd340761b03b31c022dc6328661ebacbea232bd0549"}
Sep 18 16:16:33.690472 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:33.690447    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:34.160610 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:34.160533    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:34.390577 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:34.390538    2580 generic.go:356] "Generic (PLEG): container finished" podID="68f204b6-4215-48a9-973e-d1e84ae4fed5" containerID="65a2da67083763b94ac9bfd340761b03b31c022dc6328661ebacbea232bd0549" exitCode=0
Sep 18 16:16:34.391112 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:34.390586    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ls6gx" event={"ID":"68f204b6-4215-48a9-973e-d1e84ae4fed5","Type":"ContainerDied","Data":"65a2da67083763b94ac9bfd340761b03b31c022dc6328661ebacbea232bd0549"}
Sep 18 16:16:34.570661 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:34.570626    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:34.671079 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:34.671012    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:34.769723 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:34.769683    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:34.958369 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:34.958339    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:35.117129 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:35.117105    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:35.117269 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:35.117106    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:35.117269 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:35.117208    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmscw" podUID="97bef555-d008-4f5b-a974-91fbafd1eeb1"
Sep 18 16:16:35.117269 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:35.117107    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:35.117269 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:35.117261    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs4cj" podUID="8b313705-71ee-4c64-a650-b308f048dc2a"
Sep 18 16:16:35.117516 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:35.117355    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tnb7c" podUID="c861f27f-910e-48aa-b4bc-3c20bcd911ef"
Sep 18 16:16:35.364722 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:35.364645    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:35.395037 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:35.395005    2580 generic.go:356] "Generic (PLEG): container finished" podID="68f204b6-4215-48a9-973e-d1e84ae4fed5" containerID="3ec079a50ead9d0ff4a1687e8bb50be45c41aad55eb3560d793af52263a2e259" exitCode=0
Sep 18 16:16:35.395571 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:35.395035    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ls6gx" event={"ID":"68f204b6-4215-48a9-973e-d1e84ae4fed5","Type":"ContainerDied","Data":"3ec079a50ead9d0ff4a1687e8bb50be45c41aad55eb3560d793af52263a2e259"}
Sep 18 16:16:35.962511 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:35.962461    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:36.360363 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:36.360253    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:36.400496 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:36.400474    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ls6gx" event={"ID":"68f204b6-4215-48a9-973e-d1e84ae4fed5","Type":"ContainerStarted","Data":"42ffd27e7827b1ec2625a0e5525456bfe2ef1f16d9be3ae8082a0166c2ea52e8"}
Sep 18 16:16:36.421073 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:36.421035    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ls6gx" podStartSLOduration=9.953668313 podStartE2EDuration="40.421023481s" totalImagesPullingTime="30.467355168s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:36.419743094 +0000 UTC m=+42.508058732" watchObservedRunningTime="2026-09-18 16:16:36.421023481 +0000 UTC m=+42.509339120"
Sep 18 16:16:36.562377 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:36.562354    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:36.808106 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:36.808073    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:37.117224 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:37.117155    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:37.117372 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:37.117249    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:37.117372 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:37.117260    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:37.117372 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:37.117261    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmscw" podUID="97bef555-d008-4f5b-a974-91fbafd1eeb1"
Sep 18 16:16:37.117372 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:37.117352    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tnb7c" podUID="c861f27f-910e-48aa-b4bc-3c20bcd911ef"
Sep 18 16:16:37.117503 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:37.117415    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs4cj" podUID="8b313705-71ee-4c64-a650-b308f048dc2a"
Sep 18 16:16:39.116353 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:39.116319    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:39.116353 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:39.116344    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:39.116862 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:39.116319    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:39.116862 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:39.116406    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmscw" podUID="97bef555-d008-4f5b-a974-91fbafd1eeb1"
Sep 18 16:16:39.116862 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:39.116498    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs4cj" podUID="8b313705-71ee-4c64-a650-b308f048dc2a"
Sep 18 16:16:39.116862 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:39.116564    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tnb7c" podUID="c861f27f-910e-48aa-b4bc-3c20bcd911ef"
Sep 18 16:16:41.116826 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:41.116800    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:41.116826 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:41.116822    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:41.117137 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:41.116800    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:41.117137 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:41.116898    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tnb7c" podUID="c861f27f-910e-48aa-b4bc-3c20bcd911ef"
Sep 18 16:16:41.117137 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:41.116961    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmscw" podUID="97bef555-d008-4f5b-a974-91fbafd1eeb1"
Sep 18 16:16:41.117137 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:41.117016    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs4cj" podUID="8b313705-71ee-4c64-a650-b308f048dc2a"
Sep 18 16:16:42.716596 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:42.716571    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:42.778740 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:42.778698    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:43.117169 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:43.117100    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:43.117289 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:43.117100    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:43.117289 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:43.117197    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tnb7c" podUID="c861f27f-910e-48aa-b4bc-3c20bcd911ef"
Sep 18 16:16:43.117289 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:43.117100    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:43.117289 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:43.117263    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmscw" podUID="97bef555-d008-4f5b-a974-91fbafd1eeb1"
Sep 18 16:16:43.117468 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:43.117335    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs4cj" podUID="8b313705-71ee-4c64-a650-b308f048dc2a"
Sep 18 16:16:45.117123 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:45.117084    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:45.117530 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:45.117085    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:45.117530 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:45.117191    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tnb7c" podUID="c861f27f-910e-48aa-b4bc-3c20bcd911ef"
Sep 18 16:16:45.117530 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:45.117085    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:45.117530 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:45.117293    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmscw" podUID="97bef555-d008-4f5b-a974-91fbafd1eeb1"
Sep 18 16:16:45.117530 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:45.117358    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs4cj" podUID="8b313705-71ee-4c64-a650-b308f048dc2a"
Sep 18 16:16:46.076995 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:46.076957    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:46.172152 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:46.172118    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:46.273965 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:46.273939    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:46.953999 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:46.953973    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:47.039746 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.039696    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:47.117150 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.117119    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:47.117287 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.117156    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:47.117287 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.117253    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:47.117287 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:47.117259    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmscw" podUID="97bef555-d008-4f5b-a974-91fbafd1eeb1"
Sep 18 16:16:47.117415 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:47.117357    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tnb7c" podUID="c861f27f-910e-48aa-b4bc-3c20bcd911ef"
Sep 18 16:16:47.117450 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:47.117429    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs4cj" podUID="8b313705-71ee-4c64-a650-b308f048dc2a"
Sep 18 16:16:47.203118 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.203021    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:47.292225 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.292190    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:47.359063 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.359034    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:47.363925 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.363907    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-110.ec2.internal" event="NodeReady"
Sep 18 16:16:47.364021 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.364015    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 16:16:47.415421 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.415389    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-86jqq"]
Sep 18 16:16:47.417146 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.417131    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-thwr5"]
Sep 18 16:16:47.417261 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.417245    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-86jqq"
Sep 18 16:16:47.418841 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.418823    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-thwr5"
Sep 18 16:16:47.420331 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.420312    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 16:16:47.420429 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.420411    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 16:16:47.420490 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.420466    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-44fnz\""
Sep 18 16:16:47.421487 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.421463    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 16:16:47.421579 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.421485    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 16:16:47.421579 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.421542    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 16:16:47.421748 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.421608    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nst46\""
Sep 18 16:16:47.429741 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.429668    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-thwr5"]
Sep 18 16:16:47.432604 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.432586    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-86jqq"]
Sep 18 16:16:47.476888 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.476810    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:47.515678 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.515653    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f5szk"]
Sep 18 16:16:47.517746 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.517730    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f5szk"
Sep 18 16:16:47.520577 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.520560    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 16:16:47.520783 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.520770    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 16:16:47.520858 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.520769    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 16:16:47.520858 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.520802    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 16:16:47.520858 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.520769    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pj9mb\""
Sep 18 16:16:47.529157 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.529139    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f5szk"]
Sep 18 16:16:47.559799 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.559776    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:47.564723 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.564706    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ee6c64-ba20-49bc-9d59-c86ce91a51bf-config-volume\") pod \"dns-default-86jqq\" (UID: \"f4ee6c64-ba20-49bc-9d59-c86ce91a51bf\") " pod="openshift-dns/dns-default-86jqq"
Sep 18 16:16:47.564792 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.564737    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4ee6c64-ba20-49bc-9d59-c86ce91a51bf-tmp-dir\") pod \"dns-default-86jqq\" (UID: \"f4ee6c64-ba20-49bc-9d59-c86ce91a51bf\") " pod="openshift-dns/dns-default-86jqq"
Sep 18 16:16:47.564792 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.564757    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1cea6a79-7d5a-4093-a1b8-0d07198052e3-cert\") pod \"ingress-canary-thwr5\" (UID: \"1cea6a79-7d5a-4093-a1b8-0d07198052e3\") " pod="openshift-ingress-canary/ingress-canary-thwr5"
Sep 18 16:16:47.564792 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.564781    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kstds\" (UniqueName: \"kubernetes.io/projected/1cea6a79-7d5a-4093-a1b8-0d07198052e3-kube-api-access-kstds\") pod \"ingress-canary-thwr5\" (UID: \"1cea6a79-7d5a-4093-a1b8-0d07198052e3\") " pod="openshift-ingress-canary/ingress-canary-thwr5"
Sep 18 16:16:47.564890 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.564798    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhnfm\" (UniqueName: \"kubernetes.io/projected/f4ee6c64-ba20-49bc-9d59-c86ce91a51bf-kube-api-access-bhnfm\") pod \"dns-default-86jqq\" (UID: \"f4ee6c64-ba20-49bc-9d59-c86ce91a51bf\") " pod="openshift-dns/dns-default-86jqq"
Sep 18 16:16:47.564926 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.564895    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4ee6c64-ba20-49bc-9d59-c86ce91a51bf-metrics-tls\") pod \"dns-default-86jqq\" (UID: \"f4ee6c64-ba20-49bc-9d59-c86ce91a51bf\") " pod="openshift-dns/dns-default-86jqq"
Sep 18 16:16:47.664588 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.664561    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:47.665425 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.665407    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4ee6c64-ba20-49bc-9d59-c86ce91a51bf-tmp-dir\") pod \"dns-default-86jqq\" (UID: \"f4ee6c64-ba20-49bc-9d59-c86ce91a51bf\") " pod="openshift-dns/dns-default-86jqq"
Sep 18 16:16:47.665511 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.665439    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1cea6a79-7d5a-4093-a1b8-0d07198052e3-cert\") pod \"ingress-canary-thwr5\" (UID: \"1cea6a79-7d5a-4093-a1b8-0d07198052e3\") " pod="openshift-ingress-canary/ingress-canary-thwr5"
Sep 18 16:16:47.665511 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.665470    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be090468-5a51-41dc-97fe-8a8569135cef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f5szk\" (UID: \"be090468-5a51-41dc-97fe-8a8569135cef\") " pod="openshift-insights/insights-runtime-extractor-f5szk"
Sep 18 16:16:47.665511 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.665504    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kstds\" (UniqueName: \"kubernetes.io/projected/1cea6a79-7d5a-4093-a1b8-0d07198052e3-kube-api-access-kstds\") pod \"ingress-canary-thwr5\" (UID: \"1cea6a79-7d5a-4093-a1b8-0d07198052e3\") " pod="openshift-ingress-canary/ingress-canary-thwr5"
Sep 18 16:16:47.665663 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.665531    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhnfm\" (UniqueName: \"kubernetes.io/projected/f4ee6c64-ba20-49bc-9d59-c86ce91a51bf-kube-api-access-bhnfm\") pod \"dns-default-86jqq\" (UID: \"f4ee6c64-ba20-49bc-9d59-c86ce91a51bf\") " pod="openshift-dns/dns-default-86jqq"
Sep 18 16:16:47.665663 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.665650    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/be090468-5a51-41dc-97fe-8a8569135cef-data-volume\") pod \"insights-runtime-extractor-f5szk\" (UID: \"be090468-5a51-41dc-97fe-8a8569135cef\") " pod="openshift-insights/insights-runtime-extractor-f5szk"
Sep 18 16:16:47.665819 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.665679    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8l8r\" (UniqueName: \"kubernetes.io/projected/be090468-5a51-41dc-97fe-8a8569135cef-kube-api-access-l8l8r\") pod \"insights-runtime-extractor-f5szk\" (UID: \"be090468-5a51-41dc-97fe-8a8569135cef\") " pod="openshift-insights/insights-runtime-extractor-f5szk"
Sep 18 16:16:47.665819 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.665703    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be090468-5a51-41dc-97fe-8a8569135cef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5szk\" (UID: \"be090468-5a51-41dc-97fe-8a8569135cef\") " pod="openshift-insights/insights-runtime-extractor-f5szk"
Sep 18 16:16:47.665819 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.665746    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/be090468-5a51-41dc-97fe-8a8569135cef-crio-socket\") pod \"insights-runtime-extractor-f5szk\" (UID: \"be090468-5a51-41dc-97fe-8a8569135cef\") " pod="openshift-insights/insights-runtime-extractor-f5szk"
Sep 18 16:16:47.665819 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.665762    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4ee6c64-ba20-49bc-9d59-c86ce91a51bf-tmp-dir\") pod \"dns-default-86jqq\" (UID: \"f4ee6c64-ba20-49bc-9d59-c86ce91a51bf\") " pod="openshift-dns/dns-default-86jqq"
Sep 18 16:16:47.665819 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.665792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4ee6c64-ba20-49bc-9d59-c86ce91a51bf-metrics-tls\") pod \"dns-default-86jqq\" (UID: \"f4ee6c64-ba20-49bc-9d59-c86ce91a51bf\") " pod="openshift-dns/dns-default-86jqq"
Sep 18 16:16:47.666031 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.665832    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ee6c64-ba20-49bc-9d59-c86ce91a51bf-config-volume\") pod \"dns-default-86jqq\" (UID: \"f4ee6c64-ba20-49bc-9d59-c86ce91a51bf\") " pod="openshift-dns/dns-default-86jqq"
Sep 18 16:16:47.666288 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.666263    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ee6c64-ba20-49bc-9d59-c86ce91a51bf-config-volume\") pod \"dns-default-86jqq\" (UID: \"f4ee6c64-ba20-49bc-9d59-c86ce91a51bf\") " pod="openshift-dns/dns-default-86jqq"
Sep 18 16:16:47.669144 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.669124    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4ee6c64-ba20-49bc-9d59-c86ce91a51bf-metrics-tls\") pod \"dns-default-86jqq\" (UID: \"f4ee6c64-ba20-49bc-9d59-c86ce91a51bf\") " pod="openshift-dns/dns-default-86jqq"
Sep 18 16:16:47.669252 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.669236    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1cea6a79-7d5a-4093-a1b8-0d07198052e3-cert\") pod \"ingress-canary-thwr5\" (UID: \"1cea6a79-7d5a-4093-a1b8-0d07198052e3\") " pod="openshift-ingress-canary/ingress-canary-thwr5"
Sep 18 16:16:47.673027 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.673004    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kstds\" (UniqueName: \"kubernetes.io/projected/1cea6a79-7d5a-4093-a1b8-0d07198052e3-kube-api-access-kstds\") pod \"ingress-canary-thwr5\" (UID: \"1cea6a79-7d5a-4093-a1b8-0d07198052e3\") " pod="openshift-ingress-canary/ingress-canary-thwr5"
Sep 18 16:16:47.673027 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.673018    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhnfm\" (UniqueName: \"kubernetes.io/projected/f4ee6c64-ba20-49bc-9d59-c86ce91a51bf-kube-api-access-bhnfm\") pod \"dns-default-86jqq\" (UID: \"f4ee6c64-ba20-49bc-9d59-c86ce91a51bf\") " pod="openshift-dns/dns-default-86jqq"
Sep 18 16:16:47.726854 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.726825    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-86jqq"
Sep 18 16:16:47.729806 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.729775    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:47.731684 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.731666    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-thwr5"
Sep 18 16:16:47.766872 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.766337    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be090468-5a51-41dc-97fe-8a8569135cef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f5szk\" (UID: \"be090468-5a51-41dc-97fe-8a8569135cef\") " pod="openshift-insights/insights-runtime-extractor-f5szk"
Sep 18 16:16:47.766872 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.766404    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/be090468-5a51-41dc-97fe-8a8569135cef-data-volume\") pod \"insights-runtime-extractor-f5szk\" (UID: \"be090468-5a51-41dc-97fe-8a8569135cef\") " pod="openshift-insights/insights-runtime-extractor-f5szk"
Sep 18 16:16:47.766872 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.766435    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8l8r\" (UniqueName: \"kubernetes.io/projected/be090468-5a51-41dc-97fe-8a8569135cef-kube-api-access-l8l8r\") pod \"insights-runtime-extractor-f5szk\" (UID: \"be090468-5a51-41dc-97fe-8a8569135cef\") " pod="openshift-insights/insights-runtime-extractor-f5szk"
Sep 18 16:16:47.766872 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.766477    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be090468-5a51-41dc-97fe-8a8569135cef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5szk\" (UID: \"be090468-5a51-41dc-97fe-8a8569135cef\") " pod="openshift-insights/insights-runtime-extractor-f5szk"
Sep 18 16:16:47.766872 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.766515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/be090468-5a51-41dc-97fe-8a8569135cef-crio-socket\") pod \"insights-runtime-extractor-f5szk\" (UID: \"be090468-5a51-41dc-97fe-8a8569135cef\") " pod="openshift-insights/insights-runtime-extractor-f5szk"
Sep 18 16:16:47.766872 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.766769    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/be090468-5a51-41dc-97fe-8a8569135cef-crio-socket\") pod \"insights-runtime-extractor-f5szk\" (UID: \"be090468-5a51-41dc-97fe-8a8569135cef\") " pod="openshift-insights/insights-runtime-extractor-f5szk"
Sep 18 16:16:47.767514 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.767491    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/be090468-5a51-41dc-97fe-8a8569135cef-data-volume\") pod \"insights-runtime-extractor-f5szk\" (UID: \"be090468-5a51-41dc-97fe-8a8569135cef\") " pod="openshift-insights/insights-runtime-extractor-f5szk"
Sep 18 16:16:47.767627 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.767604    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be090468-5a51-41dc-97fe-8a8569135cef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f5szk\" (UID: \"be090468-5a51-41dc-97fe-8a8569135cef\") " pod="openshift-insights/insights-runtime-extractor-f5szk"
Sep 18 16:16:47.769868 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.769843    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be090468-5a51-41dc-97fe-8a8569135cef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5szk\" (UID: \"be090468-5a51-41dc-97fe-8a8569135cef\") " pod="openshift-insights/insights-runtime-extractor-f5szk"
Sep 18 16:16:47.775295 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.775270    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8l8r\" (UniqueName: \"kubernetes.io/projected/be090468-5a51-41dc-97fe-8a8569135cef-kube-api-access-l8l8r\") pod \"insights-runtime-extractor-f5szk\" (UID: \"be090468-5a51-41dc-97fe-8a8569135cef\") " pod="openshift-insights/insights-runtime-extractor-f5szk"
Sep 18 16:16:47.825485 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.825446    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f5szk"
Sep 18 16:16:47.826646 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.826611    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:47.928887 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:47.928845    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:48.011476 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:48.011411    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-thwr5"]
Sep 18 16:16:48.014138 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:48.014115    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-86jqq"]
Sep 18 16:16:48.015321 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:16:48.015280    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cea6a79_7d5a_4093_a1b8_0d07198052e3.slice/crio-c0b1b315c5e3e65f6f9aca730d57fd3e1a04a7bdd12f262bd9af35a8c33c92c9 WatchSource:0}: Error finding container c0b1b315c5e3e65f6f9aca730d57fd3e1a04a7bdd12f262bd9af35a8c33c92c9: Status 404 returned error can't find the container with id c0b1b315c5e3e65f6f9aca730d57fd3e1a04a7bdd12f262bd9af35a8c33c92c9
Sep 18 16:16:48.016526 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:16:48.016499    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ee6c64_ba20_49bc_9d59_c86ce91a51bf.slice/crio-090b16f4dac210da75f2b479edfaf60dbe130638aeb8ddf4f31c8d1b1e7ca92e WatchSource:0}: Error finding container 090b16f4dac210da75f2b479edfaf60dbe130638aeb8ddf4f31c8d1b1e7ca92e: Status 404 returned error can't find the container with id 090b16f4dac210da75f2b479edfaf60dbe130638aeb8ddf4f31c8d1b1e7ca92e
Sep 18 16:16:48.022195 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:48.022176    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:48.082620 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:48.082593    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:48.091593 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:48.091569    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f5szk"]
Sep 18 16:16:48.094646 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:16:48.094624    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe090468_5a51_41dc_97fe_8a8569135cef.slice/crio-48b69f0f2b685b2c745725414df86e7fbdf3f5a65a56046633d10aa2fab8ca3e WatchSource:0}: Error finding container 48b69f0f2b685b2c745725414df86e7fbdf3f5a65a56046633d10aa2fab8ca3e: Status 404 returned error can't find the container with id 48b69f0f2b685b2c745725414df86e7fbdf3f5a65a56046633d10aa2fab8ca3e
Sep 18 16:16:48.215604 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:48.215567    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:48.316487 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:48.316402    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:48.427132 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:48.427092    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f5szk" event={"ID":"be090468-5a51-41dc-97fe-8a8569135cef","Type":"ContainerStarted","Data":"882c44696704858f819175343cca4b43c6e3aa81c49ef731902566e59bba7486"}
Sep 18 16:16:48.427286 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:48.427159    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f5szk" event={"ID":"be090468-5a51-41dc-97fe-8a8569135cef","Type":"ContainerStarted","Data":"48b69f0f2b685b2c745725414df86e7fbdf3f5a65a56046633d10aa2fab8ca3e"}
Sep 18 16:16:48.428274 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:48.428241    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-thwr5" event={"ID":"1cea6a79-7d5a-4093-a1b8-0d07198052e3","Type":"ContainerStarted","Data":"c0b1b315c5e3e65f6f9aca730d57fd3e1a04a7bdd12f262bd9af35a8c33c92c9"}
Sep 18 16:16:48.429390 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:48.429364    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-86jqq" event={"ID":"f4ee6c64-ba20-49bc-9d59-c86ce91a51bf","Type":"ContainerStarted","Data":"090b16f4dac210da75f2b479edfaf60dbe130638aeb8ddf4f31c8d1b1e7ca92e"}
Sep 18 16:16:48.633449 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:48.633366    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:48.748464 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:48.748405    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:49.032321 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:49.032280    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:49.116317 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:49.116279    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:16:49.116317 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:49.116320    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:16:49.116552 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:49.116320    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:16:49.120255 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:49.120234    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 16:16:49.120412 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:49.120268    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 16:16:49.120542 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:49.120527    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 16:16:49.120647 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:49.120631    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8kmlw\""
Sep 18 16:16:49.120708 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:49.120647    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 16:16:49.120765 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:49.120732    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-42l89\""
Sep 18 16:16:49.174204 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:49.173845    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:49.274632 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:49.274598    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:49.330577 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:49.330451    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:50.098076 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:50.098041    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:50.199792 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:50.199754    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:50.437269 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:50.437233    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f5szk" event={"ID":"be090468-5a51-41dc-97fe-8a8569135cef","Type":"ContainerStarted","Data":"5c4245d2f49157a2ea6bec86b441a40f00431dad2da02a2c8d5b6c700c7b3eff"}
Sep 18 16:16:50.474929 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:50.474896    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:50.532533 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:50.532487    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:51.241137 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:51.241111    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:51.273967 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:51.273938    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-f6s64"]
Sep 18 16:16:51.298144 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:51.298063    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-f6s64"]
Sep 18 16:16:51.298394 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:51.298225    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-f6s64"
Sep 18 16:16:51.301370 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:51.301290    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 16:16:51.301760 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:51.301655    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 16:16:51.306071 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:51.305120    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-hcpg2\""
Sep 18 16:16:51.306071 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:51.305504    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 18 16:16:51.306071 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:51.305812    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 18 16:16:51.306361 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:51.306111    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 16:16:51.325743 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:51.325717    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:51.395062 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:51.394976    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpfcz\" (UniqueName: \"kubernetes.io/projected/451d27be-cf0b-4599-90a2-8a3ff8ab3e28-kube-api-access-kpfcz\") pod \"prometheus-operator-7bd5d56d6-f6s64\" (UID: \"451d27be-cf0b-4599-90a2-8a3ff8ab3e28\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-f6s64"
Sep 18 16:16:51.395062 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:51.395041    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/451d27be-cf0b-4599-90a2-8a3ff8ab3e28-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-f6s64\" (UID: \"451d27be-cf0b-4599-90a2-8a3ff8ab3e28\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-f6s64"
Sep 18 16:16:51.395273 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:51.395131    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/451d27be-cf0b-4599-90a2-8a3ff8ab3e28-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-f6s64\" (UID: \"451d27be-cf0b-4599-90a2-8a3ff8ab3e28\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-f6s64"
Sep 18 16:16:51.395273 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:51.395177    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/451d27be-cf0b-4599-90a2-8a3ff8ab3e28-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-f6s64\" (UID: \"451d27be-cf0b-4599-90a2-8a3ff8ab3e28\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-f6s64"
Sep 18 16:16:51.442561 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:51.442472    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-thwr5" event={"ID":"1cea6a79-7d5a-4093-a1b8-0d07198052e3","Type":"ContainerStarted","Data":"9cf572cbe5df42a97d8164bf5749c7a8df3c1eb7873c7dcb85c50b5cdf7c0dd0"}
Sep 18 16:16:51.444164 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:51.444134    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-86jqq" event={"ID":"f4ee6c64-ba20-49bc-9d59-c86ce91a51bf","Type":"ContainerStarted","Data":"7f79b21b1ea859e1a5711ca2ab62db80d11f10446cabaa1ca605be77e7a7920d"}
Sep 18 16:16:51.495515 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:51.495482    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/451d27be-cf0b-4599-90a2-8a3ff8ab3e28-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-f6s64\" (UID: \"451d27be-cf0b-4599-90a2-8a3ff8ab3e28\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-f6s64"
Sep 18 16:16:51.495683 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:51.495538    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/451d27be-cf0b-4599-90a2-8a3ff8ab3e28-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-f6s64\" (UID: \"451d27be-cf0b-4599-90a2-8a3ff8ab3e28\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-f6s64"
Sep 18 16:16:51.495683 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:51.495589    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpfcz\" (UniqueName: \"kubernetes.io/projected/451d27be-cf0b-4599-90a2-8a3ff8ab3e28-kube-api-access-kpfcz\") pod \"prometheus-operator-7bd5d56d6-f6s64\" (UID: \"451d27be-cf0b-4599-90a2-8a3ff8ab3e28\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-f6s64"
Sep 18 16:16:51.495683 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:51.495626    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/451d27be-cf0b-4599-90a2-8a3ff8ab3e28-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-f6s64\" (UID: \"451d27be-cf0b-4599-90a2-8a3ff8ab3e28\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-f6s64"
Sep 18 16:16:51.495836 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:51.495812    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 18 16:16:51.495889 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:51.495880    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/451d27be-cf0b-4599-90a2-8a3ff8ab3e28-prometheus-operator-tls podName:451d27be-cf0b-4599-90a2-8a3ff8ab3e28 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:51.995862757 +0000 UTC m=+58.084178379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/451d27be-cf0b-4599-90a2-8a3ff8ab3e28-prometheus-operator-tls") pod "prometheus-operator-7bd5d56d6-f6s64" (UID: "451d27be-cf0b-4599-90a2-8a3ff8ab3e28") : secret "prometheus-operator-tls" not found
Sep 18 16:16:51.496364 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:51.496342    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/451d27be-cf0b-4599-90a2-8a3ff8ab3e28-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-f6s64\" (UID: \"451d27be-cf0b-4599-90a2-8a3ff8ab3e28\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-f6s64"
Sep 18 16:16:51.498463 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:51.498436    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/451d27be-cf0b-4599-90a2-8a3ff8ab3e28-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-f6s64\" (UID: \"451d27be-cf0b-4599-90a2-8a3ff8ab3e28\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-f6s64"
Sep 18 16:16:51.506064 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:51.506041    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpfcz\" (UniqueName: \"kubernetes.io/projected/451d27be-cf0b-4599-90a2-8a3ff8ab3e28-kube-api-access-kpfcz\") pod \"prometheus-operator-7bd5d56d6-f6s64\" (UID: \"451d27be-cf0b-4599-90a2-8a3ff8ab3e28\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-f6s64"
Sep 18 16:16:51.807115 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:51.807081    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:51.998940 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:51.998900    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/451d27be-cf0b-4599-90a2-8a3ff8ab3e28-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-f6s64\" (UID: \"451d27be-cf0b-4599-90a2-8a3ff8ab3e28\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-f6s64"
Sep 18 16:16:52.001526 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:52.001493    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/451d27be-cf0b-4599-90a2-8a3ff8ab3e28-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-f6s64\" (UID: \"451d27be-cf0b-4599-90a2-8a3ff8ab3e28\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-f6s64"
Sep 18 16:16:52.212571 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:52.212537    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-f6s64"
Sep 18 16:16:52.433029 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:52.432989    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:52.449225 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:52.449189    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-86jqq" event={"ID":"f4ee6c64-ba20-49bc-9d59-c86ce91a51bf","Type":"ContainerStarted","Data":"28a609dd20b783a7c3031263d136c29f5836290737d8350df9881f97b4f44bc0"}
Sep 18 16:16:52.469552 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:52.469473    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-thwr5" podStartSLOduration=2.345927904 podStartE2EDuration="5.469455082s" totalImagesPullingTime="3.123527178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:51.463588271 +0000 UTC m=+57.551903914" watchObservedRunningTime="2026-09-18 16:16:52.469455082 +0000 UTC m=+58.557770722"
Sep 18 16:16:52.469788 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:52.469768    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-86jqq" podStartSLOduration=2.342962795 podStartE2EDuration="5.469763194s" totalImagesPullingTime="3.126800399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:52.468609065 +0000 UTC m=+58.556924716" watchObservedRunningTime="2026-09-18 16:16:52.469763194 +0000 UTC m=+58.558078827"
Sep 18 16:16:52.533564 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:52.533536    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:52.634102 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:52.634054    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:52.675590 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:52.675557    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-f6s64"]
Sep 18 16:16:52.678152 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:16:52.678127    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451d27be_cf0b_4599_90a2_8a3ff8ab3e28.slice/crio-10ad5e436fef99284735193de5434d7113ee971417a7d2777ac416f0508fbdc9 WatchSource:0}: Error finding container 10ad5e436fef99284735193de5434d7113ee971417a7d2777ac416f0508fbdc9: Status 404 returned error can't find the container with id 10ad5e436fef99284735193de5434d7113ee971417a7d2777ac416f0508fbdc9
Sep 18 16:16:53.453652 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:53.453618    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-f6s64" event={"ID":"451d27be-cf0b-4599-90a2-8a3ff8ab3e28","Type":"ContainerStarted","Data":"10ad5e436fef99284735193de5434d7113ee971417a7d2777ac416f0508fbdc9"}
Sep 18 16:16:53.456106 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:53.456070    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f5szk" event={"ID":"be090468-5a51-41dc-97fe-8a8569135cef","Type":"ContainerStarted","Data":"17e40f0a835e7343479caeb53b1be2b5fb4e9643ac6d13d495fdac52e7000e1a"}
Sep 18 16:16:53.456320 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:53.456282    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-86jqq"
Sep 18 16:16:53.483110 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:53.483058    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-f5szk" podStartSLOduration=2.363261856 podStartE2EDuration="6.483038633s" totalImagesPullingTime="4.119776777s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:53.482525008 +0000 UTC m=+59.570840648" watchObservedRunningTime="2026-09-18 16:16:53.483038633 +0000 UTC m=+59.571354275"
Sep 18 16:16:54.727488 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:54.727403    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:55.405784 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:55.405748    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:55.466042 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:55.466010    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-f6s64" event={"ID":"451d27be-cf0b-4599-90a2-8a3ff8ab3e28","Type":"ContainerStarted","Data":"288ad693c422bfb9555e89dd44b2fd94f0b440f3af0f6edfe936fbefb926bbaf"}
Sep 18 16:16:55.466191 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:55.466089    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-f6s64" event={"ID":"451d27be-cf0b-4599-90a2-8a3ff8ab3e28","Type":"ContainerStarted","Data":"edbccf9424497448831740b9b7ef415c64f74c7ee85dfc93a1012b6cc4f9912e"}
Sep 18 16:16:55.484441 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:55.484389    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-f6s64" podStartSLOduration=2.733322363 podStartE2EDuration="4.484373213s" totalImagesPullingTime="1.75105085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:55.484193165 +0000 UTC m=+61.572508806" watchObservedRunningTime="2026-09-18 16:16:55.484373213 +0000 UTC m=+61.572688853"
Sep 18 16:16:55.509344 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:55.509297    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:56.888907 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:56.888873    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6645f6d9fc-tntrn"]
Sep 18 16:16:56.891686 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:56.891667    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6645f6d9fc-tntrn"
Sep 18 16:16:56.894346 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:56.894318    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 16:16:56.894454 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:56.894435    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 16:16:56.894517 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:56.894484    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 16:16:56.895497 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:56.895455    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 16:16:56.895497 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:56.895480    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 16:16:56.895497 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:56.895483    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 16:16:56.895702 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:56.895510    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4nj9v\""
Sep 18 16:16:56.895702 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:56.895504    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 16:16:56.900807 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:56.900788    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6645f6d9fc-tntrn"]
Sep 18 16:16:56.932441 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:56.932418    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1efe016f-cd24-44b4-b458-5dd3255e5898-service-ca\") pod \"console-6645f6d9fc-tntrn\" (UID: \"1efe016f-cd24-44b4-b458-5dd3255e5898\") " pod="openshift-console/console-6645f6d9fc-tntrn"
Sep 18 16:16:56.932544 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:56.932447    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1efe016f-cd24-44b4-b458-5dd3255e5898-oauth-serving-cert\") pod \"console-6645f6d9fc-tntrn\" (UID: \"1efe016f-cd24-44b4-b458-5dd3255e5898\") " pod="openshift-console/console-6645f6d9fc-tntrn"
Sep 18 16:16:56.932544 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:56.932464    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh847\" (UniqueName: \"kubernetes.io/projected/1efe016f-cd24-44b4-b458-5dd3255e5898-kube-api-access-dh847\") pod \"console-6645f6d9fc-tntrn\" (UID: \"1efe016f-cd24-44b4-b458-5dd3255e5898\") " pod="openshift-console/console-6645f6d9fc-tntrn"
Sep 18 16:16:56.932620 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:56.932550    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1efe016f-cd24-44b4-b458-5dd3255e5898-console-config\") pod \"console-6645f6d9fc-tntrn\" (UID: \"1efe016f-cd24-44b4-b458-5dd3255e5898\") " pod="openshift-console/console-6645f6d9fc-tntrn"
Sep 18 16:16:56.932620 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:56.932595    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1efe016f-cd24-44b4-b458-5dd3255e5898-console-oauth-config\") pod \"console-6645f6d9fc-tntrn\" (UID: \"1efe016f-cd24-44b4-b458-5dd3255e5898\") " pod="openshift-console/console-6645f6d9fc-tntrn"
Sep 18 16:16:56.932693 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:56.932629    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1efe016f-cd24-44b4-b458-5dd3255e5898-console-serving-cert\") pod \"console-6645f6d9fc-tntrn\" (UID: \"1efe016f-cd24-44b4-b458-5dd3255e5898\") " pod="openshift-console/console-6645f6d9fc-tntrn"
Sep 18 16:16:57.032857 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.032832    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1efe016f-cd24-44b4-b458-5dd3255e5898-service-ca\") pod \"console-6645f6d9fc-tntrn\" (UID: \"1efe016f-cd24-44b4-b458-5dd3255e5898\") " pod="openshift-console/console-6645f6d9fc-tntrn"
Sep 18 16:16:57.032978 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.032866    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1efe016f-cd24-44b4-b458-5dd3255e5898-oauth-serving-cert\") pod \"console-6645f6d9fc-tntrn\" (UID: \"1efe016f-cd24-44b4-b458-5dd3255e5898\") " pod="openshift-console/console-6645f6d9fc-tntrn"
Sep 18 16:16:57.032978 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.032891    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh847\" (UniqueName: \"kubernetes.io/projected/1efe016f-cd24-44b4-b458-5dd3255e5898-kube-api-access-dh847\") pod \"console-6645f6d9fc-tntrn\" (UID: \"1efe016f-cd24-44b4-b458-5dd3255e5898\") " pod="openshift-console/console-6645f6d9fc-tntrn"
Sep 18 16:16:57.032978 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.032952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1efe016f-cd24-44b4-b458-5dd3255e5898-console-config\") pod \"console-6645f6d9fc-tntrn\" (UID: \"1efe016f-cd24-44b4-b458-5dd3255e5898\") " pod="openshift-console/console-6645f6d9fc-tntrn"
Sep 18 16:16:57.033130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.032998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1efe016f-cd24-44b4-b458-5dd3255e5898-console-oauth-config\") pod \"console-6645f6d9fc-tntrn\" (UID: \"1efe016f-cd24-44b4-b458-5dd3255e5898\") " pod="openshift-console/console-6645f6d9fc-tntrn"
Sep 18 16:16:57.033130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.033045    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1efe016f-cd24-44b4-b458-5dd3255e5898-console-serving-cert\") pod \"console-6645f6d9fc-tntrn\" (UID: \"1efe016f-cd24-44b4-b458-5dd3255e5898\") " pod="openshift-console/console-6645f6d9fc-tntrn"
Sep 18 16:16:57.033527 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.033502    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1efe016f-cd24-44b4-b458-5dd3255e5898-oauth-serving-cert\") pod \"console-6645f6d9fc-tntrn\" (UID: \"1efe016f-cd24-44b4-b458-5dd3255e5898\") " pod="openshift-console/console-6645f6d9fc-tntrn"
Sep 18 16:16:57.033669 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.033648    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1efe016f-cd24-44b4-b458-5dd3255e5898-console-config\") pod \"console-6645f6d9fc-tntrn\" (UID: \"1efe016f-cd24-44b4-b458-5dd3255e5898\") " pod="openshift-console/console-6645f6d9fc-tntrn"
Sep 18 16:16:57.033741 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.033648    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1efe016f-cd24-44b4-b458-5dd3255e5898-service-ca\") pod \"console-6645f6d9fc-tntrn\" (UID: \"1efe016f-cd24-44b4-b458-5dd3255e5898\") " pod="openshift-console/console-6645f6d9fc-tntrn"
Sep 18 16:16:57.035376 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.035356    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1efe016f-cd24-44b4-b458-5dd3255e5898-console-oauth-config\") pod \"console-6645f6d9fc-tntrn\" (UID: \"1efe016f-cd24-44b4-b458-5dd3255e5898\") " pod="openshift-console/console-6645f6d9fc-tntrn"
Sep 18 16:16:57.035376 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.035368    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1efe016f-cd24-44b4-b458-5dd3255e5898-console-serving-cert\") pod \"console-6645f6d9fc-tntrn\" (UID: \"1efe016f-cd24-44b4-b458-5dd3255e5898\") " pod="openshift-console/console-6645f6d9fc-tntrn"
Sep 18 16:16:57.041403 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.041386    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh847\" (UniqueName: \"kubernetes.io/projected/1efe016f-cd24-44b4-b458-5dd3255e5898-kube-api-access-dh847\") pod \"console-6645f6d9fc-tntrn\" (UID: \"1efe016f-cd24-44b4-b458-5dd3255e5898\") " pod="openshift-console/console-6645f6d9fc-tntrn"
Sep 18 16:16:57.061386 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.061357    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:57.125654 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.125600    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:57.200460 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.200436    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6645f6d9fc-tntrn"
Sep 18 16:16:57.412843 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.412812    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:57.469997 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.469923    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6645f6d9fc-tntrn"]
Sep 18 16:16:57.472572 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:16:57.472546    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1efe016f_cd24_44b4_b458_5dd3255e5898.slice/crio-c7f99c9756b6ae70641dfd21928e0d5ee1f8b05dda3e8b1616e645556c096d61 WatchSource:0}: Error finding container c7f99c9756b6ae70641dfd21928e0d5ee1f8b05dda3e8b1616e645556c096d61: Status 404 returned error can't find the container with id c7f99c9756b6ae70641dfd21928e0d5ee1f8b05dda3e8b1616e645556c096d61
Sep 18 16:16:57.595119 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.595077    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:57.725320 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.725241    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:57.744990 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.744966    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-kkt48"]
Sep 18 16:16:57.747395 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.747380    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kkt48"
Sep 18 16:16:57.750341 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.750317    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 18 16:16:57.750664 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.750646    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 18 16:16:57.751859 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.751839    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-67mtg\""
Sep 18 16:16:57.761056 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.761035    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-kkt48"]
Sep 18 16:16:57.779048 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.779024    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-4zf52"]
Sep 18 16:16:57.781153 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.781138    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xpc96"]
Sep 18 16:16:57.781274 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.781259    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-4zf52"
Sep 18 16:16:57.783406 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.783391    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.783629 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.783607    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 18 16:16:57.783884 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.783866    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 18 16:16:57.783961 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.783884    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-tmqh2\""
Sep 18 16:16:57.784024 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.784013    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 18 16:16:57.785593 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.785577    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 16:16:57.786126 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.786105    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 16:16:57.786312 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.786278    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 16:16:57.786408 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.786365    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vpzqb\""
Sep 18 16:16:57.796139 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.796121    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-4zf52"]
Sep 18 16:16:57.820693 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.820648    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:57.840268 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.840245    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4eb12922-c4f9-4c98-be06-d213019e4e8f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-4zf52\" (UID: \"4eb12922-c4f9-4c98-be06-d213019e4e8f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4zf52"
Sep 18 16:16:57.840389 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.840275    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4eb12922-c4f9-4c98-be06-d213019e4e8f-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-4zf52\" (UID: \"4eb12922-c4f9-4c98-be06-d213019e4e8f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4zf52"
Sep 18 16:16:57.840389 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.840292    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71e7cb50-46f2-486f-a2bc-12d7443c8c69-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-kkt48\" (UID: \"71e7cb50-46f2-486f-a2bc-12d7443c8c69\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kkt48"
Sep 18 16:16:57.840471 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.840382    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2w29\" (UniqueName: \"kubernetes.io/projected/4eb12922-c4f9-4c98-be06-d213019e4e8f-kube-api-access-x2w29\") pod \"kube-state-metrics-84c885658c-4zf52\" (UID: \"4eb12922-c4f9-4c98-be06-d213019e4e8f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4zf52"
Sep 18 16:16:57.840471 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.840410    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/94263113-b452-406b-86d5-d01e599fcdc3-node-exporter-wtmp\") pod \"node-exporter-xpc96\" (UID: \"94263113-b452-406b-86d5-d01e599fcdc3\") " pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.840471 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.840440    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg7tw\" (UniqueName: \"kubernetes.io/projected/94263113-b452-406b-86d5-d01e599fcdc3-kube-api-access-jg7tw\") pod \"node-exporter-xpc96\" (UID: \"94263113-b452-406b-86d5-d01e599fcdc3\") " pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.840565 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.840476    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/94263113-b452-406b-86d5-d01e599fcdc3-node-exporter-tls\") pod \"node-exporter-xpc96\" (UID: \"94263113-b452-406b-86d5-d01e599fcdc3\") " pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.840565 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.840496    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/94263113-b452-406b-86d5-d01e599fcdc3-node-exporter-textfile\") pod \"node-exporter-xpc96\" (UID: \"94263113-b452-406b-86d5-d01e599fcdc3\") " pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.840565 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.840551    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4eb12922-c4f9-4c98-be06-d213019e4e8f-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-4zf52\" (UID: \"4eb12922-c4f9-4c98-be06-d213019e4e8f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4zf52"
Sep 18 16:16:57.840651 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.840572    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4eb12922-c4f9-4c98-be06-d213019e4e8f-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-4zf52\" (UID: \"4eb12922-c4f9-4c98-be06-d213019e4e8f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4zf52"
Sep 18 16:16:57.840651 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.840587    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmmkb\" (UniqueName: \"kubernetes.io/projected/71e7cb50-46f2-486f-a2bc-12d7443c8c69-kube-api-access-zmmkb\") pod \"openshift-state-metrics-86c5bc79d6-kkt48\" (UID: \"71e7cb50-46f2-486f-a2bc-12d7443c8c69\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kkt48"
Sep 18 16:16:57.840651 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.840613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/71e7cb50-46f2-486f-a2bc-12d7443c8c69-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-kkt48\" (UID: \"71e7cb50-46f2-486f-a2bc-12d7443c8c69\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kkt48"
Sep 18 16:16:57.840651 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.840647    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/94263113-b452-406b-86d5-d01e599fcdc3-root\") pod \"node-exporter-xpc96\" (UID: \"94263113-b452-406b-86d5-d01e599fcdc3\") " pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.840783 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.840667    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/94263113-b452-406b-86d5-d01e599fcdc3-node-exporter-accelerators-collector-config\") pod \"node-exporter-xpc96\" (UID: \"94263113-b452-406b-86d5-d01e599fcdc3\") " pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.840783 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.840690    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71e7cb50-46f2-486f-a2bc-12d7443c8c69-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-kkt48\" (UID: \"71e7cb50-46f2-486f-a2bc-12d7443c8c69\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kkt48"
Sep 18 16:16:57.840783 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.840743    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94263113-b452-406b-86d5-d01e599fcdc3-sys\") pod \"node-exporter-xpc96\" (UID: \"94263113-b452-406b-86d5-d01e599fcdc3\") " pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.840783 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.840775    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94263113-b452-406b-86d5-d01e599fcdc3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xpc96\" (UID: \"94263113-b452-406b-86d5-d01e599fcdc3\") " pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.841035 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.840792    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94263113-b452-406b-86d5-d01e599fcdc3-metrics-client-ca\") pod \"node-exporter-xpc96\" (UID: \"94263113-b452-406b-86d5-d01e599fcdc3\") " pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.841035 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.840810    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4eb12922-c4f9-4c98-be06-d213019e4e8f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-4zf52\" (UID: \"4eb12922-c4f9-4c98-be06-d213019e4e8f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4zf52"
Sep 18 16:16:57.921125 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.921095    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:57.941918 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.941498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2w29\" (UniqueName: \"kubernetes.io/projected/4eb12922-c4f9-4c98-be06-d213019e4e8f-kube-api-access-x2w29\") pod \"kube-state-metrics-84c885658c-4zf52\" (UID: \"4eb12922-c4f9-4c98-be06-d213019e4e8f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4zf52"
Sep 18 16:16:57.942044 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.941938    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/94263113-b452-406b-86d5-d01e599fcdc3-node-exporter-wtmp\") pod \"node-exporter-xpc96\" (UID: \"94263113-b452-406b-86d5-d01e599fcdc3\") " pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.942044 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.941960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg7tw\" (UniqueName: \"kubernetes.io/projected/94263113-b452-406b-86d5-d01e599fcdc3-kube-api-access-jg7tw\") pod \"node-exporter-xpc96\" (UID: \"94263113-b452-406b-86d5-d01e599fcdc3\") " pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.942044 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.941979    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/94263113-b452-406b-86d5-d01e599fcdc3-node-exporter-tls\") pod \"node-exporter-xpc96\" (UID: \"94263113-b452-406b-86d5-d01e599fcdc3\") " pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.942044 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.942000    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/94263113-b452-406b-86d5-d01e599fcdc3-node-exporter-textfile\") pod \"node-exporter-xpc96\" (UID: \"94263113-b452-406b-86d5-d01e599fcdc3\") " pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.942316 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.942150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4eb12922-c4f9-4c98-be06-d213019e4e8f-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-4zf52\" (UID: \"4eb12922-c4f9-4c98-be06-d213019e4e8f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4zf52"
Sep 18 16:16:57.942316 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.942184    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4eb12922-c4f9-4c98-be06-d213019e4e8f-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-4zf52\" (UID: \"4eb12922-c4f9-4c98-be06-d213019e4e8f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4zf52"
Sep 18 16:16:57.942316 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.942166    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/94263113-b452-406b-86d5-d01e599fcdc3-node-exporter-wtmp\") pod \"node-exporter-xpc96\" (UID: \"94263113-b452-406b-86d5-d01e599fcdc3\") " pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.942316 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.942207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmmkb\" (UniqueName: \"kubernetes.io/projected/71e7cb50-46f2-486f-a2bc-12d7443c8c69-kube-api-access-zmmkb\") pod \"openshift-state-metrics-86c5bc79d6-kkt48\" (UID: \"71e7cb50-46f2-486f-a2bc-12d7443c8c69\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kkt48"
Sep 18 16:16:57.942316 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.942239    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/71e7cb50-46f2-486f-a2bc-12d7443c8c69-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-kkt48\" (UID: \"71e7cb50-46f2-486f-a2bc-12d7443c8c69\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kkt48"
Sep 18 16:16:57.942316 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.942291    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/94263113-b452-406b-86d5-d01e599fcdc3-root\") pod \"node-exporter-xpc96\" (UID: \"94263113-b452-406b-86d5-d01e599fcdc3\") " pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.942541 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.942335    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/94263113-b452-406b-86d5-d01e599fcdc3-node-exporter-accelerators-collector-config\") pod \"node-exporter-xpc96\" (UID: \"94263113-b452-406b-86d5-d01e599fcdc3\") " pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.942541 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.942354    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/94263113-b452-406b-86d5-d01e599fcdc3-node-exporter-textfile\") pod \"node-exporter-xpc96\" (UID: \"94263113-b452-406b-86d5-d01e599fcdc3\") " pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.942541 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.942366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71e7cb50-46f2-486f-a2bc-12d7443c8c69-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-kkt48\" (UID: \"71e7cb50-46f2-486f-a2bc-12d7443c8c69\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kkt48"
Sep 18 16:16:57.942541 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.942425    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94263113-b452-406b-86d5-d01e599fcdc3-sys\") pod \"node-exporter-xpc96\" (UID: \"94263113-b452-406b-86d5-d01e599fcdc3\") " pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.942541 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.942470    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94263113-b452-406b-86d5-d01e599fcdc3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xpc96\" (UID: \"94263113-b452-406b-86d5-d01e599fcdc3\") " pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.942541 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.942502    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94263113-b452-406b-86d5-d01e599fcdc3-metrics-client-ca\") pod \"node-exporter-xpc96\" (UID: \"94263113-b452-406b-86d5-d01e599fcdc3\") " pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.942541 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.942528    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4eb12922-c4f9-4c98-be06-d213019e4e8f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-4zf52\" (UID: \"4eb12922-c4f9-4c98-be06-d213019e4e8f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4zf52"
Sep 18 16:16:57.942910 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.942561    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4eb12922-c4f9-4c98-be06-d213019e4e8f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-4zf52\" (UID: \"4eb12922-c4f9-4c98-be06-d213019e4e8f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4zf52"
Sep 18 16:16:57.942910 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.942586    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4eb12922-c4f9-4c98-be06-d213019e4e8f-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-4zf52\" (UID: \"4eb12922-c4f9-4c98-be06-d213019e4e8f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4zf52"
Sep 18 16:16:57.942910 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.942602    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4eb12922-c4f9-4c98-be06-d213019e4e8f-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-4zf52\" (UID: \"4eb12922-c4f9-4c98-be06-d213019e4e8f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4zf52"
Sep 18 16:16:57.942910 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.942626    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71e7cb50-46f2-486f-a2bc-12d7443c8c69-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-kkt48\" (UID: \"71e7cb50-46f2-486f-a2bc-12d7443c8c69\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kkt48"
Sep 18 16:16:57.942910 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.942669    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94263113-b452-406b-86d5-d01e599fcdc3-sys\") pod \"node-exporter-xpc96\" (UID: \"94263113-b452-406b-86d5-d01e599fcdc3\") " pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.942910 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.942708    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/94263113-b452-406b-86d5-d01e599fcdc3-root\") pod \"node-exporter-xpc96\" (UID: \"94263113-b452-406b-86d5-d01e599fcdc3\") " pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.943261 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.943238    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94263113-b452-406b-86d5-d01e599fcdc3-metrics-client-ca\") pod \"node-exporter-xpc96\" (UID: \"94263113-b452-406b-86d5-d01e599fcdc3\") " pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.943500 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.943473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/94263113-b452-406b-86d5-d01e599fcdc3-node-exporter-accelerators-collector-config\") pod \"node-exporter-xpc96\" (UID: \"94263113-b452-406b-86d5-d01e599fcdc3\") " pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.944502 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.943867    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4eb12922-c4f9-4c98-be06-d213019e4e8f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-4zf52\" (UID: \"4eb12922-c4f9-4c98-be06-d213019e4e8f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4zf52"
Sep 18 16:16:57.944502 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.944119    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4eb12922-c4f9-4c98-be06-d213019e4e8f-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-4zf52\" (UID: \"4eb12922-c4f9-4c98-be06-d213019e4e8f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4zf52"
Sep 18 16:16:57.944502 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.944459    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71e7cb50-46f2-486f-a2bc-12d7443c8c69-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-kkt48\" (UID: \"71e7cb50-46f2-486f-a2bc-12d7443c8c69\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kkt48"
Sep 18 16:16:57.945153 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.945135    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/94263113-b452-406b-86d5-d01e599fcdc3-node-exporter-tls\") pod \"node-exporter-xpc96\" (UID: \"94263113-b452-406b-86d5-d01e599fcdc3\") " pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.945264 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.945199    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71e7cb50-46f2-486f-a2bc-12d7443c8c69-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-kkt48\" (UID: \"71e7cb50-46f2-486f-a2bc-12d7443c8c69\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kkt48"
Sep 18 16:16:57.945345 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.945322    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/71e7cb50-46f2-486f-a2bc-12d7443c8c69-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-kkt48\" (UID: \"71e7cb50-46f2-486f-a2bc-12d7443c8c69\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kkt48"
Sep 18 16:16:57.945960 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.945942    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4eb12922-c4f9-4c98-be06-d213019e4e8f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-4zf52\" (UID: \"4eb12922-c4f9-4c98-be06-d213019e4e8f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4zf52"
Sep 18 16:16:57.946193 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.946179    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94263113-b452-406b-86d5-d01e599fcdc3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xpc96\" (UID: \"94263113-b452-406b-86d5-d01e599fcdc3\") " pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.946236 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.946218    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4eb12922-c4f9-4c98-be06-d213019e4e8f-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-4zf52\" (UID: \"4eb12922-c4f9-4c98-be06-d213019e4e8f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4zf52"
Sep 18 16:16:57.952869 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.952850    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg7tw\" (UniqueName: \"kubernetes.io/projected/94263113-b452-406b-86d5-d01e599fcdc3-kube-api-access-jg7tw\") pod \"node-exporter-xpc96\" (UID: \"94263113-b452-406b-86d5-d01e599fcdc3\") " pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:57.953297 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.953277    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2w29\" (UniqueName: \"kubernetes.io/projected/4eb12922-c4f9-4c98-be06-d213019e4e8f-kube-api-access-x2w29\") pod \"kube-state-metrics-84c885658c-4zf52\" (UID: \"4eb12922-c4f9-4c98-be06-d213019e4e8f\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-4zf52"
Sep 18 16:16:57.953384 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:57.953277    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmmkb\" (UniqueName: \"kubernetes.io/projected/71e7cb50-46f2-486f-a2bc-12d7443c8c69-kube-api-access-zmmkb\") pod \"openshift-state-metrics-86c5bc79d6-kkt48\" (UID: \"71e7cb50-46f2-486f-a2bc-12d7443c8c69\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kkt48"
Sep 18 16:16:58.021728 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.021659    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:58.056830 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.056799    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kkt48"
Sep 18 16:16:58.091527 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.091494    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-4zf52"
Sep 18 16:16:58.095785 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.095759    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xpc96"
Sep 18 16:16:58.159405 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.159191    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:58.355971 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.355709    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-kkt48"]
Sep 18 16:16:58.358811 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:16:58.358780    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e7cb50_46f2_486f_a2bc_12d7443c8c69.slice/crio-8378432bf3ca7dab0b15fa7aff1caadf77ed01b968ddfc662376521bdfd35d73 WatchSource:0}: Error finding container 8378432bf3ca7dab0b15fa7aff1caadf77ed01b968ddfc662376521bdfd35d73: Status 404 returned error can't find the container with id 8378432bf3ca7dab0b15fa7aff1caadf77ed01b968ddfc662376521bdfd35d73
Sep 18 16:16:58.392213 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.392187    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-4zf52"]
Sep 18 16:16:58.491232 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.491188    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-4zf52" event={"ID":"4eb12922-c4f9-4c98-be06-d213019e4e8f","Type":"ContainerStarted","Data":"8f2b72b4a5e0ae43d458c3a43212fc3e82d1293ae3dfe8df00796fca2e1a1542"}
Sep 18 16:16:58.493917 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.493853    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xpc96" event={"ID":"94263113-b452-406b-86d5-d01e599fcdc3","Type":"ContainerStarted","Data":"97e46756f38e4c7fa4c46c6906ba884ed8eff3c147104b36178ff89734bf9a30"}
Sep 18 16:16:58.496035 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.496006    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kkt48" event={"ID":"71e7cb50-46f2-486f-a2bc-12d7443c8c69","Type":"ContainerStarted","Data":"5fdfcb6861ce2599c8eecd76b02ec90f65cfa93f2add209b857cca093d3b5f7b"}
Sep 18 16:16:58.496133 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.496068    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kkt48" event={"ID":"71e7cb50-46f2-486f-a2bc-12d7443c8c69","Type":"ContainerStarted","Data":"8378432bf3ca7dab0b15fa7aff1caadf77ed01b968ddfc662376521bdfd35d73"}
Sep 18 16:16:58.498269 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.498221    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6645f6d9fc-tntrn" event={"ID":"1efe016f-cd24-44b4-b458-5dd3255e5898","Type":"ContainerStarted","Data":"c7f99c9756b6ae70641dfd21928e0d5ee1f8b05dda3e8b1616e645556c096d61"}
Sep 18 16:16:58.637355 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.637234    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:58.726429 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.726360    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:58.732486 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.732458    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 16:16:58.737667 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.737643    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.744159 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.744135    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 16:16:58.747104 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.745410    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 16:16:58.747104 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.746744    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 16:16:58.747104 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.746839    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 16:16:58.747104 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.746997    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 16:16:58.747771 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.747219    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 16:16:58.747987 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.745348    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 16:16:58.748617 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.748392    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 16:16:58.748617 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.748403    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mkwh4\""
Sep 18 16:16:58.748844 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.748825    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 16:16:58.750817 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.750093    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.750817 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.750134    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-config-volume\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.750817 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.750158    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9a75de-a38b-4303-a646-4f887cda08ab-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.750817 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.750204    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff9a75de-a38b-4303-a646-4f887cda08ab-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.750817 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.750235    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-web-config\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.750817 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.750264    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff9a75de-a38b-4303-a646-4f887cda08ab-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.750817 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.750317    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.750817 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.750347    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.750817 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.750373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff9a75de-a38b-4303-a646-4f887cda08ab-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.750817 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.750426    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.750817 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.750455    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.750817 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.750483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-529hs\" (UniqueName: \"kubernetes.io/projected/ff9a75de-a38b-4303-a646-4f887cda08ab-kube-api-access-529hs\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.750817 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.750514    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff9a75de-a38b-4303-a646-4f887cda08ab-config-out\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.757030 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.756407    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 16:16:58.851108 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.851070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-config-volume\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.851290 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.851114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9a75de-a38b-4303-a646-4f887cda08ab-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.851290 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.851167    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff9a75de-a38b-4303-a646-4f887cda08ab-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.851290 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.851202    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-web-config\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.851290 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.851233    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff9a75de-a38b-4303-a646-4f887cda08ab-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.851290 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.851264    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.851583 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.851292    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.851583 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.851339    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff9a75de-a38b-4303-a646-4f887cda08ab-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.851583 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.851391    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.851583 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.851426    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.851583 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.851454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-529hs\" (UniqueName: \"kubernetes.io/projected/ff9a75de-a38b-4303-a646-4f887cda08ab-kube-api-access-529hs\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.851583 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.851481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff9a75de-a38b-4303-a646-4f887cda08ab-config-out\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.851583 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.851528    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.851928 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:58.851648    2580 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 18 16:16:58.851928 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.851667    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff9a75de-a38b-4303-a646-4f887cda08ab-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.851928 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:58.851715    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-main-tls podName:ff9a75de-a38b-4303-a646-4f887cda08ab nodeName:}" failed. No retries permitted until 2026-09-18 16:16:59.35169599 +0000 UTC m=+65.440011607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "ff9a75de-a38b-4303-a646-4f887cda08ab") : secret "alertmanager-main-tls" not found
Sep 18 16:16:58.852279 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.852247    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff9a75de-a38b-4303-a646-4f887cda08ab-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.852528 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.852501    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9a75de-a38b-4303-a646-4f887cda08ab-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.855482 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.855423    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.855482 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.855432    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff9a75de-a38b-4303-a646-4f887cda08ab-config-out\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.856224 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.855988    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.856224 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.856074    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.856430 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.856338    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff9a75de-a38b-4303-a646-4f887cda08ab-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.856730 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.856707    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-web-config\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.857023 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.856980    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-config-volume\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.857291 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.857251    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:58.860832 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:58.860805    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-529hs\" (UniqueName: \"kubernetes.io/projected/ff9a75de-a38b-4303-a646-4f887cda08ab-kube-api-access-529hs\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:59.185512 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.185354    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:59.285074 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.284997    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:59.356812 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.356766    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:16:59.356997 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:59.356898    2580 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 18 16:16:59.356997 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:16:59.356970    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-main-tls podName:ff9a75de-a38b-4303-a646-4f887cda08ab nodeName:}" failed. No retries permitted until 2026-09-18 16:17:00.356951906 +0000 UTC m=+66.445267539 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "ff9a75de-a38b-4303-a646-4f887cda08ab") : secret "alertmanager-main-tls" not found
Sep 18 16:16:59.386017 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.385978    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:59.505564 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.505472    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kkt48" event={"ID":"71e7cb50-46f2-486f-a2bc-12d7443c8c69","Type":"ContainerStarted","Data":"0988d0a855d72b9cd84f72c66e881d16954ec5199d3db4bfc151ebf455dcdd8a"}
Sep 18 16:16:59.585736 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.585677    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:59.755560 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.755520    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7f87cdf499-fp65v"]
Sep 18 16:16:59.758507 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.758433    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:16:59.761318 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.761281    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 18 16:16:59.761800 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.761584    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 18 16:16:59.761800 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.761608    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 18 16:16:59.761800 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.761637    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 18 16:16:59.761800 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.761582    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 18 16:16:59.762059 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.761837    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-hz4g7\""
Sep 18 16:16:59.762221 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.762196    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3kmeebvlp65ko\""
Sep 18 16:16:59.773562 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.773532    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f87cdf499-fp65v"]
Sep 18 16:16:59.862480 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.862440    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20fa737b-a4d1-4ddc-af3b-f47a75433a78-metrics-client-ca\") pod \"thanos-querier-7f87cdf499-fp65v\" (UID: \"20fa737b-a4d1-4ddc-af3b-f47a75433a78\") " pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:16:59.862647 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.862566    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/20fa737b-a4d1-4ddc-af3b-f47a75433a78-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f87cdf499-fp65v\" (UID: \"20fa737b-a4d1-4ddc-af3b-f47a75433a78\") " pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:16:59.862647 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.862624    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k8xr\" (UniqueName: \"kubernetes.io/projected/20fa737b-a4d1-4ddc-af3b-f47a75433a78-kube-api-access-2k8xr\") pod \"thanos-querier-7f87cdf499-fp65v\" (UID: \"20fa737b-a4d1-4ddc-af3b-f47a75433a78\") " pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:16:59.862726 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.862657    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/20fa737b-a4d1-4ddc-af3b-f47a75433a78-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f87cdf499-fp65v\" (UID: \"20fa737b-a4d1-4ddc-af3b-f47a75433a78\") " pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:16:59.862726 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.862710    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/20fa737b-a4d1-4ddc-af3b-f47a75433a78-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f87cdf499-fp65v\" (UID: \"20fa737b-a4d1-4ddc-af3b-f47a75433a78\") " pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:16:59.862800 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.862745    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/20fa737b-a4d1-4ddc-af3b-f47a75433a78-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f87cdf499-fp65v\" (UID: \"20fa737b-a4d1-4ddc-af3b-f47a75433a78\") " pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:16:59.862835 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.862818    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/20fa737b-a4d1-4ddc-af3b-f47a75433a78-secret-thanos-querier-tls\") pod \"thanos-querier-7f87cdf499-fp65v\" (UID: \"20fa737b-a4d1-4ddc-af3b-f47a75433a78\") " pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:16:59.862874 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.862835    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/20fa737b-a4d1-4ddc-af3b-f47a75433a78-secret-grpc-tls\") pod \"thanos-querier-7f87cdf499-fp65v\" (UID: \"20fa737b-a4d1-4ddc-af3b-f47a75433a78\") " pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:16:59.963691 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.963653    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/20fa737b-a4d1-4ddc-af3b-f47a75433a78-secret-grpc-tls\") pod \"thanos-querier-7f87cdf499-fp65v\" (UID: \"20fa737b-a4d1-4ddc-af3b-f47a75433a78\") " pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:16:59.963860 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.963766    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20fa737b-a4d1-4ddc-af3b-f47a75433a78-metrics-client-ca\") pod \"thanos-querier-7f87cdf499-fp65v\" (UID: \"20fa737b-a4d1-4ddc-af3b-f47a75433a78\") " pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:16:59.963860 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.963839    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/20fa737b-a4d1-4ddc-af3b-f47a75433a78-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f87cdf499-fp65v\" (UID: \"20fa737b-a4d1-4ddc-af3b-f47a75433a78\") " pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:16:59.963976 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.963874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k8xr\" (UniqueName: \"kubernetes.io/projected/20fa737b-a4d1-4ddc-af3b-f47a75433a78-kube-api-access-2k8xr\") pod \"thanos-querier-7f87cdf499-fp65v\" (UID: \"20fa737b-a4d1-4ddc-af3b-f47a75433a78\") " pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:16:59.963976 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.963904    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/20fa737b-a4d1-4ddc-af3b-f47a75433a78-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f87cdf499-fp65v\" (UID: \"20fa737b-a4d1-4ddc-af3b-f47a75433a78\") " pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:16:59.963976 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.963942    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/20fa737b-a4d1-4ddc-af3b-f47a75433a78-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f87cdf499-fp65v\" (UID: \"20fa737b-a4d1-4ddc-af3b-f47a75433a78\") " pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:16:59.963976 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.963970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/20fa737b-a4d1-4ddc-af3b-f47a75433a78-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f87cdf499-fp65v\" (UID: \"20fa737b-a4d1-4ddc-af3b-f47a75433a78\") " pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:16:59.964172 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.964041    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/20fa737b-a4d1-4ddc-af3b-f47a75433a78-secret-thanos-querier-tls\") pod \"thanos-querier-7f87cdf499-fp65v\" (UID: \"20fa737b-a4d1-4ddc-af3b-f47a75433a78\") " pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:16:59.964677 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.964646    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20fa737b-a4d1-4ddc-af3b-f47a75433a78-metrics-client-ca\") pod \"thanos-querier-7f87cdf499-fp65v\" (UID: \"20fa737b-a4d1-4ddc-af3b-f47a75433a78\") " pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:16:59.967232 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.967185    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/20fa737b-a4d1-4ddc-af3b-f47a75433a78-secret-thanos-querier-tls\") pod \"thanos-querier-7f87cdf499-fp65v\" (UID: \"20fa737b-a4d1-4ddc-af3b-f47a75433a78\") " pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:16:59.967421 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.967283    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/20fa737b-a4d1-4ddc-af3b-f47a75433a78-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f87cdf499-fp65v\" (UID: \"20fa737b-a4d1-4ddc-af3b-f47a75433a78\") " pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:16:59.967658 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.967624    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/20fa737b-a4d1-4ddc-af3b-f47a75433a78-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f87cdf499-fp65v\" (UID: \"20fa737b-a4d1-4ddc-af3b-f47a75433a78\") " pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:16:59.967732 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.967672    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/20fa737b-a4d1-4ddc-af3b-f47a75433a78-secret-grpc-tls\") pod \"thanos-querier-7f87cdf499-fp65v\" (UID: \"20fa737b-a4d1-4ddc-af3b-f47a75433a78\") " pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:16:59.967732 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.967705    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/20fa737b-a4d1-4ddc-af3b-f47a75433a78-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f87cdf499-fp65v\" (UID: \"20fa737b-a4d1-4ddc-af3b-f47a75433a78\") " pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:16:59.968316 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.968271    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/20fa737b-a4d1-4ddc-af3b-f47a75433a78-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f87cdf499-fp65v\" (UID: \"20fa737b-a4d1-4ddc-af3b-f47a75433a78\") " pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:16:59.973099 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:16:59.973075    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k8xr\" (UniqueName: \"kubernetes.io/projected/20fa737b-a4d1-4ddc-af3b-f47a75433a78-kube-api-access-2k8xr\") pod \"thanos-querier-7f87cdf499-fp65v\" (UID: \"20fa737b-a4d1-4ddc-af3b-f47a75433a78\") " pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:17:00.071918 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:00.071626    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:17:00.368061 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:00.367975    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:17:00.370684 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:00.370659    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:17:00.552269 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:00.552217    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:17:00.642770 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:00.642686    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:00.741969 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:00.741876    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:00.846628 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:00.846589    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:01.032051 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:01.031992    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:01.783359 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:01.783320    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c861f27f-910e-48aa-b4bc-3c20bcd911ef-metrics-certs\") pod \"network-metrics-daemon-tnb7c\" (UID: \"c861f27f-910e-48aa-b4bc-3c20bcd911ef\") " pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:17:01.783804 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:01.783422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97bef555-d008-4f5b-a974-91fbafd1eeb1-original-pull-secret\") pod \"global-pull-secret-syncer-cmscw\" (UID: \"97bef555-d008-4f5b-a974-91fbafd1eeb1\") " pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:17:01.786639 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:01.786617    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 16:17:01.787623 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:01.787600    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 16:17:01.796730 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:01.796702    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97bef555-d008-4f5b-a974-91fbafd1eeb1-original-pull-secret\") pod \"global-pull-secret-syncer-cmscw\" (UID: \"97bef555-d008-4f5b-a974-91fbafd1eeb1\") " pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:17:01.796841 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:01.796829    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c861f27f-910e-48aa-b4bc-3c20bcd911ef-metrics-certs\") pod \"network-metrics-daemon-tnb7c\" (UID: \"c861f27f-910e-48aa-b4bc-3c20bcd911ef\") " pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:17:01.884338 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:01.884279    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzt9g\" (UniqueName: \"kubernetes.io/projected/8b313705-71ee-4c64-a650-b308f048dc2a-kube-api-access-kzt9g\") pod \"network-check-target-fs4cj\" (UID: \"8b313705-71ee-4c64-a650-b308f048dc2a\") " pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:17:01.887519 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:01.887497    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 16:17:01.898187 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:01.898167    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 16:17:01.908512 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:01.908489    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzt9g\" (UniqueName: \"kubernetes.io/projected/8b313705-71ee-4c64-a650-b308f048dc2a-kube-api-access-kzt9g\") pod \"network-check-target-fs4cj\" (UID: \"8b313705-71ee-4c64-a650-b308f048dc2a\") " pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:17:02.035728 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.033039    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8kmlw\""
Sep 18 16:17:02.039383 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.039091    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmscw"
Sep 18 16:17:02.039383 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.039334    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tnb7c"
Sep 18 16:17:02.047524 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.047234    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-42l89\""
Sep 18 16:17:02.057125 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.057071    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:17:02.065983 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.065957    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-d7fc747f6-kzpsc"]
Sep 18 16:17:02.068372 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.068351    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc"
Sep 18 16:17:02.072157 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.071789    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-qkblh\""
Sep 18 16:17:02.072157 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.072137    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 18 16:17:02.072157 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.072141    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 18 16:17:02.072380 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.072178    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 18 16:17:02.072380 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.072184    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5rmqi7a3sd1d1\""
Sep 18 16:17:02.072593 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.072572    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 16:17:02.084805 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.083045    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d7fc747f6-kzpsc"]
Sep 18 16:17:02.084805 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.083475    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:02.086618 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.086507    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/af93db19-c810-405f-be6a-15f98057011c-audit-log\") pod \"metrics-server-d7fc747f6-kzpsc\" (UID: \"af93db19-c810-405f-be6a-15f98057011c\") " pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc"
Sep 18 16:17:02.086618 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.086547    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/af93db19-c810-405f-be6a-15f98057011c-metrics-server-audit-profiles\") pod \"metrics-server-d7fc747f6-kzpsc\" (UID: \"af93db19-c810-405f-be6a-15f98057011c\") " pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc"
Sep 18 16:17:02.086814 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.086681    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpz6p\" (UniqueName: \"kubernetes.io/projected/af93db19-c810-405f-be6a-15f98057011c-kube-api-access-vpz6p\") pod \"metrics-server-d7fc747f6-kzpsc\" (UID: \"af93db19-c810-405f-be6a-15f98057011c\") " pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc"
Sep 18 16:17:02.086814 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.086727    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/af93db19-c810-405f-be6a-15f98057011c-secret-metrics-server-client-certs\") pod \"metrics-server-d7fc747f6-kzpsc\" (UID: \"af93db19-c810-405f-be6a-15f98057011c\") " pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc"
Sep 18 16:17:02.086814 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.086761    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af93db19-c810-405f-be6a-15f98057011c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d7fc747f6-kzpsc\" (UID: \"af93db19-c810-405f-be6a-15f98057011c\") " pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc"
Sep 18 16:17:02.086814 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.086808    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/af93db19-c810-405f-be6a-15f98057011c-client-ca-bundle\") pod \"metrics-server-d7fc747f6-kzpsc\" (UID: \"af93db19-c810-405f-be6a-15f98057011c\") " pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc"
Sep 18 16:17:02.087018 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.086840    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/af93db19-c810-405f-be6a-15f98057011c-secret-metrics-server-tls\") pod \"metrics-server-d7fc747f6-kzpsc\" (UID: \"af93db19-c810-405f-be6a-15f98057011c\") " pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc"
Sep 18 16:17:02.183570 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.183498    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:02.187382 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.187347    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/af93db19-c810-405f-be6a-15f98057011c-secret-metrics-server-tls\") pod \"metrics-server-d7fc747f6-kzpsc\" (UID: \"af93db19-c810-405f-be6a-15f98057011c\") " pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc"
Sep 18 16:17:02.187520 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.187393    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/af93db19-c810-405f-be6a-15f98057011c-audit-log\") pod \"metrics-server-d7fc747f6-kzpsc\" (UID: \"af93db19-c810-405f-be6a-15f98057011c\") " pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc"
Sep 18 16:17:02.187520 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.187419    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/af93db19-c810-405f-be6a-15f98057011c-metrics-server-audit-profiles\") pod \"metrics-server-d7fc747f6-kzpsc\" (UID: \"af93db19-c810-405f-be6a-15f98057011c\") " pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc"
Sep 18 16:17:02.187520 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.187484    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpz6p\" (UniqueName: \"kubernetes.io/projected/af93db19-c810-405f-be6a-15f98057011c-kube-api-access-vpz6p\") pod \"metrics-server-d7fc747f6-kzpsc\" (UID: \"af93db19-c810-405f-be6a-15f98057011c\") " pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc"
Sep 18 16:17:02.187675 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.187534    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/af93db19-c810-405f-be6a-15f98057011c-secret-metrics-server-client-certs\") pod \"metrics-server-d7fc747f6-kzpsc\" (UID: \"af93db19-c810-405f-be6a-15f98057011c\") " pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc"
Sep 18 16:17:02.187675 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.187563    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af93db19-c810-405f-be6a-15f98057011c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d7fc747f6-kzpsc\" (UID: \"af93db19-c810-405f-be6a-15f98057011c\") " pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc"
Sep 18 16:17:02.187675 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.187599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/af93db19-c810-405f-be6a-15f98057011c-client-ca-bundle\") pod \"metrics-server-d7fc747f6-kzpsc\" (UID: \"af93db19-c810-405f-be6a-15f98057011c\") " pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc"
Sep 18 16:17:02.187899 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.187850    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/af93db19-c810-405f-be6a-15f98057011c-audit-log\") pod \"metrics-server-d7fc747f6-kzpsc\" (UID: \"af93db19-c810-405f-be6a-15f98057011c\") " pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc"
Sep 18 16:17:02.188397 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.188336    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af93db19-c810-405f-be6a-15f98057011c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d7fc747f6-kzpsc\" (UID: \"af93db19-c810-405f-be6a-15f98057011c\") " pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc"
Sep 18 16:17:02.188860 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.188833    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/af93db19-c810-405f-be6a-15f98057011c-metrics-server-audit-profiles\") pod \"metrics-server-d7fc747f6-kzpsc\" (UID: \"af93db19-c810-405f-be6a-15f98057011c\") " pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc"
Sep 18 16:17:02.190826 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.190801    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/af93db19-c810-405f-be6a-15f98057011c-secret-metrics-server-client-certs\") pod \"metrics-server-d7fc747f6-kzpsc\" (UID: \"af93db19-c810-405f-be6a-15f98057011c\") " pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc"
Sep 18 16:17:02.191364 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.191345    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/af93db19-c810-405f-be6a-15f98057011c-secret-metrics-server-tls\") pod \"metrics-server-d7fc747f6-kzpsc\" (UID: \"af93db19-c810-405f-be6a-15f98057011c\") " pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc"
Sep 18 16:17:02.191880 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.191853    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/af93db19-c810-405f-be6a-15f98057011c-client-ca-bundle\") pod \"metrics-server-d7fc747f6-kzpsc\" (UID: \"af93db19-c810-405f-be6a-15f98057011c\") " pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc"
Sep 18 16:17:02.199544 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.199504    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpz6p\" (UniqueName: \"kubernetes.io/projected/af93db19-c810-405f-be6a-15f98057011c-kube-api-access-vpz6p\") pod \"metrics-server-d7fc747f6-kzpsc\" (UID: \"af93db19-c810-405f-be6a-15f98057011c\") " pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc"
Sep 18 16:17:02.263494 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.263464    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f87cdf499-fp65v"]
Sep 18 16:17:02.273211 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.273171    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 16:17:02.274811 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.274784    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:02.305280 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:17:02.305254    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff9a75de_a38b_4303_a646_4f887cda08ab.slice/crio-442444b556d2efeef0abd6396c0d1df3148b10b09a0676afdd7ca45e90d39ebc WatchSource:0}: Error finding container 442444b556d2efeef0abd6396c0d1df3148b10b09a0676afdd7ca45e90d39ebc: Status 404 returned error can't find the container with id 442444b556d2efeef0abd6396c0d1df3148b10b09a0676afdd7ca45e90d39ebc
Sep 18 16:17:02.382660 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.382617    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc"
Sep 18 16:17:02.457853 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.457798    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:02.500818 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.500779    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-g7fbl"]
Sep 18 16:17:02.503752 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.503730    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-g7fbl"
Sep 18 16:17:02.508349 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.508327    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 18 16:17:02.508491 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.508445    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-qv98t\""
Sep 18 16:17:02.519330 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.519253    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-4zf52" event={"ID":"4eb12922-c4f9-4c98-be06-d213019e4e8f","Type":"ContainerStarted","Data":"5432e3645e62f0328e6c1614711c98e24156f9967ebdd2e442d473fe082d1366"}
Sep 18 16:17:02.520957 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.520927    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff9a75de-a38b-4303-a646-4f887cda08ab","Type":"ContainerStarted","Data":"442444b556d2efeef0abd6396c0d1df3148b10b09a0676afdd7ca45e90d39ebc"}
Sep 18 16:17:02.522680 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.522625    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-g7fbl"]
Sep 18 16:17:02.536417 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:17:02.536384    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20fa737b_a4d1_4ddc_af3b_f47a75433a78.slice/crio-b48a33ce8a42a376535adc37c7e1fbd842d7ed4ff198852cb4c6225c3e1b8433 WatchSource:0}: Error finding container b48a33ce8a42a376535adc37c7e1fbd842d7ed4ff198852cb4c6225c3e1b8433: Status 404 returned error can't find the container with id b48a33ce8a42a376535adc37c7e1fbd842d7ed4ff198852cb4c6225c3e1b8433
Sep 18 16:17:02.591922 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.591572    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lbt2\" (UniqueName: \"kubernetes.io/projected/e94551de-4e76-488b-98ea-1d011e6e08d8-kube-api-access-9lbt2\") pod \"monitoring-plugin-5984459ff8-g7fbl\" (UID: \"e94551de-4e76-488b-98ea-1d011e6e08d8\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-g7fbl"
Sep 18 16:17:02.591922 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.591617    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e94551de-4e76-488b-98ea-1d011e6e08d8-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-g7fbl\" (UID: \"e94551de-4e76-488b-98ea-1d011e6e08d8\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-g7fbl"
Sep 18 16:17:02.591922 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.591722    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:02.692599 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.692491    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lbt2\" (UniqueName: \"kubernetes.io/projected/e94551de-4e76-488b-98ea-1d011e6e08d8-kube-api-access-9lbt2\") pod \"monitoring-plugin-5984459ff8-g7fbl\" (UID: \"e94551de-4e76-488b-98ea-1d011e6e08d8\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-g7fbl"
Sep 18 16:17:02.692599 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.692547    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e94551de-4e76-488b-98ea-1d011e6e08d8-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-g7fbl\" (UID: \"e94551de-4e76-488b-98ea-1d011e6e08d8\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-g7fbl"
Sep 18 16:17:02.698813 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.698755    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e94551de-4e76-488b-98ea-1d011e6e08d8-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-g7fbl\" (UID: \"e94551de-4e76-488b-98ea-1d011e6e08d8\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-g7fbl"
Sep 18 16:17:02.713553 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.713515    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lbt2\" (UniqueName: \"kubernetes.io/projected/e94551de-4e76-488b-98ea-1d011e6e08d8-kube-api-access-9lbt2\") pod \"monitoring-plugin-5984459ff8-g7fbl\" (UID: \"e94551de-4e76-488b-98ea-1d011e6e08d8\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-g7fbl"
Sep 18 16:17:02.794614 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.794536    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:02.814909 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.814830    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-g7fbl"
Sep 18 16:17:02.878777 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.878733    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tnb7c"]
Sep 18 16:17:02.882186 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.882141    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fs4cj"]
Sep 18 16:17:02.886128 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:17:02.886101    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b313705_71ee_4c64_a650_b308f048dc2a.slice/crio-1d522451c9a364b2be5b9147847ad015d682ba6199d170f0081f6b35fd50b4cb WatchSource:0}: Error finding container 1d522451c9a364b2be5b9147847ad015d682ba6199d170f0081f6b35fd50b4cb: Status 404 returned error can't find the container with id 1d522451c9a364b2be5b9147847ad015d682ba6199d170f0081f6b35fd50b4cb
Sep 18 16:17:02.886510 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:17:02.886483    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc861f27f_910e_48aa_b4bc_3c20bcd911ef.slice/crio-a3cd80af72922192859446b4be289268e5453db2883dbf3575fcb8337f9e654a WatchSource:0}: Error finding container a3cd80af72922192859446b4be289268e5453db2883dbf3575fcb8337f9e654a: Status 404 returned error can't find the container with id a3cd80af72922192859446b4be289268e5453db2883dbf3575fcb8337f9e654a
Sep 18 16:17:02.908296 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.908256    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cmscw"]
Sep 18 16:17:02.911045 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:17:02.911022    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97bef555_d008_4f5b_a974_91fbafd1eeb1.slice/crio-3b05583f38af6aea28d409eb4037683a636fd1b949e1b19902bc7d81f609f337 WatchSource:0}: Error finding container 3b05583f38af6aea28d409eb4037683a636fd1b949e1b19902bc7d81f609f337: Status 404 returned error can't find the container with id 3b05583f38af6aea28d409eb4037683a636fd1b949e1b19902bc7d81f609f337
Sep 18 16:17:02.923255 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:02.923224    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d7fc747f6-kzpsc"]
Sep 18 16:17:02.928899 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:17:02.928862    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf93db19_c810_405f_be6a_15f98057011c.slice/crio-5c5f47620183289d8c340783ff97a572ebbe1493f34afc083fff26b00522f0c2 WatchSource:0}: Error finding container 5c5f47620183289d8c340783ff97a572ebbe1493f34afc083fff26b00522f0c2: Status 404 returned error can't find the container with id 5c5f47620183289d8c340783ff97a572ebbe1493f34afc083fff26b00522f0c2
Sep 18 16:17:03.094106 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:03.093903    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-g7fbl"]
Sep 18 16:17:03.096556 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:17:03.096517    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94551de_4e76_488b_98ea_1d011e6e08d8.slice/crio-d4dbdfa96533c22a91c17e6ac07bc748c7377cfdd45381656111a467fde19534 WatchSource:0}: Error finding container d4dbdfa96533c22a91c17e6ac07bc748c7377cfdd45381656111a467fde19534: Status 404 returned error can't find the container with id d4dbdfa96533c22a91c17e6ac07bc748c7377cfdd45381656111a467fde19534
Sep 18 16:17:03.463343 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:03.463278    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-86jqq"
Sep 18 16:17:03.531875 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:03.531822    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kkt48" event={"ID":"71e7cb50-46f2-486f-a2bc-12d7443c8c69","Type":"ContainerStarted","Data":"cf0900dae3c4a5a9480d6e5b40fc216b7a3aee8663a150cf8b1cb25ebcce3a87"}
Sep 18 16:17:03.534627 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:03.534566    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tnb7c" event={"ID":"c861f27f-910e-48aa-b4bc-3c20bcd911ef","Type":"ContainerStarted","Data":"a3cd80af72922192859446b4be289268e5453db2883dbf3575fcb8337f9e654a"}
Sep 18 16:17:03.536595 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:03.536567    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v" event={"ID":"20fa737b-a4d1-4ddc-af3b-f47a75433a78","Type":"ContainerStarted","Data":"b48a33ce8a42a376535adc37c7e1fbd842d7ed4ff198852cb4c6225c3e1b8433"}
Sep 18 16:17:03.539126 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:03.539072    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6645f6d9fc-tntrn" event={"ID":"1efe016f-cd24-44b4-b458-5dd3255e5898","Type":"ContainerStarted","Data":"47a5d6d5b7068370e6f146ce6bffb32406df998780eb56852dcb385a007afb71"}
Sep 18 16:17:03.541167 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:03.541112    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-g7fbl" event={"ID":"e94551de-4e76-488b-98ea-1d011e6e08d8","Type":"ContainerStarted","Data":"d4dbdfa96533c22a91c17e6ac07bc748c7377cfdd45381656111a467fde19534"}
Sep 18 16:17:03.543872 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:03.543846    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc" event={"ID":"af93db19-c810-405f-be6a-15f98057011c","Type":"ContainerStarted","Data":"5c5f47620183289d8c340783ff97a572ebbe1493f34afc083fff26b00522f0c2"}
Sep 18 16:17:03.547073 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:03.547045    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-4zf52" event={"ID":"4eb12922-c4f9-4c98-be06-d213019e4e8f","Type":"ContainerStarted","Data":"fc97b0e0f15771de32d09c18481acf0c842df3bc618358a3968531637eb4bbf1"}
Sep 18 16:17:03.547166 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:03.547107    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-4zf52" event={"ID":"4eb12922-c4f9-4c98-be06-d213019e4e8f","Type":"ContainerStarted","Data":"99b212a6278e9eb5571d70f8a82f53d94de8487388de5a25736980af2e888837"}
Sep 18 16:17:03.549644 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:03.549619    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff9a75de-a38b-4303-a646-4f887cda08ab","Type":"ContainerStarted","Data":"a2bbeb001b8523163477c273b1bdc84df84c5f87b02855acdc4db03c2c490323"}
Sep 18 16:17:03.553528 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:03.553501    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cmscw" event={"ID":"97bef555-d008-4f5b-a974-91fbafd1eeb1","Type":"ContainerStarted","Data":"3b05583f38af6aea28d409eb4037683a636fd1b949e1b19902bc7d81f609f337"}
Sep 18 16:17:03.554135 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:03.554085    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-kkt48" podStartSLOduration=2.506289793 podStartE2EDuration="6.554067794s" totalImagesPullingTime="4.047778001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:17:03.552191013 +0000 UTC m=+69.640506655" watchObservedRunningTime="2026-09-18 16:17:03.554067794 +0000 UTC m=+69.642383433"
Sep 18 16:17:03.556529 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:03.556502    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fs4cj" event={"ID":"8b313705-71ee-4c64-a650-b308f048dc2a","Type":"ContainerStarted","Data":"1d522451c9a364b2be5b9147847ad015d682ba6199d170f0081f6b35fd50b4cb"}
Sep 18 16:17:03.562438 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:03.562411    2580 generic.go:356] "Generic (PLEG): container finished" podID="94263113-b452-406b-86d5-d01e599fcdc3" containerID="6608b1102d251eae340cdfb51ae112044a426dedcd2200b002d7897eec7f8bfa" exitCode=0
Sep 18 16:17:03.562552 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:03.562451    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xpc96" event={"ID":"94263113-b452-406b-86d5-d01e599fcdc3","Type":"ContainerDied","Data":"6608b1102d251eae340cdfb51ae112044a426dedcd2200b002d7897eec7f8bfa"}
Sep 18 16:17:03.619999 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:03.619920    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6645f6d9fc-tntrn" podStartSLOduration=2.532868345 podStartE2EDuration="7.61989709s" totalImagesPullingTime="5.087028745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:17:03.617230458 +0000 UTC m=+69.705546135" watchObservedRunningTime="2026-09-18 16:17:03.61989709 +0000 UTC m=+69.708212732"
Sep 18 16:17:03.642215 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:03.642155    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-4zf52" podStartSLOduration=3.063212825 podStartE2EDuration="6.642132569s" totalImagesPullingTime="3.578919744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:17:03.641129735 +0000 UTC m=+69.729445389" watchObservedRunningTime="2026-09-18 16:17:03.642132569 +0000 UTC m=+69.730448212"
Sep 18 16:17:04.134943 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.134796    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:04.156725 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.154449    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:17:04.162767 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.158044    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.167330 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.166644    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 16:17:04.167330 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.166734    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6s38c4foki2as\""
Sep 18 16:17:04.167330 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.166905    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 16:17:04.167330 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.167086    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-x8tgt\""
Sep 18 16:17:04.167330 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.167105    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 16:17:04.167330 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.167266    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 16:17:04.169927 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.168892    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 16:17:04.169927 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.169155    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 16:17:04.176096 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.175921    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 16:17:04.176235 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.176211    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 16:17:04.176404 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.176379    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 16:17:04.176498 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.176455    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 16:17:04.176728 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.176459    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 16:17:04.184628 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.183003    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 16:17:04.186817 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.186182    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 16:17:04.191774 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.191056    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 16:17:04.203295 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.203257    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:17:04.216124 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.216085    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-web-config\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.216292 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.216135    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c1bed39-9b02-4f28-aea9-364954226a49-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.216292 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.216224    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.216292 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.216276    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.216484 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.216336    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.216484 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.216366    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.216484 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.216407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c1bed39-9b02-4f28-aea9-364954226a49-config-out\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.216484 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.216437    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-config\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.216484 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.216460    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.216716 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.216489    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.216716 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.216514    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.216716 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.216539    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddmkh\" (UniqueName: \"kubernetes.io/projected/8c1bed39-9b02-4f28-aea9-364954226a49-kube-api-access-ddmkh\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.216716 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.216570    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.216716 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.216613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.216716 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.216650    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.216716 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.216677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.216716 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.216709    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.217068 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.216735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.217068 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.216764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.217068 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.216787    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.318348 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.317608    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.318348 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.317657    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.318348 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.317708    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.318348 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.317737    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.318348 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.317764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.318348 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.317788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.318348 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.317827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-web-config\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.318348 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.317853    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c1bed39-9b02-4f28-aea9-364954226a49-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.318348 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.317893    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.318348 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.317919    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.318348 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.317947    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.318348 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.317974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.318348 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.318009    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c1bed39-9b02-4f28-aea9-364954226a49-config-out\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.318348 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.318041    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-config\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.318348 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.318066    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.318348 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.318078    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.318348 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.318096    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.319283 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.318119    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.319283 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.318147    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddmkh\" (UniqueName: \"kubernetes.io/projected/8c1bed39-9b02-4f28-aea9-364954226a49-kube-api-access-ddmkh\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.319283 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.318179    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.319283 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.318221    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.319283 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.318890    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.323859 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.322895    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.323859 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.323057    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.323859 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.323521    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.324106 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.323940    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.326639 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.326260    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.327901 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.327873    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.332778 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.332248    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.348905 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.347052    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-config\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.348905 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.347487    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.348905 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.347608    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.348905 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.347928    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.348905 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.348014    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-web-config\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.350078 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.350032    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddmkh\" (UniqueName: \"kubernetes.io/projected/8c1bed39-9b02-4f28-aea9-364954226a49-kube-api-access-ddmkh\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.352951 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.352879    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.354100 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.354058    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c1bed39-9b02-4f28-aea9-364954226a49-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.354582 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.354556    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.356137 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.356080    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.359102 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.359045    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c1bed39-9b02-4f28-aea9-364954226a49-config-out\") pod \"prometheus-k8s-0\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.359765 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.359736    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:04.478987 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.478632    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:04.566166 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.566117    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:04.578279 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.578244    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerID="a2bbeb001b8523163477c273b1bdc84df84c5f87b02855acdc4db03c2c490323" exitCode=0
Sep 18 16:17:04.578476 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.578369    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff9a75de-a38b-4303-a646-4f887cda08ab","Type":"ContainerDied","Data":"a2bbeb001b8523163477c273b1bdc84df84c5f87b02855acdc4db03c2c490323"}
Sep 18 16:17:04.588233 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.588173    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xpc96" event={"ID":"94263113-b452-406b-86d5-d01e599fcdc3","Type":"ContainerStarted","Data":"298d4490e5b6e4d97f0f0894c2febe3d308a4f6356706ceb93305b10c58ca206"}
Sep 18 16:17:04.588233 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.588236    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xpc96" event={"ID":"94263113-b452-406b-86d5-d01e599fcdc3","Type":"ContainerStarted","Data":"c60e22b99dad1558f24d0bd0108f08ad2e7dc0a5d18861ca3d77631389fed98f"}
Sep 18 16:17:04.608479 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:04.607198    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xpc96" podStartSLOduration=3.062573422 podStartE2EDuration="7.607174393s" totalImagesPullingTime="4.46299782s" totalInitContainerRuntime="81.603151ms" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:17:04.605871055 +0000 UTC m=+70.694186708" watchObservedRunningTime="2026-09-18 16:17:04.607174393 +0000 UTC m=+70.695490035"
Sep 18 16:17:06.218098 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:06.218015    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:06.319471 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:06.319407    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:07.201004 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:07.200964    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6645f6d9fc-tntrn"
Sep 18 16:17:07.201177 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:07.201065    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6645f6d9fc-tntrn"
Sep 18 16:17:07.206384 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:07.206362    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6645f6d9fc-tntrn"
Sep 18 16:17:07.264695 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:07.264653    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:07.399957 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:07.399882    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:07.500698 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:07.500658    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:07.604565 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:07.604536    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6645f6d9fc-tntrn"
Sep 18 16:17:08.546201 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:08.546147    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:08.651689 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:08.651650    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:09.831289 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:09.831014    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:17:10.619858 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:10.619816    2580 generic.go:356] "Generic (PLEG): container finished" podID="8c1bed39-9b02-4f28-aea9-364954226a49" containerID="b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810" exitCode=0
Sep 18 16:17:10.620038 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:10.619946    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c1bed39-9b02-4f28-aea9-364954226a49","Type":"ContainerDied","Data":"b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810"}
Sep 18 16:17:10.620038 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:10.620021    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c1bed39-9b02-4f28-aea9-364954226a49","Type":"ContainerStarted","Data":"1c96331e88b5222a2aa42a7eb9d28429aff9766bcf85fca19a74ec419bd47786"}
Sep 18 16:17:10.627032 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:10.626999    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff9a75de-a38b-4303-a646-4f887cda08ab","Type":"ContainerStarted","Data":"2a637b049fd7793402932066bea5da0f1183fbfd8ea0089a92c3a66b2187f7f1"}
Sep 18 16:17:10.627135 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:10.627098    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff9a75de-a38b-4303-a646-4f887cda08ab","Type":"ContainerStarted","Data":"effebe3b416e9e9699d9dbc22217e74f17f42c169561e5d0b304166cba16591a"}
Sep 18 16:17:10.627203 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:10.627149    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff9a75de-a38b-4303-a646-4f887cda08ab","Type":"ContainerStarted","Data":"860b39aa99af7a3f8cf2607731b71d6c3ac2f2f359794e79b3c4779e17db71cc"}
Sep 18 16:17:10.627203 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:10.627190    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff9a75de-a38b-4303-a646-4f887cda08ab","Type":"ContainerStarted","Data":"32bf07bafb3055b123470be0fcf7848f8f4c771fda8f9029ddacf813ec07e4d2"}
Sep 18 16:17:10.627320 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:10.627231    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff9a75de-a38b-4303-a646-4f887cda08ab","Type":"ContainerStarted","Data":"f7786aee0596e613df0e184d0da6c1f1e80a7c0f981836f3c3ce4d6fdb14ef46"}
Sep 18 16:17:10.629850 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:10.629816    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cmscw" event={"ID":"97bef555-d008-4f5b-a974-91fbafd1eeb1","Type":"ContainerStarted","Data":"777b9005dba3ea03c84cf674f564c03c3695e434f02494c1816fad1f8851e107"}
Sep 18 16:17:10.632609 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:10.632238    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fs4cj" event={"ID":"8b313705-71ee-4c64-a650-b308f048dc2a","Type":"ContainerStarted","Data":"bc6ba59734c9796aa1d7fb24b59efccd71d8c0da8173fcb90bdb50e37a3a7bcd"}
Sep 18 16:17:10.632717 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:10.632678    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:17:10.635457 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:10.635393    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tnb7c" event={"ID":"c861f27f-910e-48aa-b4bc-3c20bcd911ef","Type":"ContainerStarted","Data":"71ad6c4dea17b96fe907ccf74c59c4806c4a24d15f192e7b1901a53770a6755b"}
Sep 18 16:17:10.635457 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:10.635439    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tnb7c" event={"ID":"c861f27f-910e-48aa-b4bc-3c20bcd911ef","Type":"ContainerStarted","Data":"92bb4014d81fa1ea76d2ac51f7bfeaf0dd483759b762da19b0f6d3f83ad111fc"}
Sep 18 16:17:10.639232 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:10.639206    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v" event={"ID":"20fa737b-a4d1-4ddc-af3b-f47a75433a78","Type":"ContainerStarted","Data":"ebb95c06dbd9235e4fb9944cd522e4801a374f1998d29ddd78450e6c95ae09f4"}
Sep 18 16:17:10.639349 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:10.639296    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v" event={"ID":"20fa737b-a4d1-4ddc-af3b-f47a75433a78","Type":"ContainerStarted","Data":"688f0ec19bbf6d6c3dce8da6b5398d4a8e000dd8f918c581807ef42a7ebcde8d"}
Sep 18 16:17:10.639413 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:10.639363    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v" event={"ID":"20fa737b-a4d1-4ddc-af3b-f47a75433a78","Type":"ContainerStarted","Data":"fd4f140a802b82d6aacce9c6564fb160b9355413c644f63abdb9706a9102f2e3"}
Sep 18 16:17:10.643573 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:10.642034    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-g7fbl" event={"ID":"e94551de-4e76-488b-98ea-1d011e6e08d8","Type":"ContainerStarted","Data":"e2db05bcb33941bfbb5a328bc6041e2026b3f4b8ed923dc3e0824bd5a9d7d2b0"}
Sep 18 16:17:10.643573 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:10.642858    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-g7fbl"
Sep 18 16:17:10.646184 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:10.646154    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc" event={"ID":"af93db19-c810-405f-be6a-15f98057011c","Type":"ContainerStarted","Data":"bb7691091c603375883f81abb44d3dffb7b3e216f665887a81f2f3f852bf040a"}
Sep 18 16:17:10.649374 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:10.649279    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-g7fbl"
Sep 18 16:17:10.697767 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:10.697708    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc" podStartSLOduration=2.072067175 podStartE2EDuration="8.697687773s" totalImagesPullingTime="6.625620598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:17:10.696201342 +0000 UTC m=+76.784516982" watchObservedRunningTime="2026-09-18 16:17:10.697687773 +0000 UTC m=+76.786003414"
Sep 18 16:17:10.767971 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:10.767907    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tnb7c" podStartSLOduration=68.150757567 podStartE2EDuration="1m14.767887593s" totalImagesPullingTime="6.617130026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:17:10.767424419 +0000 UTC m=+76.855740059" watchObservedRunningTime="2026-09-18 16:17:10.767887593 +0000 UTC m=+76.856203234"
Sep 18 16:17:10.769862 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:10.769814    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-g7fbl" podStartSLOduration=2.362881471 podStartE2EDuration="8.769794787s" totalImagesPullingTime="6.406913316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:17:10.730677868 +0000 UTC m=+76.818993509" watchObservedRunningTime="2026-09-18 16:17:10.769794787 +0000 UTC m=+76.858110431"
Sep 18 16:17:10.822382 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:10.822323    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fs4cj" podStartSLOduration=68.128850687 podStartE2EDuration="1m14.822282717s" totalImagesPullingTime="6.69343203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:17:10.814717904 +0000 UTC m=+76.903033543" watchObservedRunningTime="2026-09-18 16:17:10.822282717 +0000 UTC m=+76.910598360"
Sep 18 16:17:10.847121 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:10.847069    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cmscw" podStartSLOduration=68.177636794 podStartE2EDuration="1m14.84705213s" totalImagesPullingTime="6.669415336s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:17:10.845731354 +0000 UTC m=+76.934046995" watchObservedRunningTime="2026-09-18 16:17:10.84705213 +0000 UTC m=+76.935367769"
Sep 18 16:17:10.928057 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:10.927983    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:11.036805 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:11.036772    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:11.187022 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:11.186985    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:11.284350 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:11.284318    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:11.653166 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:11.653066    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff9a75de-a38b-4303-a646-4f887cda08ab","Type":"ContainerStarted","Data":"57b27177e444905d6750fe95b90c4c931842b0baaf5d2fa42d5573bfaa11417d"}
Sep 18 16:17:11.656223 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:11.656198    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v" event={"ID":"20fa737b-a4d1-4ddc-af3b-f47a75433a78","Type":"ContainerStarted","Data":"7eff1398fe90a5318d5d41b6eb3b37eaefd23bdd029ef6c053708a7111978e2b"}
Sep 18 16:17:11.656369 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:11.656262    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v" event={"ID":"20fa737b-a4d1-4ddc-af3b-f47a75433a78","Type":"ContainerStarted","Data":"0d1219d6f9a92b1f79b62c19bdc4266d4b07bf18646f76abd3c9369848af0343"}
Sep 18 16:17:11.656369 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:11.656318    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v" event={"ID":"20fa737b-a4d1-4ddc-af3b-f47a75433a78","Type":"ContainerStarted","Data":"3ae14172b2333d0ad40850e84ee7cb40d4572b4291978744c9063acf92094d29"}
Sep 18 16:17:11.656900 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:11.656880    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:17:11.685341 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:11.685278    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.741796827 podStartE2EDuration="13.685259393s" totalImagesPullingTime="7.922535556s" totalInitContainerRuntime="20.92701ms" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:17:11.68409002 +0000 UTC m=+77.772405660" watchObservedRunningTime="2026-09-18 16:17:11.685259393 +0000 UTC m=+77.773575031"
Sep 18 16:17:11.712657 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:11.712610    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v" podStartSLOduration=4.706896986 podStartE2EDuration="12.712597007s" totalImagesPullingTime="8.005700021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:17:11.71138698 +0000 UTC m=+77.799702621" watchObservedRunningTime="2026-09-18 16:17:11.712597007 +0000 UTC m=+77.800912677"
Sep 18 16:17:12.085968 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:12.085917    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:12.186122 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:12.186071    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:13.109667 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:13.109630    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:13.204466 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:13.203741    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:13.695654 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:13.695604    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:13.799964 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:13.799929    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:14.674710 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:14.674678    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c1bed39-9b02-4f28-aea9-364954226a49","Type":"ContainerStarted","Data":"9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94"}
Sep 18 16:17:14.675075 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:14.674742    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c1bed39-9b02-4f28-aea9-364954226a49","Type":"ContainerStarted","Data":"aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e"}
Sep 18 16:17:14.675075 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:14.674768    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c1bed39-9b02-4f28-aea9-364954226a49","Type":"ContainerStarted","Data":"b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf"}
Sep 18 16:17:14.675075 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:14.674796    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c1bed39-9b02-4f28-aea9-364954226a49","Type":"ContainerStarted","Data":"ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64"}
Sep 18 16:17:14.675075 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:14.674820    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c1bed39-9b02-4f28-aea9-364954226a49","Type":"ContainerStarted","Data":"247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849"}
Sep 18 16:17:14.675075 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:14.674843    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c1bed39-9b02-4f28-aea9-364954226a49","Type":"ContainerStarted","Data":"c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a"}
Sep 18 16:17:14.717321 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:14.717241    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=7.295237042 podStartE2EDuration="10.717218351s" totalImagesPullingTime="3.391165388s" totalInitContainerRuntime="30.815921ms" isStatefulPod=false podCreationTimestamp="2026-09-18 16:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:17:14.716205401 +0000 UTC m=+80.804521081" watchObservedRunningTime="2026-09-18 16:17:14.717218351 +0000 UTC m=+80.805533992"
Sep 18 16:17:15.282355 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:15.282325    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:15.373046 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:15.373007    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:16.532449 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:16.532410    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6645f6d9fc-tntrn"]
Sep 18 16:17:16.672571 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:16.672528    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:16.808236 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:16.808166    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:17.322676 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:17.322644    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:17.402950 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:17.402919    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:18.673676 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:18.673644    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7f87cdf499-fp65v"
Sep 18 16:17:19.479056 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:19.479013    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:17:22.383208 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:22.383177    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc"
Sep 18 16:17:22.383570 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:22.383240    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc"
Sep 18 16:17:26.570427 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:26.570384    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:26.647693 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:26.647661    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:41.551967 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:41.551925    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6645f6d9fc-tntrn" podUID="1efe016f-cd24-44b4-b458-5dd3255e5898" containerName="console" containerID="cri-o://47a5d6d5b7068370e6f146ce6bffb32406df998780eb56852dcb385a007afb71" gracePeriod=15
Sep 18 16:17:41.659662 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:41.659632    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fs4cj"
Sep 18 16:17:41.799144 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:41.799113    2580 generic.go:356] "Generic (PLEG): container finished" podID="1efe016f-cd24-44b4-b458-5dd3255e5898" containerID="47a5d6d5b7068370e6f146ce6bffb32406df998780eb56852dcb385a007afb71" exitCode=0
Sep 18 16:17:41.799315 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:41.799191    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6645f6d9fc-tntrn" event={"ID":"1efe016f-cd24-44b4-b458-5dd3255e5898","Type":"ContainerDied","Data":"47a5d6d5b7068370e6f146ce6bffb32406df998780eb56852dcb385a007afb71"}
Sep 18 16:17:41.846707 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:41.846675    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6645f6d9fc-tntrn"
Sep 18 16:17:41.855046 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:41.855019    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1efe016f-cd24-44b4-b458-5dd3255e5898-oauth-serving-cert\") pod \"1efe016f-cd24-44b4-b458-5dd3255e5898\" (UID: \"1efe016f-cd24-44b4-b458-5dd3255e5898\") "
Sep 18 16:17:41.855199 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:41.855082    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dh847\" (UniqueName: \"kubernetes.io/projected/1efe016f-cd24-44b4-b458-5dd3255e5898-kube-api-access-dh847\") pod \"1efe016f-cd24-44b4-b458-5dd3255e5898\" (UID: \"1efe016f-cd24-44b4-b458-5dd3255e5898\") "
Sep 18 16:17:41.855199 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:41.855109    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1efe016f-cd24-44b4-b458-5dd3255e5898-console-oauth-config\") pod \"1efe016f-cd24-44b4-b458-5dd3255e5898\" (UID: \"1efe016f-cd24-44b4-b458-5dd3255e5898\") "
Sep 18 16:17:41.855199 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:41.855136    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1efe016f-cd24-44b4-b458-5dd3255e5898-console-serving-cert\") pod \"1efe016f-cd24-44b4-b458-5dd3255e5898\" (UID: \"1efe016f-cd24-44b4-b458-5dd3255e5898\") "
Sep 18 16:17:41.855199 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:41.855167    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1efe016f-cd24-44b4-b458-5dd3255e5898-console-config\") pod \"1efe016f-cd24-44b4-b458-5dd3255e5898\" (UID: \"1efe016f-cd24-44b4-b458-5dd3255e5898\") "
Sep 18 16:17:41.855422 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:41.855201    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1efe016f-cd24-44b4-b458-5dd3255e5898-service-ca\") pod \"1efe016f-cd24-44b4-b458-5dd3255e5898\" (UID: \"1efe016f-cd24-44b4-b458-5dd3255e5898\") "
Sep 18 16:17:41.855499 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:41.855472    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1efe016f-cd24-44b4-b458-5dd3255e5898-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1efe016f-cd24-44b4-b458-5dd3255e5898" (UID: "1efe016f-cd24-44b4-b458-5dd3255e5898"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:17:41.855659 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:41.855625    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1efe016f-cd24-44b4-b458-5dd3255e5898-console-config" (OuterVolumeSpecName: "console-config") pod "1efe016f-cd24-44b4-b458-5dd3255e5898" (UID: "1efe016f-cd24-44b4-b458-5dd3255e5898"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:17:41.855737 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:41.855657    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1efe016f-cd24-44b4-b458-5dd3255e5898-service-ca" (OuterVolumeSpecName: "service-ca") pod "1efe016f-cd24-44b4-b458-5dd3255e5898" (UID: "1efe016f-cd24-44b4-b458-5dd3255e5898"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:17:41.857613 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:41.857577    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1efe016f-cd24-44b4-b458-5dd3255e5898-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1efe016f-cd24-44b4-b458-5dd3255e5898" (UID: "1efe016f-cd24-44b4-b458-5dd3255e5898"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:17:41.857715 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:41.857615    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1efe016f-cd24-44b4-b458-5dd3255e5898-kube-api-access-dh847" (OuterVolumeSpecName: "kube-api-access-dh847") pod "1efe016f-cd24-44b4-b458-5dd3255e5898" (UID: "1efe016f-cd24-44b4-b458-5dd3255e5898"). InnerVolumeSpecName "kube-api-access-dh847". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:17:41.857715 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:41.857631    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1efe016f-cd24-44b4-b458-5dd3255e5898-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1efe016f-cd24-44b4-b458-5dd3255e5898" (UID: "1efe016f-cd24-44b4-b458-5dd3255e5898"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:17:41.955983 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:41.955951    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh847\" (UniqueName: \"kubernetes.io/projected/1efe016f-cd24-44b4-b458-5dd3255e5898-kube-api-access-dh847\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:17:41.955983 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:41.955978    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1efe016f-cd24-44b4-b458-5dd3255e5898-console-oauth-config\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:17:41.955983 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:41.955987    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1efe016f-cd24-44b4-b458-5dd3255e5898-console-serving-cert\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:17:41.955983 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:41.955997    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1efe016f-cd24-44b4-b458-5dd3255e5898-console-config\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:17:41.956258 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:41.956006    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1efe016f-cd24-44b4-b458-5dd3255e5898-service-ca\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:17:41.956258 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:41.956013    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1efe016f-cd24-44b4-b458-5dd3255e5898-oauth-serving-cert\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:17:42.388194 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:42.388116    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc"
Sep 18 16:17:42.392016 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:42.391990    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-d7fc747f6-kzpsc"
Sep 18 16:17:42.803923 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:42.803895    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6645f6d9fc-tntrn"
Sep 18 16:17:42.804497 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:42.803915    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6645f6d9fc-tntrn" event={"ID":"1efe016f-cd24-44b4-b458-5dd3255e5898","Type":"ContainerDied","Data":"c7f99c9756b6ae70641dfd21928e0d5ee1f8b05dda3e8b1616e645556c096d61"}
Sep 18 16:17:42.804497 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:42.804005    2580 scope.go:117] "RemoveContainer" containerID="47a5d6d5b7068370e6f146ce6bffb32406df998780eb56852dcb385a007afb71"
Sep 18 16:17:42.821188 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:42.821162    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6645f6d9fc-tntrn"]
Sep 18 16:17:42.831714 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:42.831690    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6645f6d9fc-tntrn"]
Sep 18 16:17:44.121359 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:17:44.121328    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1efe016f-cd24-44b4-b458-5dd3255e5898" path="/var/lib/kubelet/pods/1efe016f-cd24-44b4-b458-5dd3255e5898/volumes"
Sep 18 16:18:04.479530 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:04.479477    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:04.499704 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:04.499675    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:04.912239 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:04.912163    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:05.320671 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:05.320629    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:18:16.961484 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:16.961440    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:18:17.364662 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:17.364572    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:18:18.026114 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:18.026074    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 16:18:18.026593 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:18.026510    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerName="alertmanager" containerID="cri-o://f7786aee0596e613df0e184d0da6c1f1e80a7c0f981836f3c3ce4d6fdb14ef46" gracePeriod=120
Sep 18 16:18:18.026593 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:18.026554    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerName="kube-rbac-proxy-metric" containerID="cri-o://2a637b049fd7793402932066bea5da0f1183fbfd8ea0089a92c3a66b2187f7f1" gracePeriod=120
Sep 18 16:18:18.026719 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:18.026569    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerName="kube-rbac-proxy-web" containerID="cri-o://860b39aa99af7a3f8cf2607731b71d6c3ac2f2f359794e79b3c4779e17db71cc" gracePeriod=120
Sep 18 16:18:18.026719 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:18.026615    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerName="kube-rbac-proxy" containerID="cri-o://effebe3b416e9e9699d9dbc22217e74f17f42c169561e5d0b304166cba16591a" gracePeriod=120
Sep 18 16:18:18.026719 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:18.026642    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerName="prom-label-proxy" containerID="cri-o://57b27177e444905d6750fe95b90c4c931842b0baaf5d2fa42d5573bfaa11417d" gracePeriod=120
Sep 18 16:18:18.026719 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:18.026601    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerName="config-reloader" containerID="cri-o://32bf07bafb3055b123470be0fcf7848f8f4c771fda8f9029ddacf813ec07e4d2" gracePeriod=120
Sep 18 16:18:18.954052 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:18.954021    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerID="57b27177e444905d6750fe95b90c4c931842b0baaf5d2fa42d5573bfaa11417d" exitCode=0
Sep 18 16:18:18.954052 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:18.954044    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerID="effebe3b416e9e9699d9dbc22217e74f17f42c169561e5d0b304166cba16591a" exitCode=0
Sep 18 16:18:18.954052 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:18.954050    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerID="32bf07bafb3055b123470be0fcf7848f8f4c771fda8f9029ddacf813ec07e4d2" exitCode=0
Sep 18 16:18:18.954052 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:18.954056    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerID="f7786aee0596e613df0e184d0da6c1f1e80a7c0f981836f3c3ce4d6fdb14ef46" exitCode=0
Sep 18 16:18:18.954320 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:18.954088    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff9a75de-a38b-4303-a646-4f887cda08ab","Type":"ContainerDied","Data":"57b27177e444905d6750fe95b90c4c931842b0baaf5d2fa42d5573bfaa11417d"}
Sep 18 16:18:18.954320 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:18.954149    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff9a75de-a38b-4303-a646-4f887cda08ab","Type":"ContainerDied","Data":"effebe3b416e9e9699d9dbc22217e74f17f42c169561e5d0b304166cba16591a"}
Sep 18 16:18:18.954320 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:18.954174    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff9a75de-a38b-4303-a646-4f887cda08ab","Type":"ContainerDied","Data":"32bf07bafb3055b123470be0fcf7848f8f4c771fda8f9029ddacf813ec07e4d2"}
Sep 18 16:18:18.954320 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:18.954198    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff9a75de-a38b-4303-a646-4f887cda08ab","Type":"ContainerDied","Data":"f7786aee0596e613df0e184d0da6c1f1e80a7c0f981836f3c3ce4d6fdb14ef46"}
Sep 18 16:18:19.267683 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.267660    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:19.281100 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.281075    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff9a75de-a38b-4303-a646-4f887cda08ab-config-out\") pod \"ff9a75de-a38b-4303-a646-4f887cda08ab\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") "
Sep 18 16:18:19.281221 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.281109    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-kube-rbac-proxy\") pod \"ff9a75de-a38b-4303-a646-4f887cda08ab\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") "
Sep 18 16:18:19.281221 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.281149    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-529hs\" (UniqueName: \"kubernetes.io/projected/ff9a75de-a38b-4303-a646-4f887cda08ab-kube-api-access-529hs\") pod \"ff9a75de-a38b-4303-a646-4f887cda08ab\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") "
Sep 18 16:18:19.281221 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.281179    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff9a75de-a38b-4303-a646-4f887cda08ab-metrics-client-ca\") pod \"ff9a75de-a38b-4303-a646-4f887cda08ab\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") "
Sep 18 16:18:19.281221 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.281208    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff9a75de-a38b-4303-a646-4f887cda08ab-tls-assets\") pod \"ff9a75de-a38b-4303-a646-4f887cda08ab\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") "
Sep 18 16:18:19.281446 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.281353    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-kube-rbac-proxy-metric\") pod \"ff9a75de-a38b-4303-a646-4f887cda08ab\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") "
Sep 18 16:18:19.281446 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.281405    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-kube-rbac-proxy-web\") pod \"ff9a75de-a38b-4303-a646-4f887cda08ab\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") "
Sep 18 16:18:19.281446 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.281438    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9a75de-a38b-4303-a646-4f887cda08ab-alertmanager-trusted-ca-bundle\") pod \"ff9a75de-a38b-4303-a646-4f887cda08ab\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") "
Sep 18 16:18:19.281608 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.281470    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-config-volume\") pod \"ff9a75de-a38b-4303-a646-4f887cda08ab\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") "
Sep 18 16:18:19.281608 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.281516    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-main-tls\") pod \"ff9a75de-a38b-4303-a646-4f887cda08ab\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") "
Sep 18 16:18:19.281608 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.281570    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-web-config\") pod \"ff9a75de-a38b-4303-a646-4f887cda08ab\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") "
Sep 18 16:18:19.281608 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.281579    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff9a75de-a38b-4303-a646-4f887cda08ab-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "ff9a75de-a38b-4303-a646-4f887cda08ab" (UID: "ff9a75de-a38b-4303-a646-4f887cda08ab"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:18:19.281608 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.281599    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-cluster-tls-config\") pod \"ff9a75de-a38b-4303-a646-4f887cda08ab\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") "
Sep 18 16:18:19.281863 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.281630    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff9a75de-a38b-4303-a646-4f887cda08ab-alertmanager-main-db\") pod \"ff9a75de-a38b-4303-a646-4f887cda08ab\" (UID: \"ff9a75de-a38b-4303-a646-4f887cda08ab\") "
Sep 18 16:18:19.281930 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.281908    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff9a75de-a38b-4303-a646-4f887cda08ab-metrics-client-ca\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:19.282517 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.282489    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff9a75de-a38b-4303-a646-4f887cda08ab-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "ff9a75de-a38b-4303-a646-4f887cda08ab" (UID: "ff9a75de-a38b-4303-a646-4f887cda08ab"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:18:19.282630 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.282592    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff9a75de-a38b-4303-a646-4f887cda08ab-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "ff9a75de-a38b-4303-a646-4f887cda08ab" (UID: "ff9a75de-a38b-4303-a646-4f887cda08ab"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:18:19.284395 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.284361    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff9a75de-a38b-4303-a646-4f887cda08ab-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ff9a75de-a38b-4303-a646-4f887cda08ab" (UID: "ff9a75de-a38b-4303-a646-4f887cda08ab"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:18:19.284483 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.284406    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-config-volume" (OuterVolumeSpecName: "config-volume") pod "ff9a75de-a38b-4303-a646-4f887cda08ab" (UID: "ff9a75de-a38b-4303-a646-4f887cda08ab"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:18:19.284678 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.284594    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "ff9a75de-a38b-4303-a646-4f887cda08ab" (UID: "ff9a75de-a38b-4303-a646-4f887cda08ab"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:18:19.284835 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.284777    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff9a75de-a38b-4303-a646-4f887cda08ab-kube-api-access-529hs" (OuterVolumeSpecName: "kube-api-access-529hs") pod "ff9a75de-a38b-4303-a646-4f887cda08ab" (UID: "ff9a75de-a38b-4303-a646-4f887cda08ab"). InnerVolumeSpecName "kube-api-access-529hs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:18:19.284935 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.284914    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "ff9a75de-a38b-4303-a646-4f887cda08ab" (UID: "ff9a75de-a38b-4303-a646-4f887cda08ab"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:18:19.284989 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.284930    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff9a75de-a38b-4303-a646-4f887cda08ab-config-out" (OuterVolumeSpecName: "config-out") pod "ff9a75de-a38b-4303-a646-4f887cda08ab" (UID: "ff9a75de-a38b-4303-a646-4f887cda08ab"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:18:19.285443 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.285419    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "ff9a75de-a38b-4303-a646-4f887cda08ab" (UID: "ff9a75de-a38b-4303-a646-4f887cda08ab"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:18:19.286182 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.286160    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "ff9a75de-a38b-4303-a646-4f887cda08ab" (UID: "ff9a75de-a38b-4303-a646-4f887cda08ab"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:18:19.291573 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.291550    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "ff9a75de-a38b-4303-a646-4f887cda08ab" (UID: "ff9a75de-a38b-4303-a646-4f887cda08ab"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:18:19.299086 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.299058    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-web-config" (OuterVolumeSpecName: "web-config") pod "ff9a75de-a38b-4303-a646-4f887cda08ab" (UID: "ff9a75de-a38b-4303-a646-4f887cda08ab"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:18:19.382673 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.382638    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:19.382673 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.382667    2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9a75de-a38b-4303-a646-4f887cda08ab-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:19.382673 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.382677    2580 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-config-volume\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:19.382888 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.382686    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-main-tls\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:19.382888 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.382696    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-web-config\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:19.382888 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.382704    2580 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-cluster-tls-config\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:19.382888 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.382713    2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff9a75de-a38b-4303-a646-4f887cda08ab-alertmanager-main-db\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:19.382888 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.382721    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff9a75de-a38b-4303-a646-4f887cda08ab-config-out\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:19.382888 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.382729    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:19.382888 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.382738    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-529hs\" (UniqueName: \"kubernetes.io/projected/ff9a75de-a38b-4303-a646-4f887cda08ab-kube-api-access-529hs\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:19.382888 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.382746    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff9a75de-a38b-4303-a646-4f887cda08ab-tls-assets\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:19.382888 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.382754    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff9a75de-a38b-4303-a646-4f887cda08ab-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:19.965156 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.965119    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerID="2a637b049fd7793402932066bea5da0f1183fbfd8ea0089a92c3a66b2187f7f1" exitCode=0
Sep 18 16:18:19.965156 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.965146    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerID="860b39aa99af7a3f8cf2607731b71d6c3ac2f2f359794e79b3c4779e17db71cc" exitCode=0
Sep 18 16:18:19.965406 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.965188    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff9a75de-a38b-4303-a646-4f887cda08ab","Type":"ContainerDied","Data":"2a637b049fd7793402932066bea5da0f1183fbfd8ea0089a92c3a66b2187f7f1"}
Sep 18 16:18:19.965406 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.965210    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:19.965406 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.965276    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff9a75de-a38b-4303-a646-4f887cda08ab","Type":"ContainerDied","Data":"860b39aa99af7a3f8cf2607731b71d6c3ac2f2f359794e79b3c4779e17db71cc"}
Sep 18 16:18:19.965406 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.965333    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff9a75de-a38b-4303-a646-4f887cda08ab","Type":"ContainerDied","Data":"442444b556d2efeef0abd6396c0d1df3148b10b09a0676afdd7ca45e90d39ebc"}
Sep 18 16:18:19.965406 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.965354    2580 scope.go:117] "RemoveContainer" containerID="57b27177e444905d6750fe95b90c4c931842b0baaf5d2fa42d5573bfaa11417d"
Sep 18 16:18:19.974523 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.974480    2580 scope.go:117] "RemoveContainer" containerID="2a637b049fd7793402932066bea5da0f1183fbfd8ea0089a92c3a66b2187f7f1"
Sep 18 16:18:19.981468 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.981452    2580 scope.go:117] "RemoveContainer" containerID="effebe3b416e9e9699d9dbc22217e74f17f42c169561e5d0b304166cba16591a"
Sep 18 16:18:19.989070 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.989051    2580 scope.go:117] "RemoveContainer" containerID="860b39aa99af7a3f8cf2607731b71d6c3ac2f2f359794e79b3c4779e17db71cc"
Sep 18 16:18:19.990409 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.990378    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 16:18:19.997656 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.997637    2580 scope.go:117] "RemoveContainer" containerID="32bf07bafb3055b123470be0fcf7848f8f4c771fda8f9029ddacf813ec07e4d2"
Sep 18 16:18:19.997842 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:19.997825    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 16:18:20.004567 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.004550    2580 scope.go:117] "RemoveContainer" containerID="f7786aee0596e613df0e184d0da6c1f1e80a7c0f981836f3c3ce4d6fdb14ef46"
Sep 18 16:18:20.012272 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.012252    2580 scope.go:117] "RemoveContainer" containerID="a2bbeb001b8523163477c273b1bdc84df84c5f87b02855acdc4db03c2c490323"
Sep 18 16:18:20.019615 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.019596    2580 scope.go:117] "RemoveContainer" containerID="57b27177e444905d6750fe95b90c4c931842b0baaf5d2fa42d5573bfaa11417d"
Sep 18 16:18:20.019912 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:18:20.019891    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57b27177e444905d6750fe95b90c4c931842b0baaf5d2fa42d5573bfaa11417d\": container with ID starting with 57b27177e444905d6750fe95b90c4c931842b0baaf5d2fa42d5573bfaa11417d not found: ID does not exist" containerID="57b27177e444905d6750fe95b90c4c931842b0baaf5d2fa42d5573bfaa11417d"
Sep 18 16:18:20.019981 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.019925    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b27177e444905d6750fe95b90c4c931842b0baaf5d2fa42d5573bfaa11417d"} err="failed to get container status \"57b27177e444905d6750fe95b90c4c931842b0baaf5d2fa42d5573bfaa11417d\": rpc error: code = NotFound desc = could not find container \"57b27177e444905d6750fe95b90c4c931842b0baaf5d2fa42d5573bfaa11417d\": container with ID starting with 57b27177e444905d6750fe95b90c4c931842b0baaf5d2fa42d5573bfaa11417d not found: ID does not exist"
Sep 18 16:18:20.019981 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.019969    2580 scope.go:117] "RemoveContainer" containerID="2a637b049fd7793402932066bea5da0f1183fbfd8ea0089a92c3a66b2187f7f1"
Sep 18 16:18:20.020253 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:18:20.020235    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a637b049fd7793402932066bea5da0f1183fbfd8ea0089a92c3a66b2187f7f1\": container with ID starting with 2a637b049fd7793402932066bea5da0f1183fbfd8ea0089a92c3a66b2187f7f1 not found: ID does not exist" containerID="2a637b049fd7793402932066bea5da0f1183fbfd8ea0089a92c3a66b2187f7f1"
Sep 18 16:18:20.020292 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.020259    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a637b049fd7793402932066bea5da0f1183fbfd8ea0089a92c3a66b2187f7f1"} err="failed to get container status \"2a637b049fd7793402932066bea5da0f1183fbfd8ea0089a92c3a66b2187f7f1\": rpc error: code = NotFound desc = could not find container \"2a637b049fd7793402932066bea5da0f1183fbfd8ea0089a92c3a66b2187f7f1\": container with ID starting with 2a637b049fd7793402932066bea5da0f1183fbfd8ea0089a92c3a66b2187f7f1 not found: ID does not exist"
Sep 18 16:18:20.020292 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.020274    2580 scope.go:117] "RemoveContainer" containerID="effebe3b416e9e9699d9dbc22217e74f17f42c169561e5d0b304166cba16591a"
Sep 18 16:18:20.020579 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:18:20.020561    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"effebe3b416e9e9699d9dbc22217e74f17f42c169561e5d0b304166cba16591a\": container with ID starting with effebe3b416e9e9699d9dbc22217e74f17f42c169561e5d0b304166cba16591a not found: ID does not exist" containerID="effebe3b416e9e9699d9dbc22217e74f17f42c169561e5d0b304166cba16591a"
Sep 18 16:18:20.020616 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.020584    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"effebe3b416e9e9699d9dbc22217e74f17f42c169561e5d0b304166cba16591a"} err="failed to get container status \"effebe3b416e9e9699d9dbc22217e74f17f42c169561e5d0b304166cba16591a\": rpc error: code = NotFound desc = could not find container \"effebe3b416e9e9699d9dbc22217e74f17f42c169561e5d0b304166cba16591a\": container with ID starting with effebe3b416e9e9699d9dbc22217e74f17f42c169561e5d0b304166cba16591a not found: ID does not exist"
Sep 18 16:18:20.020616 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.020598    2580 scope.go:117] "RemoveContainer" containerID="860b39aa99af7a3f8cf2607731b71d6c3ac2f2f359794e79b3c4779e17db71cc"
Sep 18 16:18:20.020852 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:18:20.020835    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"860b39aa99af7a3f8cf2607731b71d6c3ac2f2f359794e79b3c4779e17db71cc\": container with ID starting with 860b39aa99af7a3f8cf2607731b71d6c3ac2f2f359794e79b3c4779e17db71cc not found: ID does not exist" containerID="860b39aa99af7a3f8cf2607731b71d6c3ac2f2f359794e79b3c4779e17db71cc"
Sep 18 16:18:20.020913 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.020859    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"860b39aa99af7a3f8cf2607731b71d6c3ac2f2f359794e79b3c4779e17db71cc"} err="failed to get container status \"860b39aa99af7a3f8cf2607731b71d6c3ac2f2f359794e79b3c4779e17db71cc\": rpc error: code = NotFound desc = could not find container \"860b39aa99af7a3f8cf2607731b71d6c3ac2f2f359794e79b3c4779e17db71cc\": container with ID starting with 860b39aa99af7a3f8cf2607731b71d6c3ac2f2f359794e79b3c4779e17db71cc not found: ID does not exist"
Sep 18 16:18:20.020913 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.020879    2580 scope.go:117] "RemoveContainer" containerID="32bf07bafb3055b123470be0fcf7848f8f4c771fda8f9029ddacf813ec07e4d2"
Sep 18 16:18:20.021137 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:18:20.021121    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32bf07bafb3055b123470be0fcf7848f8f4c771fda8f9029ddacf813ec07e4d2\": container with ID starting with 32bf07bafb3055b123470be0fcf7848f8f4c771fda8f9029ddacf813ec07e4d2 not found: ID does not exist" containerID="32bf07bafb3055b123470be0fcf7848f8f4c771fda8f9029ddacf813ec07e4d2"
Sep 18 16:18:20.021180 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.021142    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32bf07bafb3055b123470be0fcf7848f8f4c771fda8f9029ddacf813ec07e4d2"} err="failed to get container status \"32bf07bafb3055b123470be0fcf7848f8f4c771fda8f9029ddacf813ec07e4d2\": rpc error: code = NotFound desc = could not find container \"32bf07bafb3055b123470be0fcf7848f8f4c771fda8f9029ddacf813ec07e4d2\": container with ID starting with 32bf07bafb3055b123470be0fcf7848f8f4c771fda8f9029ddacf813ec07e4d2 not found: ID does not exist"
Sep 18 16:18:20.021180 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.021156    2580 scope.go:117] "RemoveContainer" containerID="f7786aee0596e613df0e184d0da6c1f1e80a7c0f981836f3c3ce4d6fdb14ef46"
Sep 18 16:18:20.021390 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:18:20.021373    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7786aee0596e613df0e184d0da6c1f1e80a7c0f981836f3c3ce4d6fdb14ef46\": container with ID starting with f7786aee0596e613df0e184d0da6c1f1e80a7c0f981836f3c3ce4d6fdb14ef46 not found: ID does not exist" containerID="f7786aee0596e613df0e184d0da6c1f1e80a7c0f981836f3c3ce4d6fdb14ef46"
Sep 18 16:18:20.021436 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.021394    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7786aee0596e613df0e184d0da6c1f1e80a7c0f981836f3c3ce4d6fdb14ef46"} err="failed to get container status \"f7786aee0596e613df0e184d0da6c1f1e80a7c0f981836f3c3ce4d6fdb14ef46\": rpc error: code = NotFound desc = could not find container \"f7786aee0596e613df0e184d0da6c1f1e80a7c0f981836f3c3ce4d6fdb14ef46\": container with ID starting with f7786aee0596e613df0e184d0da6c1f1e80a7c0f981836f3c3ce4d6fdb14ef46 not found: ID does not exist"
Sep 18 16:18:20.021436 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.021406    2580 scope.go:117] "RemoveContainer" containerID="a2bbeb001b8523163477c273b1bdc84df84c5f87b02855acdc4db03c2c490323"
Sep 18 16:18:20.021621 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:18:20.021605    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2bbeb001b8523163477c273b1bdc84df84c5f87b02855acdc4db03c2c490323\": container with ID starting with a2bbeb001b8523163477c273b1bdc84df84c5f87b02855acdc4db03c2c490323 not found: ID does not exist" containerID="a2bbeb001b8523163477c273b1bdc84df84c5f87b02855acdc4db03c2c490323"
Sep 18 16:18:20.021671 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.021622    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2bbeb001b8523163477c273b1bdc84df84c5f87b02855acdc4db03c2c490323"} err="failed to get container status \"a2bbeb001b8523163477c273b1bdc84df84c5f87b02855acdc4db03c2c490323\": rpc error: code = NotFound desc = could not find container \"a2bbeb001b8523163477c273b1bdc84df84c5f87b02855acdc4db03c2c490323\": container with ID starting with a2bbeb001b8523163477c273b1bdc84df84c5f87b02855acdc4db03c2c490323 not found: ID does not exist"
Sep 18 16:18:20.021671 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.021634    2580 scope.go:117] "RemoveContainer" containerID="57b27177e444905d6750fe95b90c4c931842b0baaf5d2fa42d5573bfaa11417d"
Sep 18 16:18:20.021830 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.021813    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b27177e444905d6750fe95b90c4c931842b0baaf5d2fa42d5573bfaa11417d"} err="failed to get container status \"57b27177e444905d6750fe95b90c4c931842b0baaf5d2fa42d5573bfaa11417d\": rpc error: code = NotFound desc = could not find container \"57b27177e444905d6750fe95b90c4c931842b0baaf5d2fa42d5573bfaa11417d\": container with ID starting with 57b27177e444905d6750fe95b90c4c931842b0baaf5d2fa42d5573bfaa11417d not found: ID does not exist"
Sep 18 16:18:20.021892 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.021831    2580 scope.go:117] "RemoveContainer" containerID="2a637b049fd7793402932066bea5da0f1183fbfd8ea0089a92c3a66b2187f7f1"
Sep 18 16:18:20.022073 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.022052    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a637b049fd7793402932066bea5da0f1183fbfd8ea0089a92c3a66b2187f7f1"} err="failed to get container status \"2a637b049fd7793402932066bea5da0f1183fbfd8ea0089a92c3a66b2187f7f1\": rpc error: code = NotFound desc = could not find container \"2a637b049fd7793402932066bea5da0f1183fbfd8ea0089a92c3a66b2187f7f1\": container with ID starting with 2a637b049fd7793402932066bea5da0f1183fbfd8ea0089a92c3a66b2187f7f1 not found: ID does not exist"
Sep 18 16:18:20.022149 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.022074    2580 scope.go:117] "RemoveContainer" containerID="effebe3b416e9e9699d9dbc22217e74f17f42c169561e5d0b304166cba16591a"
Sep 18 16:18:20.022898 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.022351    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"effebe3b416e9e9699d9dbc22217e74f17f42c169561e5d0b304166cba16591a"} err="failed to get container status \"effebe3b416e9e9699d9dbc22217e74f17f42c169561e5d0b304166cba16591a\": rpc error: code = NotFound desc = could not find container \"effebe3b416e9e9699d9dbc22217e74f17f42c169561e5d0b304166cba16591a\": container with ID starting with effebe3b416e9e9699d9dbc22217e74f17f42c169561e5d0b304166cba16591a not found: ID does not exist"
Sep 18 16:18:20.022898 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.022579    2580 scope.go:117] "RemoveContainer" containerID="860b39aa99af7a3f8cf2607731b71d6c3ac2f2f359794e79b3c4779e17db71cc"
Sep 18 16:18:20.023030 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.022958    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"860b39aa99af7a3f8cf2607731b71d6c3ac2f2f359794e79b3c4779e17db71cc"} err="failed to get container status \"860b39aa99af7a3f8cf2607731b71d6c3ac2f2f359794e79b3c4779e17db71cc\": rpc error: code = NotFound desc = could not find container \"860b39aa99af7a3f8cf2607731b71d6c3ac2f2f359794e79b3c4779e17db71cc\": container with ID starting with 860b39aa99af7a3f8cf2607731b71d6c3ac2f2f359794e79b3c4779e17db71cc not found: ID does not exist"
Sep 18 16:18:20.023030 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.022988    2580 scope.go:117] "RemoveContainer" containerID="32bf07bafb3055b123470be0fcf7848f8f4c771fda8f9029ddacf813ec07e4d2"
Sep 18 16:18:20.023531 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.023450    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32bf07bafb3055b123470be0fcf7848f8f4c771fda8f9029ddacf813ec07e4d2"} err="failed to get container status \"32bf07bafb3055b123470be0fcf7848f8f4c771fda8f9029ddacf813ec07e4d2\": rpc error: code = NotFound desc = could not find container \"32bf07bafb3055b123470be0fcf7848f8f4c771fda8f9029ddacf813ec07e4d2\": container with ID starting with 32bf07bafb3055b123470be0fcf7848f8f4c771fda8f9029ddacf813ec07e4d2 not found: ID does not exist"
Sep 18 16:18:20.023531 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.023477    2580 scope.go:117] "RemoveContainer" containerID="f7786aee0596e613df0e184d0da6c1f1e80a7c0f981836f3c3ce4d6fdb14ef46"
Sep 18 16:18:20.023818 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.023795    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7786aee0596e613df0e184d0da6c1f1e80a7c0f981836f3c3ce4d6fdb14ef46"} err="failed to get container status \"f7786aee0596e613df0e184d0da6c1f1e80a7c0f981836f3c3ce4d6fdb14ef46\": rpc error: code = NotFound desc = could not find container \"f7786aee0596e613df0e184d0da6c1f1e80a7c0f981836f3c3ce4d6fdb14ef46\": container with ID starting with f7786aee0596e613df0e184d0da6c1f1e80a7c0f981836f3c3ce4d6fdb14ef46 not found: ID does not exist"
Sep 18 16:18:20.023889 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.023820    2580 scope.go:117] "RemoveContainer" containerID="a2bbeb001b8523163477c273b1bdc84df84c5f87b02855acdc4db03c2c490323"
Sep 18 16:18:20.024110 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.024091    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2bbeb001b8523163477c273b1bdc84df84c5f87b02855acdc4db03c2c490323"} err="failed to get container status \"a2bbeb001b8523163477c273b1bdc84df84c5f87b02855acdc4db03c2c490323\": rpc error: code = NotFound desc = could not find container \"a2bbeb001b8523163477c273b1bdc84df84c5f87b02855acdc4db03c2c490323\": container with ID starting with a2bbeb001b8523163477c273b1bdc84df84c5f87b02855acdc4db03c2c490323 not found: ID does not exist"
Sep 18 16:18:20.024673 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.024657    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 16:18:20.024967 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.024952    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerName="alertmanager"
Sep 18 16:18:20.024967 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.024966    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerName="alertmanager"
Sep 18 16:18:20.025087 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.024977    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerName="prom-label-proxy"
Sep 18 16:18:20.025087 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.024983    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerName="prom-label-proxy"
Sep 18 16:18:20.025087 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.024997    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerName="init-config-reloader"
Sep 18 16:18:20.025087 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.025006    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerName="init-config-reloader"
Sep 18 16:18:20.025087 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.025015    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1efe016f-cd24-44b4-b458-5dd3255e5898" containerName="console"
Sep 18 16:18:20.025087 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.025023    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1efe016f-cd24-44b4-b458-5dd3255e5898" containerName="console"
Sep 18 16:18:20.025087 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.025032    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerName="kube-rbac-proxy"
Sep 18 16:18:20.025087 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.025040    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerName="kube-rbac-proxy"
Sep 18 16:18:20.025087 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.025054    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerName="kube-rbac-proxy-web"
Sep 18 16:18:20.025087 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.025060    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerName="kube-rbac-proxy-web"
Sep 18 16:18:20.025087 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.025070    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerName="config-reloader"
Sep 18 16:18:20.025087 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.025075    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerName="config-reloader"
Sep 18 16:18:20.025087 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.025082    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerName="kube-rbac-proxy-metric"
Sep 18 16:18:20.025087 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.025087    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerName="kube-rbac-proxy-metric"
Sep 18 16:18:20.025636 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.025137    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerName="kube-rbac-proxy-metric"
Sep 18 16:18:20.025636 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.025146    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerName="config-reloader"
Sep 18 16:18:20.025636 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.025152    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1efe016f-cd24-44b4-b458-5dd3255e5898" containerName="console"
Sep 18 16:18:20.025636 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.025161    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerName="kube-rbac-proxy"
Sep 18 16:18:20.025636 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.025170    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerName="kube-rbac-proxy-web"
Sep 18 16:18:20.025636 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.025177    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerName="prom-label-proxy"
Sep 18 16:18:20.025636 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.025186    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff9a75de-a38b-4303-a646-4f887cda08ab" containerName="alertmanager"
Sep 18 16:18:20.028333 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.028315    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.032335 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.030817    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 16:18:20.032335 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.031394    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 16:18:20.032335 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.031679    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 16:18:20.032335 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.031916    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 16:18:20.032335 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.032065    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 16:18:20.032335 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.032231    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 16:18:20.032657 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.032420    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mkwh4\""
Sep 18 16:18:20.032657 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.032488    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 16:18:20.032753 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.032660    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 16:18:20.040834 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.040814    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 16:18:20.051382 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.050535    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 16:18:20.087479 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.087454    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a50e7b55-b1f2-49ae-9065-3123aacf551a-config-volume\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.087582 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.087483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a50e7b55-b1f2-49ae-9065-3123aacf551a-config-out\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.087582 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.087500    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a50e7b55-b1f2-49ae-9065-3123aacf551a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.087582 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.087519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a50e7b55-b1f2-49ae-9065-3123aacf551a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.087678 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.087572    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlx97\" (UniqueName: \"kubernetes.io/projected/a50e7b55-b1f2-49ae-9065-3123aacf551a-kube-api-access-zlx97\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.087678 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.087624    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a50e7b55-b1f2-49ae-9065-3123aacf551a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.087678 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.087656    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a50e7b55-b1f2-49ae-9065-3123aacf551a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.087807 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.087680    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a50e7b55-b1f2-49ae-9065-3123aacf551a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.087807 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.087718    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a50e7b55-b1f2-49ae-9065-3123aacf551a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.087807 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.087788    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a50e7b55-b1f2-49ae-9065-3123aacf551a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.087936 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.087816    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a50e7b55-b1f2-49ae-9065-3123aacf551a-web-config\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.087936 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.087844    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a50e7b55-b1f2-49ae-9065-3123aacf551a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.087936 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.087897    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a50e7b55-b1f2-49ae-9065-3123aacf551a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.124855 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.124831    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff9a75de-a38b-4303-a646-4f887cda08ab" path="/var/lib/kubelet/pods/ff9a75de-a38b-4303-a646-4f887cda08ab/volumes"
Sep 18 16:18:20.188745 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.188717    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a50e7b55-b1f2-49ae-9065-3123aacf551a-config-volume\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.188834 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.188747    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a50e7b55-b1f2-49ae-9065-3123aacf551a-config-out\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.188834 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.188766    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a50e7b55-b1f2-49ae-9065-3123aacf551a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.188834 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.188784    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a50e7b55-b1f2-49ae-9065-3123aacf551a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.188990 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.188914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlx97\" (UniqueName: \"kubernetes.io/projected/a50e7b55-b1f2-49ae-9065-3123aacf551a-kube-api-access-zlx97\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.188990 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.188952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a50e7b55-b1f2-49ae-9065-3123aacf551a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.188990 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.188987    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a50e7b55-b1f2-49ae-9065-3123aacf551a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.189141 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.189010    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a50e7b55-b1f2-49ae-9065-3123aacf551a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.189141 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.189054    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a50e7b55-b1f2-49ae-9065-3123aacf551a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.189141 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.189114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a50e7b55-b1f2-49ae-9065-3123aacf551a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.189289 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.189145    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a50e7b55-b1f2-49ae-9065-3123aacf551a-web-config\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.189289 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.189183    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a50e7b55-b1f2-49ae-9065-3123aacf551a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.189289 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.189219    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a50e7b55-b1f2-49ae-9065-3123aacf551a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.189472 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.189325    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a50e7b55-b1f2-49ae-9065-3123aacf551a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.189799 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.189771    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a50e7b55-b1f2-49ae-9065-3123aacf551a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.191963 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.191655    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a50e7b55-b1f2-49ae-9065-3123aacf551a-config-out\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.191963 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.191823    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a50e7b55-b1f2-49ae-9065-3123aacf551a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.191963 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.191902    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a50e7b55-b1f2-49ae-9065-3123aacf551a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.192163 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.192009    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a50e7b55-b1f2-49ae-9065-3123aacf551a-config-volume\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.192163 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.192097    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a50e7b55-b1f2-49ae-9065-3123aacf551a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.192274 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.192247    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a50e7b55-b1f2-49ae-9065-3123aacf551a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.192531 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.192511    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a50e7b55-b1f2-49ae-9065-3123aacf551a-web-config\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.192531 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.192524    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a50e7b55-b1f2-49ae-9065-3123aacf551a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.192722 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.192707    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a50e7b55-b1f2-49ae-9065-3123aacf551a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.193763 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.193743    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a50e7b55-b1f2-49ae-9065-3123aacf551a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.196270 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.196252    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlx97\" (UniqueName: \"kubernetes.io/projected/a50e7b55-b1f2-49ae-9065-3123aacf551a-kube-api-access-zlx97\") pod \"alertmanager-main-0\" (UID: \"a50e7b55-b1f2-49ae-9065-3123aacf551a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.349095 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.349035    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:18:20.627267 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.627197    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 16:18:20.630574 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:18:20.630549    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda50e7b55_b1f2_49ae_9065_3123aacf551a.slice/crio-6d0809426799955da8871dfb2616618586362bed142adc06329ccf9606d0b8c1 WatchSource:0}: Error finding container 6d0809426799955da8871dfb2616618586362bed142adc06329ccf9606d0b8c1: Status 404 returned error can't find the container with id 6d0809426799955da8871dfb2616618586362bed142adc06329ccf9606d0b8c1
Sep 18 16:18:20.971785 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.971753    2580 generic.go:356] "Generic (PLEG): container finished" podID="a50e7b55-b1f2-49ae-9065-3123aacf551a" containerID="73b3f0d59a4f0b26696a10949998d5b5f982d0eed4cc6abc8c8050a843c1c74d" exitCode=0
Sep 18 16:18:20.971934 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.971839    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a50e7b55-b1f2-49ae-9065-3123aacf551a","Type":"ContainerDied","Data":"73b3f0d59a4f0b26696a10949998d5b5f982d0eed4cc6abc8c8050a843c1c74d"}
Sep 18 16:18:20.971934 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:20.971901    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a50e7b55-b1f2-49ae-9065-3123aacf551a","Type":"ContainerStarted","Data":"6d0809426799955da8871dfb2616618586362bed142adc06329ccf9606d0b8c1"}
Sep 18 16:18:21.978999 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:21.978963    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a50e7b55-b1f2-49ae-9065-3123aacf551a","Type":"ContainerStarted","Data":"e3f3eb434d2959b2c3871198b9a3a038b991fd066f38beee2c79d1ee8498b80c"}
Sep 18 16:18:21.979395 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:21.979030    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a50e7b55-b1f2-49ae-9065-3123aacf551a","Type":"ContainerStarted","Data":"4503e99c547bfed0d704a5fbab91489e76af6b839d0d47d6530e106fa131df0a"}
Sep 18 16:18:21.979395 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:21.979058    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a50e7b55-b1f2-49ae-9065-3123aacf551a","Type":"ContainerStarted","Data":"eb31d2dff0a2f39bd6f34d0f2e0b2fd4c07ac3c1cda7ad535fbcd9b7b937d045"}
Sep 18 16:18:21.979395 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:21.979083    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a50e7b55-b1f2-49ae-9065-3123aacf551a","Type":"ContainerStarted","Data":"7244e533b316599064e122f85d570c5437c1231b164eb9fb395bc8a4c2c4034d"}
Sep 18 16:18:21.979395 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:21.979107    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a50e7b55-b1f2-49ae-9065-3123aacf551a","Type":"ContainerStarted","Data":"09cbac56affe577ce3e292248cdc6ee9668bbf04872e404dcef9b99c09177687"}
Sep 18 16:18:21.979395 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:21.979132    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a50e7b55-b1f2-49ae-9065-3123aacf551a","Type":"ContainerStarted","Data":"ffd76838032cf66871363a22803410a3e927fdf86dcc2b11bf5c870a7a68171f"}
Sep 18 16:18:22.009541 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.009469    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.991177948 podStartE2EDuration="2.009447622s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.269674ms" isStatefulPod=false podCreationTimestamp="2026-09-18 16:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:18:22.007273235 +0000 UTC m=+148.095588876" watchObservedRunningTime="2026-09-18 16:18:22.009447622 +0000 UTC m=+148.097763261"
Sep 18 16:18:22.353915 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.353832    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:18:22.354398 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.354174    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c1bed39-9b02-4f28-aea9-364954226a49" containerName="prometheus" containerID="cri-o://c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a" gracePeriod=600
Sep 18 16:18:22.354398 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.354185    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c1bed39-9b02-4f28-aea9-364954226a49" containerName="kube-rbac-proxy" containerID="cri-o://aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e" gracePeriod=600
Sep 18 16:18:22.354398 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.354218    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c1bed39-9b02-4f28-aea9-364954226a49" containerName="config-reloader" containerID="cri-o://247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849" gracePeriod=600
Sep 18 16:18:22.354398 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.354245    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c1bed39-9b02-4f28-aea9-364954226a49" containerName="kube-rbac-proxy-web" containerID="cri-o://b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf" gracePeriod=600
Sep 18 16:18:22.354398 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.354241    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c1bed39-9b02-4f28-aea9-364954226a49" containerName="thanos-sidecar" containerID="cri-o://ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64" gracePeriod=600
Sep 18 16:18:22.354398 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.354218    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c1bed39-9b02-4f28-aea9-364954226a49" containerName="kube-rbac-proxy-thanos" containerID="cri-o://9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94" gracePeriod=600
Sep 18 16:18:22.594083 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.594048    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:22.609437 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.609376    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-configmap-metrics-client-ca\") pod \"8c1bed39-9b02-4f28-aea9-364954226a49\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") "
Sep 18 16:18:22.609437 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.609408    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c1bed39-9b02-4f28-aea9-364954226a49-config-out\") pod \"8c1bed39-9b02-4f28-aea9-364954226a49\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") "
Sep 18 16:18:22.609437 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.609426    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-k8s-rulefiles-0\") pod \"8c1bed39-9b02-4f28-aea9-364954226a49\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") "
Sep 18 16:18:22.609675 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.609442    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-k8s-rulefiles-1\") pod \"8c1bed39-9b02-4f28-aea9-364954226a49\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") "
Sep 18 16:18:22.609675 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.609466    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-grpc-tls\") pod \"8c1bed39-9b02-4f28-aea9-364954226a49\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") "
Sep 18 16:18:22.609675 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.609494    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-prometheus-k8s-tls\") pod \"8c1bed39-9b02-4f28-aea9-364954226a49\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") "
Sep 18 16:18:22.609839 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.609793    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "8c1bed39-9b02-4f28-aea9-364954226a49" (UID: "8c1bed39-9b02-4f28-aea9-364954226a49"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:18:22.610037 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.610009    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "8c1bed39-9b02-4f28-aea9-364954226a49" (UID: "8c1bed39-9b02-4f28-aea9-364954226a49"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:18:22.610205 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.610156    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-kube-rbac-proxy\") pod \"8c1bed39-9b02-4f28-aea9-364954226a49\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") "
Sep 18 16:18:22.610275 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.610209    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8c1bed39-9b02-4f28-aea9-364954226a49\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") "
Sep 18 16:18:22.610275 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.610239    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddmkh\" (UniqueName: \"kubernetes.io/projected/8c1bed39-9b02-4f28-aea9-364954226a49-kube-api-access-ddmkh\") pod \"8c1bed39-9b02-4f28-aea9-364954226a49\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") "
Sep 18 16:18:22.610392 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.610278    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-thanos-prometheus-http-client-file\") pod \"8c1bed39-9b02-4f28-aea9-364954226a49\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") "
Sep 18 16:18:22.610392 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.610329    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-web-config\") pod \"8c1bed39-9b02-4f28-aea9-364954226a49\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") "
Sep 18 16:18:22.610392 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.610368    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-k8s-db\") pod \"8c1bed39-9b02-4f28-aea9-364954226a49\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") "
Sep 18 16:18:22.610546 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.610396    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c1bed39-9b02-4f28-aea9-364954226a49-tls-assets\") pod \"8c1bed39-9b02-4f28-aea9-364954226a49\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") "
Sep 18 16:18:22.610546 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.610420    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-configmap-kubelet-serving-ca-bundle\") pod \"8c1bed39-9b02-4f28-aea9-364954226a49\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") "
Sep 18 16:18:22.610546 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.610444    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8c1bed39-9b02-4f28-aea9-364954226a49\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") "
Sep 18 16:18:22.610546 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.610506    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-config\") pod \"8c1bed39-9b02-4f28-aea9-364954226a49\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") "
Sep 18 16:18:22.610546 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.610538    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-k8s-rulefiles-2\") pod \"8c1bed39-9b02-4f28-aea9-364954226a49\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") "
Sep 18 16:18:22.610794 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.610561    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-trusted-ca-bundle\") pod \"8c1bed39-9b02-4f28-aea9-364954226a49\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") "
Sep 18 16:18:22.610794 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.610610    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-configmap-serving-certs-ca-bundle\") pod \"8c1bed39-9b02-4f28-aea9-364954226a49\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") "
Sep 18 16:18:22.610794 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.610646    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-metrics-client-certs\") pod \"8c1bed39-9b02-4f28-aea9-364954226a49\" (UID: \"8c1bed39-9b02-4f28-aea9-364954226a49\") "
Sep 18 16:18:22.610932 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.610879    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "8c1bed39-9b02-4f28-aea9-364954226a49" (UID: "8c1bed39-9b02-4f28-aea9-364954226a49"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:18:22.610932 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.610910    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:22.610932 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.610928    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-configmap-metrics-client-ca\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:22.611955 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.611891    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "8c1bed39-9b02-4f28-aea9-364954226a49" (UID: "8c1bed39-9b02-4f28-aea9-364954226a49"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:18:22.613056 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.612773    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "8c1bed39-9b02-4f28-aea9-364954226a49" (UID: "8c1bed39-9b02-4f28-aea9-364954226a49"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:18:22.613923 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.613866    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "8c1bed39-9b02-4f28-aea9-364954226a49" (UID: "8c1bed39-9b02-4f28-aea9-364954226a49"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:18:22.614645 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.614616    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "8c1bed39-9b02-4f28-aea9-364954226a49" (UID: "8c1bed39-9b02-4f28-aea9-364954226a49"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:18:22.615097 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.614915    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "8c1bed39-9b02-4f28-aea9-364954226a49" (UID: "8c1bed39-9b02-4f28-aea9-364954226a49"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:18:22.615189 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.615137    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "8c1bed39-9b02-4f28-aea9-364954226a49" (UID: "8c1bed39-9b02-4f28-aea9-364954226a49"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:18:22.615926 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.615894    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c1bed39-9b02-4f28-aea9-364954226a49-config-out" (OuterVolumeSpecName: "config-out") pod "8c1bed39-9b02-4f28-aea9-364954226a49" (UID: "8c1bed39-9b02-4f28-aea9-364954226a49"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:18:22.616106 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.616077    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "8c1bed39-9b02-4f28-aea9-364954226a49" (UID: "8c1bed39-9b02-4f28-aea9-364954226a49"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:18:22.616258 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.616220    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c1bed39-9b02-4f28-aea9-364954226a49-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8c1bed39-9b02-4f28-aea9-364954226a49" (UID: "8c1bed39-9b02-4f28-aea9-364954226a49"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:18:22.616920 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.616895    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "8c1bed39-9b02-4f28-aea9-364954226a49" (UID: "8c1bed39-9b02-4f28-aea9-364954226a49"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:18:22.617030 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.616994    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "8c1bed39-9b02-4f28-aea9-364954226a49" (UID: "8c1bed39-9b02-4f28-aea9-364954226a49"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:18:22.617131 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.617104    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c1bed39-9b02-4f28-aea9-364954226a49-kube-api-access-ddmkh" (OuterVolumeSpecName: "kube-api-access-ddmkh") pod "8c1bed39-9b02-4f28-aea9-364954226a49" (UID: "8c1bed39-9b02-4f28-aea9-364954226a49"). InnerVolumeSpecName "kube-api-access-ddmkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:18:22.617396 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.617368    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "8c1bed39-9b02-4f28-aea9-364954226a49" (UID: "8c1bed39-9b02-4f28-aea9-364954226a49"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:18:22.617489 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.617463    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-config" (OuterVolumeSpecName: "config") pod "8c1bed39-9b02-4f28-aea9-364954226a49" (UID: "8c1bed39-9b02-4f28-aea9-364954226a49"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:18:22.617566 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.617542    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "8c1bed39-9b02-4f28-aea9-364954226a49" (UID: "8c1bed39-9b02-4f28-aea9-364954226a49"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:18:22.617889 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.617870    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "8c1bed39-9b02-4f28-aea9-364954226a49" (UID: "8c1bed39-9b02-4f28-aea9-364954226a49"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:18:22.628419 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.628388    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-web-config" (OuterVolumeSpecName: "web-config") pod "8c1bed39-9b02-4f28-aea9-364954226a49" (UID: "8c1bed39-9b02-4f28-aea9-364954226a49"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:18:22.716042 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.712451    2580 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-config\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:22.716042 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.712493    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:22.716042 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.712510    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:22.716042 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.712525    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:22.716042 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.712539    2580 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-metrics-client-certs\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:22.716042 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.712556    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c1bed39-9b02-4f28-aea9-364954226a49-config-out\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:22.716042 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.712570    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:22.716042 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.712585    2580 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-grpc-tls\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:22.716042 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.712598    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:22.716042 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.712615    2580 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-kube-rbac-proxy\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:22.716042 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.712630    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:22.716042 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.712646    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddmkh\" (UniqueName: \"kubernetes.io/projected/8c1bed39-9b02-4f28-aea9-364954226a49-kube-api-access-ddmkh\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:22.716042 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.712660    2580 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:22.716042 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.712680    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-web-config\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:22.716042 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.712693    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c1bed39-9b02-4f28-aea9-364954226a49-prometheus-k8s-db\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:22.716042 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.712706    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c1bed39-9b02-4f28-aea9-364954226a49-tls-assets\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:22.716042 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.712719    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c1bed39-9b02-4f28-aea9-364954226a49-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:22.716042 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.712737    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c1bed39-9b02-4f28-aea9-364954226a49-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:18:22.985595 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.985567    2580 generic.go:356] "Generic (PLEG): container finished" podID="8c1bed39-9b02-4f28-aea9-364954226a49" containerID="9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94" exitCode=0
Sep 18 16:18:22.985595 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.985591    2580 generic.go:356] "Generic (PLEG): container finished" podID="8c1bed39-9b02-4f28-aea9-364954226a49" containerID="aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e" exitCode=0
Sep 18 16:18:22.985595 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.985598    2580 generic.go:356] "Generic (PLEG): container finished" podID="8c1bed39-9b02-4f28-aea9-364954226a49" containerID="b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf" exitCode=0
Sep 18 16:18:22.985993 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.985603    2580 generic.go:356] "Generic (PLEG): container finished" podID="8c1bed39-9b02-4f28-aea9-364954226a49" containerID="ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64" exitCode=0
Sep 18 16:18:22.985993 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.985609    2580 generic.go:356] "Generic (PLEG): container finished" podID="8c1bed39-9b02-4f28-aea9-364954226a49" containerID="247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849" exitCode=0
Sep 18 16:18:22.985993 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.985614    2580 generic.go:356] "Generic (PLEG): container finished" podID="8c1bed39-9b02-4f28-aea9-364954226a49" containerID="c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a" exitCode=0
Sep 18 16:18:22.985993 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.985651    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:22.985993 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.985648    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c1bed39-9b02-4f28-aea9-364954226a49","Type":"ContainerDied","Data":"9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94"}
Sep 18 16:18:22.985993 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.985739    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c1bed39-9b02-4f28-aea9-364954226a49","Type":"ContainerDied","Data":"aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e"}
Sep 18 16:18:22.985993 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.985768    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c1bed39-9b02-4f28-aea9-364954226a49","Type":"ContainerDied","Data":"b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf"}
Sep 18 16:18:22.985993 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.985795    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c1bed39-9b02-4f28-aea9-364954226a49","Type":"ContainerDied","Data":"ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64"}
Sep 18 16:18:22.985993 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.985805    2580 scope.go:117] "RemoveContainer" containerID="9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94"
Sep 18 16:18:22.985993 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.985828    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c1bed39-9b02-4f28-aea9-364954226a49","Type":"ContainerDied","Data":"247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849"}
Sep 18 16:18:22.985993 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.985852    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c1bed39-9b02-4f28-aea9-364954226a49","Type":"ContainerDied","Data":"c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a"}
Sep 18 16:18:22.985993 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.985882    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c1bed39-9b02-4f28-aea9-364954226a49","Type":"ContainerDied","Data":"1c96331e88b5222a2aa42a7eb9d28429aff9766bcf85fca19a74ec419bd47786"}
Sep 18 16:18:22.993976 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:22.993958    2580 scope.go:117] "RemoveContainer" containerID="aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e"
Sep 18 16:18:23.000692 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.000677    2580 scope.go:117] "RemoveContainer" containerID="b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf"
Sep 18 16:18:23.007543 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.007522    2580 scope.go:117] "RemoveContainer" containerID="ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64"
Sep 18 16:18:23.008985 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.008968    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:18:23.013788 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.013768    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:18:23.014622 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.014597    2580 scope.go:117] "RemoveContainer" containerID="247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849"
Sep 18 16:18:23.020966 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.020950    2580 scope.go:117] "RemoveContainer" containerID="c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a"
Sep 18 16:18:23.027620 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.027606    2580 scope.go:117] "RemoveContainer" containerID="b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810"
Sep 18 16:18:23.033959 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.033942    2580 scope.go:117] "RemoveContainer" containerID="9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94"
Sep 18 16:18:23.034215 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:18:23.034198    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94\": container with ID starting with 9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94 not found: ID does not exist" containerID="9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94"
Sep 18 16:18:23.034270 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.034222    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94"} err="failed to get container status \"9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94\": rpc error: code = NotFound desc = could not find container \"9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94\": container with ID starting with 9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94 not found: ID does not exist"
Sep 18 16:18:23.034270 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.034239    2580 scope.go:117] "RemoveContainer" containerID="aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e"
Sep 18 16:18:23.034475 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:18:23.034459    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e\": container with ID starting with aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e not found: ID does not exist" containerID="aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e"
Sep 18 16:18:23.034529 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.034480    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e"} err="failed to get container status \"aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e\": rpc error: code = NotFound desc = could not find container \"aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e\": container with ID starting with aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e not found: ID does not exist"
Sep 18 16:18:23.034529 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.034493    2580 scope.go:117] "RemoveContainer" containerID="b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf"
Sep 18 16:18:23.034751 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:18:23.034735    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf\": container with ID starting with b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf not found: ID does not exist" containerID="b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf"
Sep 18 16:18:23.034801 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.034754    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf"} err="failed to get container status \"b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf\": rpc error: code = NotFound desc = could not find container \"b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf\": container with ID starting with b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf not found: ID does not exist"
Sep 18 16:18:23.034801 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.034767    2580 scope.go:117] "RemoveContainer" containerID="ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64"
Sep 18 16:18:23.034977 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:18:23.034961    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64\": container with ID starting with ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64 not found: ID does not exist" containerID="ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64"
Sep 18 16:18:23.035012 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.034981    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64"} err="failed to get container status \"ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64\": rpc error: code = NotFound desc = could not find container \"ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64\": container with ID starting with ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64 not found: ID does not exist"
Sep 18 16:18:23.035012 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.034994    2580 scope.go:117] "RemoveContainer" containerID="247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849"
Sep 18 16:18:23.035208 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:18:23.035192    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849\": container with ID starting with 247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849 not found: ID does not exist" containerID="247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849"
Sep 18 16:18:23.035250 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.035212    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849"} err="failed to get container status \"247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849\": rpc error: code = NotFound desc = could not find container \"247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849\": container with ID starting with 247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849 not found: ID does not exist"
Sep 18 16:18:23.035250 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.035223    2580 scope.go:117] "RemoveContainer" containerID="c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a"
Sep 18 16:18:23.035429 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:18:23.035414    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a\": container with ID starting with c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a not found: ID does not exist" containerID="c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a"
Sep 18 16:18:23.035474 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.035433    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a"} err="failed to get container status \"c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a\": rpc error: code = NotFound desc = could not find container \"c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a\": container with ID starting with c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a not found: ID does not exist"
Sep 18 16:18:23.035474 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.035444    2580 scope.go:117] "RemoveContainer" containerID="b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810"
Sep 18 16:18:23.035672 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:18:23.035656    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810\": container with ID starting with b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810 not found: ID does not exist" containerID="b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810"
Sep 18 16:18:23.035713 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.035675    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810"} err="failed to get container status \"b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810\": rpc error: code = NotFound desc = could not find container \"b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810\": container with ID starting with b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810 not found: ID does not exist"
Sep 18 16:18:23.035713 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.035687    2580 scope.go:117] "RemoveContainer" containerID="9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94"
Sep 18 16:18:23.035900 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.035884    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94"} err="failed to get container status \"9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94\": rpc error: code = NotFound desc = could not find container \"9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94\": container with ID starting with 9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94 not found: ID does not exist"
Sep 18 16:18:23.035942 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.035901    2580 scope.go:117] "RemoveContainer" containerID="aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e"
Sep 18 16:18:23.036107 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.036091    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e"} err="failed to get container status \"aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e\": rpc error: code = NotFound desc = could not find container \"aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e\": container with ID starting with aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e not found: ID does not exist"
Sep 18 16:18:23.036107 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.036107    2580 scope.go:117] "RemoveContainer" containerID="b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf"
Sep 18 16:18:23.036296 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.036281    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf"} err="failed to get container status \"b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf\": rpc error: code = NotFound desc = could not find container \"b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf\": container with ID starting with b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf not found: ID does not exist"
Sep 18 16:18:23.036296 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.036311    2580 scope.go:117] "RemoveContainer" containerID="ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64"
Sep 18 16:18:23.036564 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.036545    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64"} err="failed to get container status \"ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64\": rpc error: code = NotFound desc = could not find container \"ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64\": container with ID starting with ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64 not found: ID does not exist"
Sep 18 16:18:23.036609 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.036565    2580 scope.go:117] "RemoveContainer" containerID="247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849"
Sep 18 16:18:23.036789 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.036774    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849"} err="failed to get container status \"247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849\": rpc error: code = NotFound desc = could not find container \"247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849\": container with ID starting with 247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849 not found: ID does not exist"
Sep 18 16:18:23.036831 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.036789    2580 scope.go:117] "RemoveContainer" containerID="c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a"
Sep 18 16:18:23.036998 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.036979    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a"} err="failed to get container status \"c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a\": rpc error: code = NotFound desc = could not find container \"c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a\": container with ID starting with c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a not found: ID does not exist"
Sep 18 16:18:23.037050 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.036999    2580 scope.go:117] "RemoveContainer" containerID="b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810"
Sep 18 16:18:23.037200 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.037184    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810"} err="failed to get container status \"b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810\": rpc error: code = NotFound desc = could not find container \"b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810\": container with ID starting with b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810 not found: ID does not exist"
Sep 18 16:18:23.037200 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.037199    2580 scope.go:117] "RemoveContainer" containerID="9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94"
Sep 18 16:18:23.037419 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.037403    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94"} err="failed to get container status \"9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94\": rpc error: code = NotFound desc = could not find container \"9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94\": container with ID starting with 9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94 not found: ID does not exist"
Sep 18 16:18:23.037463 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.037420    2580 scope.go:117] "RemoveContainer" containerID="aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e"
Sep 18 16:18:23.037630 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.037614    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e"} err="failed to get container status \"aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e\": rpc error: code = NotFound desc = could not find container \"aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e\": container with ID starting with aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e not found: ID does not exist"
Sep 18 16:18:23.037675 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.037630    2580 scope.go:117] "RemoveContainer" containerID="b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf"
Sep 18 16:18:23.037857 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.037842    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf"} err="failed to get container status \"b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf\": rpc error: code = NotFound desc = could not find container \"b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf\": container with ID starting with b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf not found: ID does not exist"
Sep 18 16:18:23.037857 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.037856    2580 scope.go:117] "RemoveContainer" containerID="ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64"
Sep 18 16:18:23.038052 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.038036    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64"} err="failed to get container status \"ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64\": rpc error: code = NotFound desc = could not find container \"ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64\": container with ID starting with ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64 not found: ID does not exist"
Sep 18 16:18:23.038094 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.038052    2580 scope.go:117] "RemoveContainer" containerID="247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849"
Sep 18 16:18:23.038235 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.038218    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849"} err="failed to get container status \"247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849\": rpc error: code = NotFound desc = could not find container \"247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849\": container with ID starting with 247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849 not found: ID does not exist"
Sep 18 16:18:23.038286 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.038235    2580 scope.go:117] "RemoveContainer" containerID="c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a"
Sep 18 16:18:23.038463 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.038447    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a"} err="failed to get container status \"c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a\": rpc error: code = NotFound desc = could not find container \"c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a\": container with ID starting with c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a not found: ID does not exist"
Sep 18 16:18:23.038463 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.038462    2580 scope.go:117] "RemoveContainer" containerID="b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810"
Sep 18 16:18:23.038662 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.038645    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810"} err="failed to get container status \"b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810\": rpc error: code = NotFound desc = could not find container \"b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810\": container with ID starting with b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810 not found: ID does not exist"
Sep 18 16:18:23.038700 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.038662    2580 scope.go:117] "RemoveContainer" containerID="9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94"
Sep 18 16:18:23.038844 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.038827    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94"} err="failed to get container status \"9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94\": rpc error: code = NotFound desc = could not find container \"9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94\": container with ID starting with 9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94 not found: ID does not exist"
Sep 18 16:18:23.038844 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.038843    2580 scope.go:117] "RemoveContainer" containerID="aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e"
Sep 18 16:18:23.039031 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.039015    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e"} err="failed to get container status \"aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e\": rpc error: code = NotFound desc = could not find container \"aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e\": container with ID starting with aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e not found: ID does not exist"
Sep 18 16:18:23.039068 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.039030    2580 scope.go:117] "RemoveContainer" containerID="b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf"
Sep 18 16:18:23.039242 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.039224    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf"} err="failed to get container status \"b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf\": rpc error: code = NotFound desc = could not find container \"b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf\": container with ID starting with b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf not found: ID does not exist"
Sep 18 16:18:23.039242 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.039241    2580 scope.go:117] "RemoveContainer" containerID="ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64"
Sep 18 16:18:23.039684 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.039611    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64"} err="failed to get container status \"ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64\": rpc error: code = NotFound desc = could not find container \"ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64\": container with ID starting with ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64 not found: ID does not exist"
Sep 18 16:18:23.039684 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.039637    2580 scope.go:117] "RemoveContainer" containerID="247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849"
Sep 18 16:18:23.039978 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.039953    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849"} err="failed to get container status \"247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849\": rpc error: code = NotFound desc = could not find container \"247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849\": container with ID starting with 247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849 not found: ID does not exist"
Sep 18 16:18:23.039978 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.039978    2580 scope.go:117] "RemoveContainer" containerID="c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a"
Sep 18 16:18:23.040296 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.040266    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a"} err="failed to get container status \"c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a\": rpc error: code = NotFound desc = could not find container \"c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a\": container with ID starting with c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a not found: ID does not exist"
Sep 18 16:18:23.040296 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.040315    2580 scope.go:117] "RemoveContainer" containerID="b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810"
Sep 18 16:18:23.040603 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.040576    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810"} err="failed to get container status \"b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810\": rpc error: code = NotFound desc = could not find container \"b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810\": container with ID starting with b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810 not found: ID does not exist"
Sep 18 16:18:23.040699 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.040604    2580 scope.go:117] "RemoveContainer" containerID="9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94"
Sep 18 16:18:23.040853 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.040835    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94"} err="failed to get container status \"9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94\": rpc error: code = NotFound desc = could not find container \"9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94\": container with ID starting with 9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94 not found: ID does not exist"
Sep 18 16:18:23.040905 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.040854    2580 scope.go:117] "RemoveContainer" containerID="aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e"
Sep 18 16:18:23.041111 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041095    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e"} err="failed to get container status \"aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e\": rpc error: code = NotFound desc = could not find container \"aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e\": container with ID starting with aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e not found: ID does not exist"
Sep 18 16:18:23.041163 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041112    2580 scope.go:117] "RemoveContainer" containerID="b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf"
Sep 18 16:18:23.041280 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041263    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:18:23.041410 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041369    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf"} err="failed to get container status \"b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf\": rpc error: code = NotFound desc = could not find container \"b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf\": container with ID starting with b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf not found: ID does not exist"
Sep 18 16:18:23.041410 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041403    2580 scope.go:117] "RemoveContainer" containerID="ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64"
Sep 18 16:18:23.041597 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041576    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c1bed39-9b02-4f28-aea9-364954226a49" containerName="kube-rbac-proxy-thanos"
Sep 18 16:18:23.041597 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041597    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c1bed39-9b02-4f28-aea9-364954226a49" containerName="kube-rbac-proxy-thanos"
Sep 18 16:18:23.041742 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041612    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c1bed39-9b02-4f28-aea9-364954226a49" containerName="thanos-sidecar"
Sep 18 16:18:23.041742 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041618    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c1bed39-9b02-4f28-aea9-364954226a49" containerName="thanos-sidecar"
Sep 18 16:18:23.041742 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041628    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c1bed39-9b02-4f28-aea9-364954226a49" containerName="prometheus"
Sep 18 16:18:23.041742 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041633    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c1bed39-9b02-4f28-aea9-364954226a49" containerName="prometheus"
Sep 18 16:18:23.041742 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041641    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c1bed39-9b02-4f28-aea9-364954226a49" containerName="init-config-reloader"
Sep 18 16:18:23.041742 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041646    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c1bed39-9b02-4f28-aea9-364954226a49" containerName="init-config-reloader"
Sep 18 16:18:23.041742 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041653    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c1bed39-9b02-4f28-aea9-364954226a49" containerName="kube-rbac-proxy"
Sep 18 16:18:23.041742 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041657    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c1bed39-9b02-4f28-aea9-364954226a49" containerName="kube-rbac-proxy"
Sep 18 16:18:23.041742 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041652    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64"} err="failed to get container status \"ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64\": rpc error: code = NotFound desc = could not find container \"ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64\": container with ID starting with ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64 not found: ID does not exist"
Sep 18 16:18:23.041742 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041668    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c1bed39-9b02-4f28-aea9-364954226a49" containerName="kube-rbac-proxy-web"
Sep 18 16:18:23.041742 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041675    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c1bed39-9b02-4f28-aea9-364954226a49" containerName="kube-rbac-proxy-web"
Sep 18 16:18:23.041742 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041675    2580 scope.go:117] "RemoveContainer" containerID="247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849"
Sep 18 16:18:23.041742 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041681    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c1bed39-9b02-4f28-aea9-364954226a49" containerName="config-reloader"
Sep 18 16:18:23.041742 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041686    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c1bed39-9b02-4f28-aea9-364954226a49" containerName="config-reloader"
Sep 18 16:18:23.041742 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041732    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c1bed39-9b02-4f28-aea9-364954226a49" containerName="kube-rbac-proxy-web"
Sep 18 16:18:23.041742 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041740    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c1bed39-9b02-4f28-aea9-364954226a49" containerName="config-reloader"
Sep 18 16:18:23.041742 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041746    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c1bed39-9b02-4f28-aea9-364954226a49" containerName="thanos-sidecar"
Sep 18 16:18:23.041742 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041752    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c1bed39-9b02-4f28-aea9-364954226a49" containerName="kube-rbac-proxy-thanos"
Sep 18 16:18:23.042395 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041758    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c1bed39-9b02-4f28-aea9-364954226a49" containerName="kube-rbac-proxy"
Sep 18 16:18:23.042395 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041764    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c1bed39-9b02-4f28-aea9-364954226a49" containerName="prometheus"
Sep 18 16:18:23.042395 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041919    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849"} err="failed to get container status \"247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849\": rpc error: code = NotFound desc = could not find container \"247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849\": container with ID starting with 247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849 not found: ID does not exist"
Sep 18 16:18:23.042395 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.041940    2580 scope.go:117] "RemoveContainer" containerID="c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a"
Sep 18 16:18:23.042395 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.042181    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a"} err="failed to get container status \"c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a\": rpc error: code = NotFound desc = could not find container \"c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a\": container with ID starting with c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a not found: ID does not exist"
Sep 18 16:18:23.042395 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.042201    2580 scope.go:117] "RemoveContainer" containerID="b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810"
Sep 18 16:18:23.042612 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.042450    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810"} err="failed to get container status \"b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810\": rpc error: code = NotFound desc = could not find container \"b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810\": container with ID starting with b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810 not found: ID does not exist"
Sep 18 16:18:23.042612 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.042467    2580 scope.go:117] "RemoveContainer" containerID="9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94"
Sep 18 16:18:23.042719 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.042698    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94"} err="failed to get container status \"9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94\": rpc error: code = NotFound desc = could not find container \"9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94\": container with ID starting with 9e36c030f536afafa959724fcbf90390576d90f0768a96ac7ca9ad36852d4b94 not found: ID does not exist"
Sep 18 16:18:23.042773 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.042722    2580 scope.go:117] "RemoveContainer" containerID="aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e"
Sep 18 16:18:23.042946 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.042929    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e"} err="failed to get container status \"aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e\": rpc error: code = NotFound desc = could not find container \"aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e\": container with ID starting with aa34bae29b2f34e08022be5655a90bb5c14cd714a9d5cf7fe64d6fd89d5baf9e not found: ID does not exist"
Sep 18 16:18:23.042990 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.042947    2580 scope.go:117] "RemoveContainer" containerID="b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf"
Sep 18 16:18:23.043190 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.043171    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf"} err="failed to get container status \"b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf\": rpc error: code = NotFound desc = could not find container \"b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf\": container with ID starting with b4b3a6b7179dc5d1660d475df3dda1e638d465f734cf1e1c0b9e3c9f5f3c60bf not found: ID does not exist"
Sep 18 16:18:23.043232 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.043198    2580 scope.go:117] "RemoveContainer" containerID="ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64"
Sep 18 16:18:23.043438 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.043420    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64"} err="failed to get container status \"ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64\": rpc error: code = NotFound desc = could not find container \"ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64\": container with ID starting with ff61dbd497a42d635109892ebb063624364cfdb9be497d82b58ae64b6a458f64 not found: ID does not exist"
Sep 18 16:18:23.043502 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.043438    2580 scope.go:117] "RemoveContainer" containerID="247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849"
Sep 18 16:18:23.043622 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.043606    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849"} err="failed to get container status \"247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849\": rpc error: code = NotFound desc = could not find container \"247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849\": container with ID starting with 247b224364ef7d47afe08a61845cde90c99ab1bd63f497fb3ded70c375fb0849 not found: ID does not exist"
Sep 18 16:18:23.043690 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.043623    2580 scope.go:117] "RemoveContainer" containerID="c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a"
Sep 18 16:18:23.043845 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.043827    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a"} err="failed to get container status \"c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a\": rpc error: code = NotFound desc = could not find container \"c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a\": container with ID starting with c02261f3aa65371711c0b1f386e6c55a3492f77afdd742e508e4df40e93ce35a not found: ID does not exist"
Sep 18 16:18:23.043952 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.043846    2580 scope.go:117] "RemoveContainer" containerID="b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810"
Sep 18 16:18:23.044096 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.044073    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810"} err="failed to get container status \"b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810\": rpc error: code = NotFound desc = could not find container \"b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810\": container with ID starting with b03ca8deafa5e1a87625a13da131172a618a97addf434b5eb4f747da3a763810 not found: ID does not exist"
Sep 18 16:18:23.045967 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.045950    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.051288 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.048628    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 16:18:23.051288 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.048843    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 16:18:23.051288 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.048963    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 16:18:23.051288 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.049113    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 16:18:23.051288 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.049507    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 16:18:23.051288 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.049580    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 16:18:23.051288 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.049823    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6s38c4foki2as\""
Sep 18 16:18:23.051288 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.050032    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 16:18:23.051288 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.050207    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 16:18:23.051288 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.050404    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 16:18:23.051288 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.050494    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 16:18:23.052159 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.051440    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-x8tgt\""
Sep 18 16:18:23.052159 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.051571    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 16:18:23.052159 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.051934    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 16:18:23.052799 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.052775    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 16:18:23.054646 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.054628    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 16:18:23.063748 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.063723    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:18:23.114808 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.114782    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4f639657-04f0-4cae-9299-a44af3495a3b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.114917 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.114826    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4f639657-04f0-4cae-9299-a44af3495a3b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.114917 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.114849    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4f639657-04f0-4cae-9299-a44af3495a3b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.114917 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.114865    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f639657-04f0-4cae-9299-a44af3495a3b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.114917 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.114880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f639657-04f0-4cae-9299-a44af3495a3b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.114917 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.114894    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4f639657-04f0-4cae-9299-a44af3495a3b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.114917 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.114908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4f639657-04f0-4cae-9299-a44af3495a3b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.115104 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.114924    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4f639657-04f0-4cae-9299-a44af3495a3b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.115104 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.114943    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4f639657-04f0-4cae-9299-a44af3495a3b-config\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.115104 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.114978    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4f639657-04f0-4cae-9299-a44af3495a3b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.115104 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.114993    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4f639657-04f0-4cae-9299-a44af3495a3b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.115104 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.115032    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qnng\" (UniqueName: \"kubernetes.io/projected/4f639657-04f0-4cae-9299-a44af3495a3b-kube-api-access-8qnng\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.115104 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.115064    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f639657-04f0-4cae-9299-a44af3495a3b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.115104 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.115082    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4f639657-04f0-4cae-9299-a44af3495a3b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.115104 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.115100    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4f639657-04f0-4cae-9299-a44af3495a3b-web-config\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.115351 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.115122    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4f639657-04f0-4cae-9299-a44af3495a3b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.115351 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.115143    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4f639657-04f0-4cae-9299-a44af3495a3b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.115351 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.115174    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f639657-04f0-4cae-9299-a44af3495a3b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.115351 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.115219    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4f639657-04f0-4cae-9299-a44af3495a3b-config-out\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.115351 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.115243    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4f639657-04f0-4cae-9299-a44af3495a3b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.216076 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.216035    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4f639657-04f0-4cae-9299-a44af3495a3b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.216076 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.216081    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4f639657-04f0-4cae-9299-a44af3495a3b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.216340 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.216099    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f639657-04f0-4cae-9299-a44af3495a3b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.216340 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.216115    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4f639657-04f0-4cae-9299-a44af3495a3b-config-out\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.216340 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.216133    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4f639657-04f0-4cae-9299-a44af3495a3b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.216340 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.216183    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4f639657-04f0-4cae-9299-a44af3495a3b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.216340 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.216206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4f639657-04f0-4cae-9299-a44af3495a3b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.216340 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.216233    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4f639657-04f0-4cae-9299-a44af3495a3b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.216340 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.216268    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f639657-04f0-4cae-9299-a44af3495a3b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.216340 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.216295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f639657-04f0-4cae-9299-a44af3495a3b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.216726 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.216349    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4f639657-04f0-4cae-9299-a44af3495a3b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.216726 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.216374    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4f639657-04f0-4cae-9299-a44af3495a3b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.216726 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.216426    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4f639657-04f0-4cae-9299-a44af3495a3b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.216726 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.216459    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4f639657-04f0-4cae-9299-a44af3495a3b-config\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.216726 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.216508    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4f639657-04f0-4cae-9299-a44af3495a3b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.216726 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.216502    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4f639657-04f0-4cae-9299-a44af3495a3b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.216726 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.216551    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4f639657-04f0-4cae-9299-a44af3495a3b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.216726 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.216575    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qnng\" (UniqueName: \"kubernetes.io/projected/4f639657-04f0-4cae-9299-a44af3495a3b-kube-api-access-8qnng\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.216726 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.216611    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f639657-04f0-4cae-9299-a44af3495a3b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.216726 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.216636    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4f639657-04f0-4cae-9299-a44af3495a3b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.216726 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.216670    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4f639657-04f0-4cae-9299-a44af3495a3b-web-config\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.217231 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.216804    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4f639657-04f0-4cae-9299-a44af3495a3b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.217231 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.216909    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f639657-04f0-4cae-9299-a44af3495a3b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.219271 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.219243    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4f639657-04f0-4cae-9299-a44af3495a3b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.219371 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.219340    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4f639657-04f0-4cae-9299-a44af3495a3b-web-config\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.219427 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.219381    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4f639657-04f0-4cae-9299-a44af3495a3b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.219462 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.219426    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4f639657-04f0-4cae-9299-a44af3495a3b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.219612 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.219594    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4f639657-04f0-4cae-9299-a44af3495a3b-config-out\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.219675 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.219649    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4f639657-04f0-4cae-9299-a44af3495a3b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.219970 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.219878    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4f639657-04f0-4cae-9299-a44af3495a3b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.220055 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.220007    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f639657-04f0-4cae-9299-a44af3495a3b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.220128 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.220102    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f639657-04f0-4cae-9299-a44af3495a3b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.220195 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.220157    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4f639657-04f0-4cae-9299-a44af3495a3b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.220408 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.220381    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4f639657-04f0-4cae-9299-a44af3495a3b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.220861 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.220839    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f639657-04f0-4cae-9299-a44af3495a3b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.221608 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.221587    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4f639657-04f0-4cae-9299-a44af3495a3b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.222039 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.222019    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4f639657-04f0-4cae-9299-a44af3495a3b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.222188 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.222170    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4f639657-04f0-4cae-9299-a44af3495a3b-config\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.222381 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.222364    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4f639657-04f0-4cae-9299-a44af3495a3b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.228353 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.228334    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qnng\" (UniqueName: \"kubernetes.io/projected/4f639657-04f0-4cae-9299-a44af3495a3b-kube-api-access-8qnng\") pod \"prometheus-k8s-0\" (UID: \"4f639657-04f0-4cae-9299-a44af3495a3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.356406 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.356346    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:23.635822 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.635799    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:18:23.638043 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:18:23.638003    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f639657_04f0_4cae_9299_a44af3495a3b.slice/crio-2f5494edea8cb45db5a032ace58ecad1e4cd20668bf88f5517a2004bf6a33386 WatchSource:0}: Error finding container 2f5494edea8cb45db5a032ace58ecad1e4cd20668bf88f5517a2004bf6a33386: Status 404 returned error can't find the container with id 2f5494edea8cb45db5a032ace58ecad1e4cd20668bf88f5517a2004bf6a33386
Sep 18 16:18:23.992197 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.992153    2580 generic.go:356] "Generic (PLEG): container finished" podID="4f639657-04f0-4cae-9299-a44af3495a3b" containerID="0375f2778d3f6f64c0c3ccdad8fd46b5fbe48c6a387deb5e125d8b76f0bdfe8b" exitCode=0
Sep 18 16:18:23.992637 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.992329    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4f639657-04f0-4cae-9299-a44af3495a3b","Type":"ContainerDied","Data":"0375f2778d3f6f64c0c3ccdad8fd46b5fbe48c6a387deb5e125d8b76f0bdfe8b"}
Sep 18 16:18:23.992637 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:23.992443    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4f639657-04f0-4cae-9299-a44af3495a3b","Type":"ContainerStarted","Data":"2f5494edea8cb45db5a032ace58ecad1e4cd20668bf88f5517a2004bf6a33386"}
Sep 18 16:18:24.124124 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:24.124096    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c1bed39-9b02-4f28-aea9-364954226a49" path="/var/lib/kubelet/pods/8c1bed39-9b02-4f28-aea9-364954226a49/volumes"
Sep 18 16:18:25.005322 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:25.005273    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4f639657-04f0-4cae-9299-a44af3495a3b","Type":"ContainerStarted","Data":"fcb45021a4a798658fe723ac1468f2751677e70d5d4902ae088ca45aff113cbb"}
Sep 18 16:18:25.005758 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:25.005371    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4f639657-04f0-4cae-9299-a44af3495a3b","Type":"ContainerStarted","Data":"a2c8024f8812c80a2d6bc518cdd59854be719a53c6275c617d640abf46f0dd73"}
Sep 18 16:18:25.005758 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:25.005421    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4f639657-04f0-4cae-9299-a44af3495a3b","Type":"ContainerStarted","Data":"1243f743455daecf7324af86a6db37453beb03151b9b9c045227a953afa03450"}
Sep 18 16:18:25.005758 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:25.005464    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4f639657-04f0-4cae-9299-a44af3495a3b","Type":"ContainerStarted","Data":"38e1611ec4de48d760b150b5f5ed31d4be815b08c8d1fcb1686760778a3a403c"}
Sep 18 16:18:25.005758 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:25.005506    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4f639657-04f0-4cae-9299-a44af3495a3b","Type":"ContainerStarted","Data":"c6b29517b496971390721a0f03ac05be2ea05450586febe8ad74e3d678561068"}
Sep 18 16:18:25.005758 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:25.005546    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4f639657-04f0-4cae-9299-a44af3495a3b","Type":"ContainerStarted","Data":"4ee102c5a43ff7593f87983b7ff3010cbbd18f795795de72523a8dec1e91fb03"}
Sep 18 16:18:25.042557 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:25.042514    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.023561225 podStartE2EDuration="2.042498625s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.9374ms" isStatefulPod=false podCreationTimestamp="2026-09-18 16:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:18:25.040505548 +0000 UTC m=+151.128821213" watchObservedRunningTime="2026-09-18 16:18:25.042498625 +0000 UTC m=+151.130814264"
Sep 18 16:18:26.612567 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:26.612533    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:18:26.713541 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:26.713512    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:18:28.357192 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:28.357153    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:18:37.901411 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:37.901333    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:18:38.002273 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:38.002244    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:18:38.065703 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:38.065680    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:18:54.956704 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:18:54.956671    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:19:23.357279 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:19:23.357243    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:19:23.376883 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:19:23.372963    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:19:24.241556 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:19:24.241526    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:19:32.225190 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:19:32.225161    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:19:32.324656 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:19:32.324622    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:19:44.230016 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:19:44.229975    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:19:44.332528 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:19:44.332495    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:19:47.256737 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:19:47.256698    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:19:47.330149 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:19:47.330117    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:19:57.732894 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:19:57.732858    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:19:57.833405 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:19:57.833366    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:19:58.181729 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:19:58.181681    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:19:58.287581 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:19:58.287548    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:19:58.344404 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:19:58.344376    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:00.027738 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:20:00.027705    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:00.084425 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:20:00.084396    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:05.282422 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:20:05.282391    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:15.909501 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:20:15.909418    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:55.963458 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:20:55.963435    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 16:21:04.527183 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:21:04.527145    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:06.450968 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:21:06.450922    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:39.610719 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:21:39.610633    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:39.708613 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:21:39.708581    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:39.768885 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:21:39.768857    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:42.627979 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:21:42.627940    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:42.722898 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:21:42.722859    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:42.794148 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:21:42.794123    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:02.456112 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.456081    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c6c96cb85-rc9lz"]
Sep 18 16:22:02.458470 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.458452    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6c96cb85-rc9lz"
Sep 18 16:22:02.461437 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.461408    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 16:22:02.461551 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.461512    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 16:22:02.461676 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.461654    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 16:22:02.462497 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.462469    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 16:22:02.462612 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.462495    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 16:22:02.462612 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.462502    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 16:22:02.462612 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.462586    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4nj9v\""
Sep 18 16:22:02.462961 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.462948    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 16:22:02.467449 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.467432    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 16:22:02.471898 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.471877    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c6c96cb85-rc9lz"]
Sep 18 16:22:02.518806 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.518770    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:02.583760 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.583730    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:02.584119 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.584100    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbnd6\" (UniqueName: \"kubernetes.io/projected/04664e65-e692-46ad-8c1a-22e7c399965f-kube-api-access-mbnd6\") pod \"console-7c6c96cb85-rc9lz\" (UID: \"04664e65-e692-46ad-8c1a-22e7c399965f\") " pod="openshift-console/console-7c6c96cb85-rc9lz"
Sep 18 16:22:02.584173 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.584140    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04664e65-e692-46ad-8c1a-22e7c399965f-console-oauth-config\") pod \"console-7c6c96cb85-rc9lz\" (UID: \"04664e65-e692-46ad-8c1a-22e7c399965f\") " pod="openshift-console/console-7c6c96cb85-rc9lz"
Sep 18 16:22:02.584173 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.584164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04664e65-e692-46ad-8c1a-22e7c399965f-oauth-serving-cert\") pod \"console-7c6c96cb85-rc9lz\" (UID: \"04664e65-e692-46ad-8c1a-22e7c399965f\") " pod="openshift-console/console-7c6c96cb85-rc9lz"
Sep 18 16:22:02.584242 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.584226    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04664e65-e692-46ad-8c1a-22e7c399965f-console-serving-cert\") pod \"console-7c6c96cb85-rc9lz\" (UID: \"04664e65-e692-46ad-8c1a-22e7c399965f\") " pod="openshift-console/console-7c6c96cb85-rc9lz"
Sep 18 16:22:02.584276 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.584252    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04664e65-e692-46ad-8c1a-22e7c399965f-console-config\") pod \"console-7c6c96cb85-rc9lz\" (UID: \"04664e65-e692-46ad-8c1a-22e7c399965f\") " pod="openshift-console/console-7c6c96cb85-rc9lz"
Sep 18 16:22:02.584322 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.584286    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04664e65-e692-46ad-8c1a-22e7c399965f-service-ca\") pod \"console-7c6c96cb85-rc9lz\" (UID: \"04664e65-e692-46ad-8c1a-22e7c399965f\") " pod="openshift-console/console-7c6c96cb85-rc9lz"
Sep 18 16:22:02.584359 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.584348    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04664e65-e692-46ad-8c1a-22e7c399965f-trusted-ca-bundle\") pod \"console-7c6c96cb85-rc9lz\" (UID: \"04664e65-e692-46ad-8c1a-22e7c399965f\") " pod="openshift-console/console-7c6c96cb85-rc9lz"
Sep 18 16:22:02.685268 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.685238    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04664e65-e692-46ad-8c1a-22e7c399965f-console-config\") pod \"console-7c6c96cb85-rc9lz\" (UID: \"04664e65-e692-46ad-8c1a-22e7c399965f\") " pod="openshift-console/console-7c6c96cb85-rc9lz"
Sep 18 16:22:02.685473 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.685276    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04664e65-e692-46ad-8c1a-22e7c399965f-service-ca\") pod \"console-7c6c96cb85-rc9lz\" (UID: \"04664e65-e692-46ad-8c1a-22e7c399965f\") " pod="openshift-console/console-7c6c96cb85-rc9lz"
Sep 18 16:22:02.685473 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.685318    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04664e65-e692-46ad-8c1a-22e7c399965f-trusted-ca-bundle\") pod \"console-7c6c96cb85-rc9lz\" (UID: \"04664e65-e692-46ad-8c1a-22e7c399965f\") " pod="openshift-console/console-7c6c96cb85-rc9lz"
Sep 18 16:22:02.685473 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.685379    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbnd6\" (UniqueName: \"kubernetes.io/projected/04664e65-e692-46ad-8c1a-22e7c399965f-kube-api-access-mbnd6\") pod \"console-7c6c96cb85-rc9lz\" (UID: \"04664e65-e692-46ad-8c1a-22e7c399965f\") " pod="openshift-console/console-7c6c96cb85-rc9lz"
Sep 18 16:22:02.685473 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.685422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04664e65-e692-46ad-8c1a-22e7c399965f-console-oauth-config\") pod \"console-7c6c96cb85-rc9lz\" (UID: \"04664e65-e692-46ad-8c1a-22e7c399965f\") " pod="openshift-console/console-7c6c96cb85-rc9lz"
Sep 18 16:22:02.685686 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.685580    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04664e65-e692-46ad-8c1a-22e7c399965f-oauth-serving-cert\") pod \"console-7c6c96cb85-rc9lz\" (UID: \"04664e65-e692-46ad-8c1a-22e7c399965f\") " pod="openshift-console/console-7c6c96cb85-rc9lz"
Sep 18 16:22:02.685686 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.685619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04664e65-e692-46ad-8c1a-22e7c399965f-console-serving-cert\") pod \"console-7c6c96cb85-rc9lz\" (UID: \"04664e65-e692-46ad-8c1a-22e7c399965f\") " pod="openshift-console/console-7c6c96cb85-rc9lz"
Sep 18 16:22:02.686065 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.686025    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04664e65-e692-46ad-8c1a-22e7c399965f-console-config\") pod \"console-7c6c96cb85-rc9lz\" (UID: \"04664e65-e692-46ad-8c1a-22e7c399965f\") " pod="openshift-console/console-7c6c96cb85-rc9lz"
Sep 18 16:22:02.686174 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.686088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04664e65-e692-46ad-8c1a-22e7c399965f-service-ca\") pod \"console-7c6c96cb85-rc9lz\" (UID: \"04664e65-e692-46ad-8c1a-22e7c399965f\") " pod="openshift-console/console-7c6c96cb85-rc9lz"
Sep 18 16:22:02.686259 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.686238    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04664e65-e692-46ad-8c1a-22e7c399965f-oauth-serving-cert\") pod \"console-7c6c96cb85-rc9lz\" (UID: \"04664e65-e692-46ad-8c1a-22e7c399965f\") " pod="openshift-console/console-7c6c96cb85-rc9lz"
Sep 18 16:22:02.686359 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.686342    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04664e65-e692-46ad-8c1a-22e7c399965f-trusted-ca-bundle\") pod \"console-7c6c96cb85-rc9lz\" (UID: \"04664e65-e692-46ad-8c1a-22e7c399965f\") " pod="openshift-console/console-7c6c96cb85-rc9lz"
Sep 18 16:22:02.687829 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.687807    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04664e65-e692-46ad-8c1a-22e7c399965f-console-oauth-config\") pod \"console-7c6c96cb85-rc9lz\" (UID: \"04664e65-e692-46ad-8c1a-22e7c399965f\") " pod="openshift-console/console-7c6c96cb85-rc9lz"
Sep 18 16:22:02.687904 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.687855    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04664e65-e692-46ad-8c1a-22e7c399965f-console-serving-cert\") pod \"console-7c6c96cb85-rc9lz\" (UID: \"04664e65-e692-46ad-8c1a-22e7c399965f\") " pod="openshift-console/console-7c6c96cb85-rc9lz"
Sep 18 16:22:02.693946 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.693921    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbnd6\" (UniqueName: \"kubernetes.io/projected/04664e65-e692-46ad-8c1a-22e7c399965f-kube-api-access-mbnd6\") pod \"console-7c6c96cb85-rc9lz\" (UID: \"04664e65-e692-46ad-8c1a-22e7c399965f\") " pod="openshift-console/console-7c6c96cb85-rc9lz"
Sep 18 16:22:02.768369 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:02.768285    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6c96cb85-rc9lz"
Sep 18 16:22:03.042950 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:03.042903    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c6c96cb85-rc9lz"]
Sep 18 16:22:03.045107 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:22:03.045081    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04664e65_e692_46ad_8c1a_22e7c399965f.slice/crio-ccb68a93f9ceaea2015cecd3bc916915828a2f8d1e0e1387c476edafd847b0ad WatchSource:0}: Error finding container ccb68a93f9ceaea2015cecd3bc916915828a2f8d1e0e1387c476edafd847b0ad: Status 404 returned error can't find the container with id ccb68a93f9ceaea2015cecd3bc916915828a2f8d1e0e1387c476edafd847b0ad
Sep 18 16:22:03.834482 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:03.834446    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6c96cb85-rc9lz" event={"ID":"04664e65-e692-46ad-8c1a-22e7c399965f","Type":"ContainerStarted","Data":"dafeaf4ce04c3f380edf75e8a4e4ef9c33b1fd9d7262459f4be1cc96b493d083"}
Sep 18 16:22:03.834867 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:03.834503    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6c96cb85-rc9lz" event={"ID":"04664e65-e692-46ad-8c1a-22e7c399965f","Type":"ContainerStarted","Data":"ccb68a93f9ceaea2015cecd3bc916915828a2f8d1e0e1387c476edafd847b0ad"}
Sep 18 16:22:03.855589 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:03.855540    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c6c96cb85-rc9lz" podStartSLOduration=1.855524849 podStartE2EDuration="1.855524849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:22:03.854394018 +0000 UTC m=+369.942709663" watchObservedRunningTime="2026-09-18 16:22:03.855524849 +0000 UTC m=+369.943840490"
Sep 18 16:22:07.400718 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:07.400686    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:07.491811 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:07.491783    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:12.768687 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:12.768645    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c6c96cb85-rc9lz"
Sep 18 16:22:12.769133 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:12.768721    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c6c96cb85-rc9lz"
Sep 18 16:22:12.773288 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:12.773265    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c6c96cb85-rc9lz"
Sep 18 16:22:12.871567 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:12.871537    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c6c96cb85-rc9lz"
Sep 18 16:22:13.044176 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:13.044084    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:13.143254 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:13.143222    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:13.201663 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:13.201634    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:14.364945 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:14.364908    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:14.644666 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:14.644591    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:14.860981 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:14.860950    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:14.929615 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:14.929588    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:15.092737 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:15.092708    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:15.348660 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:15.348542    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:15.373067 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:15.373036    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-szpl5"]
Sep 18 16:22:15.377345 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:15.377323    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-szpl5"
Sep 18 16:22:15.380161 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:15.380138    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 18 16:22:15.380265 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:15.380169    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-jrfbn\""
Sep 18 16:22:15.380986 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:15.380968    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 18 16:22:15.381047 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:15.380976    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 18 16:22:15.381047 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:15.381006    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 18 16:22:15.388879 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:15.388858    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-szpl5"]
Sep 18 16:22:15.449675 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:15.449645    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:15.493171 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:15.493138    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6f8540df-b1e4-4bcc-8aec-1abf9ee69e90-certificates\") pod \"keda-admission-6c4d864b86-szpl5\" (UID: \"6f8540df-b1e4-4bcc-8aec-1abf9ee69e90\") " pod="openshift-keda/keda-admission-6c4d864b86-szpl5"
Sep 18 16:22:15.493351 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:15.493271    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db2x7\" (UniqueName: \"kubernetes.io/projected/6f8540df-b1e4-4bcc-8aec-1abf9ee69e90-kube-api-access-db2x7\") pod \"keda-admission-6c4d864b86-szpl5\" (UID: \"6f8540df-b1e4-4bcc-8aec-1abf9ee69e90\") " pod="openshift-keda/keda-admission-6c4d864b86-szpl5"
Sep 18 16:22:15.549413 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:15.549368    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:15.594033 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:15.594001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db2x7\" (UniqueName: \"kubernetes.io/projected/6f8540df-b1e4-4bcc-8aec-1abf9ee69e90-kube-api-access-db2x7\") pod \"keda-admission-6c4d864b86-szpl5\" (UID: \"6f8540df-b1e4-4bcc-8aec-1abf9ee69e90\") " pod="openshift-keda/keda-admission-6c4d864b86-szpl5"
Sep 18 16:22:15.594198 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:15.594039    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6f8540df-b1e4-4bcc-8aec-1abf9ee69e90-certificates\") pod \"keda-admission-6c4d864b86-szpl5\" (UID: \"6f8540df-b1e4-4bcc-8aec-1abf9ee69e90\") " pod="openshift-keda/keda-admission-6c4d864b86-szpl5"
Sep 18 16:22:15.596488 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:15.596466    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6f8540df-b1e4-4bcc-8aec-1abf9ee69e90-certificates\") pod \"keda-admission-6c4d864b86-szpl5\" (UID: \"6f8540df-b1e4-4bcc-8aec-1abf9ee69e90\") " pod="openshift-keda/keda-admission-6c4d864b86-szpl5"
Sep 18 16:22:15.602858 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:15.602797    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db2x7\" (UniqueName: \"kubernetes.io/projected/6f8540df-b1e4-4bcc-8aec-1abf9ee69e90-kube-api-access-db2x7\") pod \"keda-admission-6c4d864b86-szpl5\" (UID: \"6f8540df-b1e4-4bcc-8aec-1abf9ee69e90\") " pod="openshift-keda/keda-admission-6c4d864b86-szpl5"
Sep 18 16:22:15.688745 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:15.688703    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-szpl5"
Sep 18 16:22:15.749844 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:15.749812    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:15.862555 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:15.862460    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:15.966591 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:15.966568    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-szpl5"]
Sep 18 16:22:15.969122 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:22:15.969095    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f8540df_b1e4_4bcc_8aec_1abf9ee69e90.slice/crio-7b31d62f4ac3558fdc8c652b4609b31db3c1b71f111ced9a477d35cd92d314da WatchSource:0}: Error finding container 7b31d62f4ac3558fdc8c652b4609b31db3c1b71f111ced9a477d35cd92d314da: Status 404 returned error can't find the container with id 7b31d62f4ac3558fdc8c652b4609b31db3c1b71f111ced9a477d35cd92d314da
Sep 18 16:22:15.970452 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:15.970435    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:22:16.049382 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:16.049355    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:16.185850 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:16.185823    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:16.286541 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:16.286491    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:16.885540 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:16.885500    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-szpl5" event={"ID":"6f8540df-b1e4-4bcc-8aec-1abf9ee69e90","Type":"ContainerStarted","Data":"7b31d62f4ac3558fdc8c652b4609b31db3c1b71f111ced9a477d35cd92d314da"}
Sep 18 16:22:18.895329 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:18.895213    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-szpl5" event={"ID":"6f8540df-b1e4-4bcc-8aec-1abf9ee69e90","Type":"ContainerStarted","Data":"97892bbf68ba3dc5c72bb27eec821af919cfa8828b4862ab13391e07bb762047"}
Sep 18 16:22:18.895329 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:18.895274    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-szpl5"
Sep 18 16:22:18.913689 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:18.913647    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-szpl5" podStartSLOduration=1.384762886 podStartE2EDuration="3.913633286s" totalImagesPullingTime="2.5288704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:22:18.912185142 +0000 UTC m=+385.000500774" watchObservedRunningTime="2026-09-18 16:22:18.913633286 +0000 UTC m=+385.001948923"
Sep 18 16:22:39.901679 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:22:39.901649    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-szpl5"
Sep 18 16:23:05.886965 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:23:05.886924    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:23:05.954288 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:23:05.954253    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:23:09.561640 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:23:09.561561    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:23:32.570791 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:23:32.570752    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:23:32.658088 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:23:32.658059    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:23:32.730625 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:23:32.730597    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:23:32.888632 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:23:32.888544    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:23:33.009123 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:23:33.009092    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:23:33.213643 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:23:33.213612    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:23:33.317528 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:23:33.317500    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:23:33.388463 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:23:33.388438    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:24:08.251710 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:08.251667    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:24:08.354320 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:08.354268    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:24:49.271077 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:49.270995    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:24:49.313801 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:49.313771    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-n4vd5"]
Sep 18 16:24:49.316198 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:49.316183    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-n4vd5"
Sep 18 16:24:49.319580 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:49.319557    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 18 16:24:49.319709 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:49.319583    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 16:24:49.319709 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:49.319583    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 18 16:24:49.320691 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:49.320671    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-rthtp\""
Sep 18 16:24:49.320790 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:49.320695    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 16:24:49.329605 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:49.329583    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-n4vd5"]
Sep 18 16:24:49.351904 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:49.351881    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:24:49.407160 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:49.407138    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fae0365-7eda-4270-a2a0-0dbbbbf9dc49-metrics-certs\") pod \"odh-model-controller-7b966d7984-n4vd5\" (UID: \"1fae0365-7eda-4270-a2a0-0dbbbbf9dc49\") " pod="kserve/odh-model-controller-7b966d7984-n4vd5"
Sep 18 16:24:49.407251 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:49.407166    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fae0365-7eda-4270-a2a0-0dbbbbf9dc49-cert\") pod \"odh-model-controller-7b966d7984-n4vd5\" (UID: \"1fae0365-7eda-4270-a2a0-0dbbbbf9dc49\") " pod="kserve/odh-model-controller-7b966d7984-n4vd5"
Sep 18 16:24:49.407251 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:49.407199    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pfdb\" (UniqueName: \"kubernetes.io/projected/1fae0365-7eda-4270-a2a0-0dbbbbf9dc49-kube-api-access-7pfdb\") pod \"odh-model-controller-7b966d7984-n4vd5\" (UID: \"1fae0365-7eda-4270-a2a0-0dbbbbf9dc49\") " pod="kserve/odh-model-controller-7b966d7984-n4vd5"
Sep 18 16:24:49.460121 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:49.460095    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:24:49.508279 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:49.508261    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fae0365-7eda-4270-a2a0-0dbbbbf9dc49-metrics-certs\") pod \"odh-model-controller-7b966d7984-n4vd5\" (UID: \"1fae0365-7eda-4270-a2a0-0dbbbbf9dc49\") " pod="kserve/odh-model-controller-7b966d7984-n4vd5"
Sep 18 16:24:49.508397 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:49.508289    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fae0365-7eda-4270-a2a0-0dbbbbf9dc49-cert\") pod \"odh-model-controller-7b966d7984-n4vd5\" (UID: \"1fae0365-7eda-4270-a2a0-0dbbbbf9dc49\") " pod="kserve/odh-model-controller-7b966d7984-n4vd5"
Sep 18 16:24:49.508397 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:49.508340    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pfdb\" (UniqueName: \"kubernetes.io/projected/1fae0365-7eda-4270-a2a0-0dbbbbf9dc49-kube-api-access-7pfdb\") pod \"odh-model-controller-7b966d7984-n4vd5\" (UID: \"1fae0365-7eda-4270-a2a0-0dbbbbf9dc49\") " pod="kserve/odh-model-controller-7b966d7984-n4vd5"
Sep 18 16:24:49.508471 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:24:49.508452    2580 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 18 16:24:49.508525 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:24:49.508514    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fae0365-7eda-4270-a2a0-0dbbbbf9dc49-cert podName:1fae0365-7eda-4270-a2a0-0dbbbbf9dc49 nodeName:}" failed. No retries permitted until 2026-09-18 16:24:50.008500067 +0000 UTC m=+536.096815686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1fae0365-7eda-4270-a2a0-0dbbbbf9dc49-cert") pod "odh-model-controller-7b966d7984-n4vd5" (UID: "1fae0365-7eda-4270-a2a0-0dbbbbf9dc49") : secret "odh-model-controller-webhook-cert" not found
Sep 18 16:24:49.510589 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:49.510565    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fae0365-7eda-4270-a2a0-0dbbbbf9dc49-metrics-certs\") pod \"odh-model-controller-7b966d7984-n4vd5\" (UID: \"1fae0365-7eda-4270-a2a0-0dbbbbf9dc49\") " pod="kserve/odh-model-controller-7b966d7984-n4vd5"
Sep 18 16:24:49.516096 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:49.516075    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pfdb\" (UniqueName: \"kubernetes.io/projected/1fae0365-7eda-4270-a2a0-0dbbbbf9dc49-kube-api-access-7pfdb\") pod \"odh-model-controller-7b966d7984-n4vd5\" (UID: \"1fae0365-7eda-4270-a2a0-0dbbbbf9dc49\") " pod="kserve/odh-model-controller-7b966d7984-n4vd5"
Sep 18 16:24:49.778901 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:49.778870    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:24:49.887870 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:49.887842    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:24:50.011982 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:50.011956    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fae0365-7eda-4270-a2a0-0dbbbbf9dc49-cert\") pod \"odh-model-controller-7b966d7984-n4vd5\" (UID: \"1fae0365-7eda-4270-a2a0-0dbbbbf9dc49\") " pod="kserve/odh-model-controller-7b966d7984-n4vd5"
Sep 18 16:24:50.014203 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:50.014186    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fae0365-7eda-4270-a2a0-0dbbbbf9dc49-cert\") pod \"odh-model-controller-7b966d7984-n4vd5\" (UID: \"1fae0365-7eda-4270-a2a0-0dbbbbf9dc49\") " pod="kserve/odh-model-controller-7b966d7984-n4vd5"
Sep 18 16:24:50.226131 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:50.226096    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-n4vd5"
Sep 18 16:24:50.299118 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:50.298986    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:24:50.369016 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:50.368981    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:24:50.501709 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:50.499185    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-n4vd5"]
Sep 18 16:24:51.473400 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:51.473326    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-n4vd5" event={"ID":"1fae0365-7eda-4270-a2a0-0dbbbbf9dc49","Type":"ContainerStarted","Data":"d1d5b54fd3ca1553d78e6adb1d79d942a09e2bb9a710300c443fc25466e09e43"}
Sep 18 16:24:53.483938 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:53.483905    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-n4vd5" event={"ID":"1fae0365-7eda-4270-a2a0-0dbbbbf9dc49","Type":"ContainerStarted","Data":"58311e63825e833952c21634b158e9ed48d38d7412549925b1a2d8c5e0f2694c"}
Sep 18 16:24:53.484283 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:53.484122    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-n4vd5"
Sep 18 16:24:53.501926 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:24:53.501881    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-n4vd5" podStartSLOduration=2.059215791 podStartE2EDuration="4.501865442s" totalImagesPullingTime="2.442649651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:24:53.499939155 +0000 UTC m=+539.588254795" watchObservedRunningTime="2026-09-18 16:24:53.501865442 +0000 UTC m=+539.590181081"
Sep 18 16:25:04.490485 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:25:04.490454    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-n4vd5"
Sep 18 16:25:05.332144 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:25:05.332109    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-ft4kd"]
Sep 18 16:25:05.335376 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:25:05.335359    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-ft4kd"
Sep 18 16:25:05.337988 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:25:05.337966    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 18 16:25:05.338130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:25:05.338104    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 18 16:25:05.338130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:25:05.338115    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-dqqdl\""
Sep 18 16:25:05.345254 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:25:05.345232    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-ft4kd"]
Sep 18 16:25:05.436558 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:25:05.436524    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c8d9\" (UniqueName: \"kubernetes.io/projected/a25c8493-10c1-4f75-bee2-fbfde5d9179b-kube-api-access-6c8d9\") pod \"s3-init-ft4kd\" (UID: \"a25c8493-10c1-4f75-bee2-fbfde5d9179b\") " pod="kserve/s3-init-ft4kd"
Sep 18 16:25:05.436717 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:25:05.436584    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a25c8493-10c1-4f75-bee2-fbfde5d9179b-model-cache\") pod \"s3-init-ft4kd\" (UID: \"a25c8493-10c1-4f75-bee2-fbfde5d9179b\") " pod="kserve/s3-init-ft4kd"
Sep 18 16:25:05.537512 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:25:05.537476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a25c8493-10c1-4f75-bee2-fbfde5d9179b-model-cache\") pod \"s3-init-ft4kd\" (UID: \"a25c8493-10c1-4f75-bee2-fbfde5d9179b\") " pod="kserve/s3-init-ft4kd"
Sep 18 16:25:05.537888 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:25:05.537558    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c8d9\" (UniqueName: \"kubernetes.io/projected/a25c8493-10c1-4f75-bee2-fbfde5d9179b-kube-api-access-6c8d9\") pod \"s3-init-ft4kd\" (UID: \"a25c8493-10c1-4f75-bee2-fbfde5d9179b\") " pod="kserve/s3-init-ft4kd"
Sep 18 16:25:05.537888 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:25:05.537800    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a25c8493-10c1-4f75-bee2-fbfde5d9179b-model-cache\") pod \"s3-init-ft4kd\" (UID: \"a25c8493-10c1-4f75-bee2-fbfde5d9179b\") " pod="kserve/s3-init-ft4kd"
Sep 18 16:25:05.545928 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:25:05.545907    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c8d9\" (UniqueName: \"kubernetes.io/projected/a25c8493-10c1-4f75-bee2-fbfde5d9179b-kube-api-access-6c8d9\") pod \"s3-init-ft4kd\" (UID: \"a25c8493-10c1-4f75-bee2-fbfde5d9179b\") " pod="kserve/s3-init-ft4kd"
Sep 18 16:25:05.663091 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:25:05.663004    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-ft4kd"
Sep 18 16:25:05.934070 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:25:05.934045    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-ft4kd"]
Sep 18 16:25:05.936316 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:25:05.936268    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda25c8493_10c1_4f75_bee2_fbfde5d9179b.slice/crio-320a40abd587959720f8dd86a95d41f6cbb4fb8f7d1b0dfe55bbd00f8429ee79 WatchSource:0}: Error finding container 320a40abd587959720f8dd86a95d41f6cbb4fb8f7d1b0dfe55bbd00f8429ee79: Status 404 returned error can't find the container with id 320a40abd587959720f8dd86a95d41f6cbb4fb8f7d1b0dfe55bbd00f8429ee79
Sep 18 16:25:06.536068 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:25:06.536034    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ft4kd" event={"ID":"a25c8493-10c1-4f75-bee2-fbfde5d9179b","Type":"ContainerStarted","Data":"320a40abd587959720f8dd86a95d41f6cbb4fb8f7d1b0dfe55bbd00f8429ee79"}
Sep 18 16:25:10.557147 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:25:10.557111    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ft4kd" event={"ID":"a25c8493-10c1-4f75-bee2-fbfde5d9179b","Type":"ContainerStarted","Data":"9634689994e23a4fbcc7e1715e89d22a86379d8ca73123fba64a591de7798f76"}
Sep 18 16:25:12.569198 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:25:12.569109    2580 generic.go:356] "Generic (PLEG): container finished" podID="a25c8493-10c1-4f75-bee2-fbfde5d9179b" containerID="9634689994e23a4fbcc7e1715e89d22a86379d8ca73123fba64a591de7798f76" exitCode=0
Sep 18 16:25:12.569198 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:25:12.569180    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ft4kd" event={"ID":"a25c8493-10c1-4f75-bee2-fbfde5d9179b","Type":"ContainerDied","Data":"9634689994e23a4fbcc7e1715e89d22a86379d8ca73123fba64a591de7798f76"}
Sep 18 16:25:17.593689 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:25:17.593650    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ft4kd" event={"ID":"a25c8493-10c1-4f75-bee2-fbfde5d9179b","Type":"ContainerStarted","Data":"a24f559d8e18313aa5fb2a82040f7cf3171808767b43c8967909f53f0de12258"}
Sep 18 16:25:17.611318 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:25:17.611261    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-ft4kd" podStartSLOduration=1.959039444 podStartE2EDuration="12.611243105s" totalImagesPullingTime="8.506303773s" totalInitContainerRuntime="2.145899888s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:25:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:25:17.610917009 +0000 UTC m=+563.699232650" watchObservedRunningTime="2026-09-18 16:25:17.611243105 +0000 UTC m=+563.699558746"
Sep 18 16:26:21.839289 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:21.839256    2580 generic.go:356] "Generic (PLEG): container finished" podID="a25c8493-10c1-4f75-bee2-fbfde5d9179b" containerID="a24f559d8e18313aa5fb2a82040f7cf3171808767b43c8967909f53f0de12258" exitCode=0
Sep 18 16:26:21.839673 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:21.839330    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ft4kd" event={"ID":"a25c8493-10c1-4f75-bee2-fbfde5d9179b","Type":"ContainerDied","Data":"a24f559d8e18313aa5fb2a82040f7cf3171808767b43c8967909f53f0de12258"}
Sep 18 16:26:22.961429 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:22.961405    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-ft4kd"
Sep 18 16:26:23.081390 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:23.081357    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a25c8493-10c1-4f75-bee2-fbfde5d9179b-model-cache\") pod \"a25c8493-10c1-4f75-bee2-fbfde5d9179b\" (UID: \"a25c8493-10c1-4f75-bee2-fbfde5d9179b\") "
Sep 18 16:26:23.081583 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:23.081422    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6c8d9\" (UniqueName: \"kubernetes.io/projected/a25c8493-10c1-4f75-bee2-fbfde5d9179b-kube-api-access-6c8d9\") pod \"a25c8493-10c1-4f75-bee2-fbfde5d9179b\" (UID: \"a25c8493-10c1-4f75-bee2-fbfde5d9179b\") "
Sep 18 16:26:23.083593 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:23.083563    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25c8493-10c1-4f75-bee2-fbfde5d9179b-kube-api-access-6c8d9" (OuterVolumeSpecName: "kube-api-access-6c8d9") pod "a25c8493-10c1-4f75-bee2-fbfde5d9179b" (UID: "a25c8493-10c1-4f75-bee2-fbfde5d9179b"). InnerVolumeSpecName "kube-api-access-6c8d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:26:23.103437 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:23.103377    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25c8493-10c1-4f75-bee2-fbfde5d9179b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a25c8493-10c1-4f75-bee2-fbfde5d9179b" (UID: "a25c8493-10c1-4f75-bee2-fbfde5d9179b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:26:23.182280 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:23.182251    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a25c8493-10c1-4f75-bee2-fbfde5d9179b-model-cache\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:26:23.182418 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:23.182284    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c8d9\" (UniqueName: \"kubernetes.io/projected/a25c8493-10c1-4f75-bee2-fbfde5d9179b-kube-api-access-6c8d9\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:26:23.848898 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:23.848861    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ft4kd" event={"ID":"a25c8493-10c1-4f75-bee2-fbfde5d9179b","Type":"ContainerDied","Data":"320a40abd587959720f8dd86a95d41f6cbb4fb8f7d1b0dfe55bbd00f8429ee79"}
Sep 18 16:26:23.848898 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:23.848898    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="320a40abd587959720f8dd86a95d41f6cbb4fb8f7d1b0dfe55bbd00f8429ee79"
Sep 18 16:26:23.848898 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:23.848903    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-ft4kd"
Sep 18 16:26:24.235349 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:24.235318    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:25.522590 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:25.522537    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:29.065480 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:29.065447    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:29.165151 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:29.165121    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:37.134756 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.134726    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:37.246319 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.246273    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"]
Sep 18 16:26:37.246654 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.246640    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a25c8493-10c1-4f75-bee2-fbfde5d9179b" containerName="s3-init"
Sep 18 16:26:37.246698 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.246655    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a25c8493-10c1-4f75-bee2-fbfde5d9179b" containerName="s3-init"
Sep 18 16:26:37.246698 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.246666    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a25c8493-10c1-4f75-bee2-fbfde5d9179b" containerName="download-hf-model"
Sep 18 16:26:37.246698 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.246672    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a25c8493-10c1-4f75-bee2-fbfde5d9179b" containerName="download-hf-model"
Sep 18 16:26:37.246795 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.246730    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a25c8493-10c1-4f75-bee2-fbfde5d9179b" containerName="s3-init"
Sep 18 16:26:37.250329 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.250310    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"
Sep 18 16:26:37.253853 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.253831    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 16:26:37.253954 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.253910    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
Sep 18 16:26:37.253954 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.253946    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z57rx\""
Sep 18 16:26:37.254410 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.254392    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 16:26:37.254512 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.254412    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 16:26:37.254713 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.254694    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
Sep 18 16:26:37.254819 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.254697    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 16:26:37.263449 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.263428    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"]
Sep 18 16:26:37.280739 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.280707    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:37.347093 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.347060    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6"]
Sep 18 16:26:37.350481 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.350459    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6"
Sep 18 16:26:37.353070 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.353047    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-191f9-predictor-serving-cert\""
Sep 18 16:26:37.353070 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.353062    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-191f9-kube-rbac-proxy-sar-config\""
Sep 18 16:26:37.353392 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.353374    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:37.361929 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.361909    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6"]
Sep 18 16:26:37.405344 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.405246    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3f067c6-43a9-48de-9acb-7c1fcc44389e-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl\" (UID: \"a3f067c6-43a9-48de-9acb-7c1fcc44389e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"
Sep 18 16:26:37.405344 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.405316    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3f067c6-43a9-48de-9acb-7c1fcc44389e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl\" (UID: \"a3f067c6-43a9-48de-9acb-7c1fcc44389e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"
Sep 18 16:26:37.405571 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.405347    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z67hm\" (UniqueName: \"kubernetes.io/projected/a3f067c6-43a9-48de-9acb-7c1fcc44389e-kube-api-access-z67hm\") pod \"isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl\" (UID: \"a3f067c6-43a9-48de-9acb-7c1fcc44389e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"
Sep 18 16:26:37.405571 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.405371    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3f067c6-43a9-48de-9acb-7c1fcc44389e-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl\" (UID: \"a3f067c6-43a9-48de-9acb-7c1fcc44389e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"
Sep 18 16:26:37.405571 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.405396    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3f067c6-43a9-48de-9acb-7c1fcc44389e-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl\" (UID: \"a3f067c6-43a9-48de-9acb-7c1fcc44389e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"
Sep 18 16:26:37.434882 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.434848    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:37.506560 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.506519    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z67hm\" (UniqueName: \"kubernetes.io/projected/a3f067c6-43a9-48de-9acb-7c1fcc44389e-kube-api-access-z67hm\") pod \"isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl\" (UID: \"a3f067c6-43a9-48de-9acb-7c1fcc44389e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"
Sep 18 16:26:37.506741 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.506574    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3f067c6-43a9-48de-9acb-7c1fcc44389e-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl\" (UID: \"a3f067c6-43a9-48de-9acb-7c1fcc44389e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"
Sep 18 16:26:37.506741 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.506600    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3f067c6-43a9-48de-9acb-7c1fcc44389e-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl\" (UID: \"a3f067c6-43a9-48de-9acb-7c1fcc44389e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"
Sep 18 16:26:37.506741 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.506695    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3f067c6-43a9-48de-9acb-7c1fcc44389e-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl\" (UID: \"a3f067c6-43a9-48de-9acb-7c1fcc44389e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"
Sep 18 16:26:37.506901 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.506751    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3f067c6-43a9-48de-9acb-7c1fcc44389e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl\" (UID: \"a3f067c6-43a9-48de-9acb-7c1fcc44389e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"
Sep 18 16:26:37.506901 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.506845    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cdefa0f-50a9-40a3-8571-c90a98531b0e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-191f9-predictor-6f8b8449b5-shdc6\" (UID: \"3cdefa0f-50a9-40a3-8571-c90a98531b0e\") " pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6"
Sep 18 16:26:37.506901 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.506884    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwf5s\" (UniqueName: \"kubernetes.io/projected/3cdefa0f-50a9-40a3-8571-c90a98531b0e-kube-api-access-xwf5s\") pod \"success-200-isvc-191f9-predictor-6f8b8449b5-shdc6\" (UID: \"3cdefa0f-50a9-40a3-8571-c90a98531b0e\") " pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6"
Sep 18 16:26:37.507056 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.506916    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cdefa0f-50a9-40a3-8571-c90a98531b0e-proxy-tls\") pod \"success-200-isvc-191f9-predictor-6f8b8449b5-shdc6\" (UID: \"3cdefa0f-50a9-40a3-8571-c90a98531b0e\") " pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6"
Sep 18 16:26:37.507152 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.507130    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3f067c6-43a9-48de-9acb-7c1fcc44389e-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl\" (UID: \"a3f067c6-43a9-48de-9acb-7c1fcc44389e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"
Sep 18 16:26:37.507438 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.507415    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3f067c6-43a9-48de-9acb-7c1fcc44389e-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl\" (UID: \"a3f067c6-43a9-48de-9acb-7c1fcc44389e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"
Sep 18 16:26:37.507529 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.507476    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3f067c6-43a9-48de-9acb-7c1fcc44389e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl\" (UID: \"a3f067c6-43a9-48de-9acb-7c1fcc44389e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"
Sep 18 16:26:37.509433 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.509413    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3f067c6-43a9-48de-9acb-7c1fcc44389e-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl\" (UID: \"a3f067c6-43a9-48de-9acb-7c1fcc44389e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"
Sep 18 16:26:37.519620 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.519599    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z67hm\" (UniqueName: \"kubernetes.io/projected/a3f067c6-43a9-48de-9acb-7c1fcc44389e-kube-api-access-z67hm\") pod \"isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl\" (UID: \"a3f067c6-43a9-48de-9acb-7c1fcc44389e\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"
Sep 18 16:26:37.535663 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.535635    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:37.560792 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.560766    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"
Sep 18 16:26:37.569332 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.569310    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"]
Sep 18 16:26:37.573082 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.573064    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"
Sep 18 16:26:37.575571 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.575550    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
Sep 18 16:26:37.575571 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.575563    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
Sep 18 16:26:37.583859 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.583827    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"]
Sep 18 16:26:37.607389 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.607362    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cdefa0f-50a9-40a3-8571-c90a98531b0e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-191f9-predictor-6f8b8449b5-shdc6\" (UID: \"3cdefa0f-50a9-40a3-8571-c90a98531b0e\") " pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6"
Sep 18 16:26:37.607593 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.607395    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwf5s\" (UniqueName: \"kubernetes.io/projected/3cdefa0f-50a9-40a3-8571-c90a98531b0e-kube-api-access-xwf5s\") pod \"success-200-isvc-191f9-predictor-6f8b8449b5-shdc6\" (UID: \"3cdefa0f-50a9-40a3-8571-c90a98531b0e\") " pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6"
Sep 18 16:26:37.607593 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.607415    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cdefa0f-50a9-40a3-8571-c90a98531b0e-proxy-tls\") pod \"success-200-isvc-191f9-predictor-6f8b8449b5-shdc6\" (UID: \"3cdefa0f-50a9-40a3-8571-c90a98531b0e\") " pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6"
Sep 18 16:26:37.607593 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:26:37.607552    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-191f9-predictor-serving-cert: secret "success-200-isvc-191f9-predictor-serving-cert" not found
Sep 18 16:26:37.607783 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:26:37.607619    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cdefa0f-50a9-40a3-8571-c90a98531b0e-proxy-tls podName:3cdefa0f-50a9-40a3-8571-c90a98531b0e nodeName:}" failed. No retries permitted until 2026-09-18 16:26:38.107598734 +0000 UTC m=+644.195914365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3cdefa0f-50a9-40a3-8571-c90a98531b0e-proxy-tls") pod "success-200-isvc-191f9-predictor-6f8b8449b5-shdc6" (UID: "3cdefa0f-50a9-40a3-8571-c90a98531b0e") : secret "success-200-isvc-191f9-predictor-serving-cert" not found
Sep 18 16:26:37.607849 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.607780    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:37.608081 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.608059    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cdefa0f-50a9-40a3-8571-c90a98531b0e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-191f9-predictor-6f8b8449b5-shdc6\" (UID: \"3cdefa0f-50a9-40a3-8571-c90a98531b0e\") " pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6"
Sep 18 16:26:37.619988 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.619964    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwf5s\" (UniqueName: \"kubernetes.io/projected/3cdefa0f-50a9-40a3-8571-c90a98531b0e-kube-api-access-xwf5s\") pod \"success-200-isvc-191f9-predictor-6f8b8449b5-shdc6\" (UID: \"3cdefa0f-50a9-40a3-8571-c90a98531b0e\") " pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6"
Sep 18 16:26:37.708097 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.708063    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/656d406f-8dff-43e6-95f5-4908cfcc3179-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jp7sd\" (UID: \"656d406f-8dff-43e6-95f5-4908cfcc3179\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"
Sep 18 16:26:37.708097 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.708099    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjmm8\" (UniqueName: \"kubernetes.io/projected/656d406f-8dff-43e6-95f5-4908cfcc3179-kube-api-access-qjmm8\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jp7sd\" (UID: \"656d406f-8dff-43e6-95f5-4908cfcc3179\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"
Sep 18 16:26:37.708329 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.708169    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/656d406f-8dff-43e6-95f5-4908cfcc3179-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jp7sd\" (UID: \"656d406f-8dff-43e6-95f5-4908cfcc3179\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"
Sep 18 16:26:37.708329 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.708192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/656d406f-8dff-43e6-95f5-4908cfcc3179-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jp7sd\" (UID: \"656d406f-8dff-43e6-95f5-4908cfcc3179\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"
Sep 18 16:26:37.708329 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.708207    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/656d406f-8dff-43e6-95f5-4908cfcc3179-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jp7sd\" (UID: \"656d406f-8dff-43e6-95f5-4908cfcc3179\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"
Sep 18 16:26:37.709920 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.709896    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:37.741706 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.741675    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"]
Sep 18 16:26:37.745505 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.745485    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"
Sep 18 16:26:37.747985 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.747963    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Sep 18 16:26:37.748085 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.747960    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Sep 18 16:26:37.757095 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.757070    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"]
Sep 18 16:26:37.808716 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.808684    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk\" (UID: \"7dd084c0-f257-44bd-8197-18bc1aa1b0d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"
Sep 18 16:26:37.808846 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.808722    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/656d406f-8dff-43e6-95f5-4908cfcc3179-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jp7sd\" (UID: \"656d406f-8dff-43e6-95f5-4908cfcc3179\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"
Sep 18 16:26:37.808846 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.808770    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk\" (UID: \"7dd084c0-f257-44bd-8197-18bc1aa1b0d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"
Sep 18 16:26:37.808846 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.808806    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjmm8\" (UniqueName: \"kubernetes.io/projected/656d406f-8dff-43e6-95f5-4908cfcc3179-kube-api-access-qjmm8\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jp7sd\" (UID: \"656d406f-8dff-43e6-95f5-4908cfcc3179\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"
Sep 18 16:26:37.808985 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.808871    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk\" (UID: \"7dd084c0-f257-44bd-8197-18bc1aa1b0d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"
Sep 18 16:26:37.808985 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.808914    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk\" (UID: \"7dd084c0-f257-44bd-8197-18bc1aa1b0d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"
Sep 18 16:26:37.809063 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.808996    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/656d406f-8dff-43e6-95f5-4908cfcc3179-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jp7sd\" (UID: \"656d406f-8dff-43e6-95f5-4908cfcc3179\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"
Sep 18 16:26:37.809063 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.809042    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49r8t\" (UniqueName: \"kubernetes.io/projected/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-kube-api-access-49r8t\") pod \"isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk\" (UID: \"7dd084c0-f257-44bd-8197-18bc1aa1b0d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"
Sep 18 16:26:37.809159 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.809091    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/656d406f-8dff-43e6-95f5-4908cfcc3179-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jp7sd\" (UID: \"656d406f-8dff-43e6-95f5-4908cfcc3179\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"
Sep 18 16:26:37.809159 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.809096    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/656d406f-8dff-43e6-95f5-4908cfcc3179-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jp7sd\" (UID: \"656d406f-8dff-43e6-95f5-4908cfcc3179\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"
Sep 18 16:26:37.809159 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.809117    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/656d406f-8dff-43e6-95f5-4908cfcc3179-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jp7sd\" (UID: \"656d406f-8dff-43e6-95f5-4908cfcc3179\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"
Sep 18 16:26:37.809326 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:26:37.809212    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-predictor-serving-cert: secret "isvc-xgboost-graph-predictor-serving-cert" not found
Sep 18 16:26:37.809326 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:26:37.809289    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/656d406f-8dff-43e6-95f5-4908cfcc3179-proxy-tls podName:656d406f-8dff-43e6-95f5-4908cfcc3179 nodeName:}" failed. No retries permitted until 2026-09-18 16:26:38.30926895 +0000 UTC m=+644.397584590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/656d406f-8dff-43e6-95f5-4908cfcc3179-proxy-tls") pod "isvc-xgboost-graph-predictor-555fc69d56-jp7sd" (UID: "656d406f-8dff-43e6-95f5-4908cfcc3179") : secret "isvc-xgboost-graph-predictor-serving-cert" not found
Sep 18 16:26:37.809608 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.809590    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/656d406f-8dff-43e6-95f5-4908cfcc3179-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jp7sd\" (UID: \"656d406f-8dff-43e6-95f5-4908cfcc3179\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"
Sep 18 16:26:37.809665 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.809640    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/656d406f-8dff-43e6-95f5-4908cfcc3179-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jp7sd\" (UID: \"656d406f-8dff-43e6-95f5-4908cfcc3179\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"
Sep 18 16:26:37.821202 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.821178    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjmm8\" (UniqueName: \"kubernetes.io/projected/656d406f-8dff-43e6-95f5-4908cfcc3179-kube-api-access-qjmm8\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jp7sd\" (UID: \"656d406f-8dff-43e6-95f5-4908cfcc3179\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"
Sep 18 16:26:37.823393 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.823369    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:37.842409 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.842391    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"]
Sep 18 16:26:37.843385 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:26:37.843362    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f067c6_43a9_48de_9acb_7c1fcc44389e.slice/crio-c9bee4a3f1e5de7ab92984e7ffa76fd7341978dd04d5871397af4759ce646651 WatchSource:0}: Error finding container c9bee4a3f1e5de7ab92984e7ffa76fd7341978dd04d5871397af4759ce646651: Status 404 returned error can't find the container with id c9bee4a3f1e5de7ab92984e7ffa76fd7341978dd04d5871397af4759ce646651
Sep 18 16:26:37.909754 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.909721    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk\" (UID: \"7dd084c0-f257-44bd-8197-18bc1aa1b0d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"
Sep 18 16:26:37.909862 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.909773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk\" (UID: \"7dd084c0-f257-44bd-8197-18bc1aa1b0d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"
Sep 18 16:26:37.909862 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.909840    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk\" (UID: \"7dd084c0-f257-44bd-8197-18bc1aa1b0d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"
Sep 18 16:26:37.909978 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.909885    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk\" (UID: \"7dd084c0-f257-44bd-8197-18bc1aa1b0d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"
Sep 18 16:26:37.909978 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.909947    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49r8t\" (UniqueName: \"kubernetes.io/projected/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-kube-api-access-49r8t\") pod \"isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk\" (UID: \"7dd084c0-f257-44bd-8197-18bc1aa1b0d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"
Sep 18 16:26:37.910359 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:26:37.910248    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-serving-cert: secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Sep 18 16:26:37.910472 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:26:37.910364    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-proxy-tls podName:7dd084c0-f257-44bd-8197-18bc1aa1b0d9 nodeName:}" failed. No retries permitted until 2026-09-18 16:26:38.410343717 +0000 UTC m=+644.498659351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-proxy-tls") pod "isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk" (UID: "7dd084c0-f257-44bd-8197-18bc1aa1b0d9") : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Sep 18 16:26:37.910472 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.910460    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk\" (UID: \"7dd084c0-f257-44bd-8197-18bc1aa1b0d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"
Sep 18 16:26:37.910680 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.910656    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk\" (UID: \"7dd084c0-f257-44bd-8197-18bc1aa1b0d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"
Sep 18 16:26:37.910764 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.910749    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk\" (UID: \"7dd084c0-f257-44bd-8197-18bc1aa1b0d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"
Sep 18 16:26:37.914693 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.914670    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl" event={"ID":"a3f067c6-43a9-48de-9acb-7c1fcc44389e","Type":"ContainerStarted","Data":"5738df0a9a84707cc617d0d76f745dc7c73dfbe1870a323b891cecb5041ea102"}
Sep 18 16:26:37.914776 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.914743    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl" event={"ID":"a3f067c6-43a9-48de-9acb-7c1fcc44389e","Type":"ContainerStarted","Data":"c9bee4a3f1e5de7ab92984e7ffa76fd7341978dd04d5871397af4759ce646651"}
Sep 18 16:26:37.919347 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:37.919330    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49r8t\" (UniqueName: \"kubernetes.io/projected/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-kube-api-access-49r8t\") pod \"isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk\" (UID: \"7dd084c0-f257-44bd-8197-18bc1aa1b0d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"
Sep 18 16:26:38.112287 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:38.112196    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cdefa0f-50a9-40a3-8571-c90a98531b0e-proxy-tls\") pod \"success-200-isvc-191f9-predictor-6f8b8449b5-shdc6\" (UID: \"3cdefa0f-50a9-40a3-8571-c90a98531b0e\") " pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6"
Sep 18 16:26:38.114615 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:38.114584    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cdefa0f-50a9-40a3-8571-c90a98531b0e-proxy-tls\") pod \"success-200-isvc-191f9-predictor-6f8b8449b5-shdc6\" (UID: \"3cdefa0f-50a9-40a3-8571-c90a98531b0e\") " pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6"
Sep 18 16:26:38.260873 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:38.260837    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6"
Sep 18 16:26:38.288883 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:38.288849    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:38.314723 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:38.314392    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/656d406f-8dff-43e6-95f5-4908cfcc3179-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jp7sd\" (UID: \"656d406f-8dff-43e6-95f5-4908cfcc3179\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"
Sep 18 16:26:38.314892 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:26:38.314744    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-predictor-serving-cert: secret "isvc-xgboost-graph-predictor-serving-cert" not found
Sep 18 16:26:38.314892 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:26:38.314829    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/656d406f-8dff-43e6-95f5-4908cfcc3179-proxy-tls podName:656d406f-8dff-43e6-95f5-4908cfcc3179 nodeName:}" failed. No retries permitted until 2026-09-18 16:26:39.314809578 +0000 UTC m=+645.403125216 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/656d406f-8dff-43e6-95f5-4908cfcc3179-proxy-tls") pod "isvc-xgboost-graph-predictor-555fc69d56-jp7sd" (UID: "656d406f-8dff-43e6-95f5-4908cfcc3179") : secret "isvc-xgboost-graph-predictor-serving-cert" not found
Sep 18 16:26:38.416023 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:38.415935    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk\" (UID: \"7dd084c0-f257-44bd-8197-18bc1aa1b0d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"
Sep 18 16:26:38.418447 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:38.418407    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk\" (UID: \"7dd084c0-f257-44bd-8197-18bc1aa1b0d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"
Sep 18 16:26:38.507520 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:38.507486    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:38.534981 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:38.534844    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6"]
Sep 18 16:26:38.536970 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:26:38.536947    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cdefa0f_50a9_40a3_8571_c90a98531b0e.slice/crio-4488359f1642470bead5deddc60788b30c68f713fb38852b31bac169e08c7263 WatchSource:0}: Error finding container 4488359f1642470bead5deddc60788b30c68f713fb38852b31bac169e08c7263: Status 404 returned error can't find the container with id 4488359f1642470bead5deddc60788b30c68f713fb38852b31bac169e08c7263
Sep 18 16:26:38.656483 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:38.656447    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"
Sep 18 16:26:38.919945 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:38.919868    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6" event={"ID":"3cdefa0f-50a9-40a3-8571-c90a98531b0e","Type":"ContainerStarted","Data":"4488359f1642470bead5deddc60788b30c68f713fb38852b31bac169e08c7263"}
Sep 18 16:26:38.929600 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:38.929580    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"]
Sep 18 16:26:38.931562 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:26:38.931537    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd084c0_f257_44bd_8197_18bc1aa1b0d9.slice/crio-a098f6400b5316fe3751f1cff5061170ec758db726b89c08117716c2863c63b5 WatchSource:0}: Error finding container a098f6400b5316fe3751f1cff5061170ec758db726b89c08117716c2863c63b5: Status 404 returned error can't find the container with id a098f6400b5316fe3751f1cff5061170ec758db726b89c08117716c2863c63b5
Sep 18 16:26:39.326333 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:39.326276    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/656d406f-8dff-43e6-95f5-4908cfcc3179-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jp7sd\" (UID: \"656d406f-8dff-43e6-95f5-4908cfcc3179\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"
Sep 18 16:26:39.342842 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:39.342776    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/656d406f-8dff-43e6-95f5-4908cfcc3179-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-jp7sd\" (UID: \"656d406f-8dff-43e6-95f5-4908cfcc3179\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"
Sep 18 16:26:39.393905 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:39.393867    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"
Sep 18 16:26:39.704555 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:39.704517    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"]
Sep 18 16:26:39.721460 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:26:39.720515    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod656d406f_8dff_43e6_95f5_4908cfcc3179.slice/crio-b4387263d3d2b4cb420cd1d7b894c60d69f3545280a5503582452fb07a02e0b0 WatchSource:0}: Error finding container b4387263d3d2b4cb420cd1d7b894c60d69f3545280a5503582452fb07a02e0b0: Status 404 returned error can't find the container with id b4387263d3d2b4cb420cd1d7b894c60d69f3545280a5503582452fb07a02e0b0
Sep 18 16:26:39.931586 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:39.931288    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd" event={"ID":"656d406f-8dff-43e6-95f5-4908cfcc3179","Type":"ContainerStarted","Data":"772b37a5fa8b37d69ea46b40b9d14b7cb1d73120196c41b54ce53e18b70eb845"}
Sep 18 16:26:39.931586 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:39.931388    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd" event={"ID":"656d406f-8dff-43e6-95f5-4908cfcc3179","Type":"ContainerStarted","Data":"b4387263d3d2b4cb420cd1d7b894c60d69f3545280a5503582452fb07a02e0b0"}
Sep 18 16:26:39.934259 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:39.934168    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk" event={"ID":"7dd084c0-f257-44bd-8197-18bc1aa1b0d9","Type":"ContainerStarted","Data":"c06d2dc11c7be3c4dae4f31c32d4fd7345550adffa702fe50ab5220a38d55e6f"}
Sep 18 16:26:39.934259 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:39.934229    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk" event={"ID":"7dd084c0-f257-44bd-8197-18bc1aa1b0d9","Type":"ContainerStarted","Data":"a098f6400b5316fe3751f1cff5061170ec758db726b89c08117716c2863c63b5"}
Sep 18 16:26:43.968116 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:43.968083    2580 generic.go:356] "Generic (PLEG): container finished" podID="7dd084c0-f257-44bd-8197-18bc1aa1b0d9" containerID="c06d2dc11c7be3c4dae4f31c32d4fd7345550adffa702fe50ab5220a38d55e6f" exitCode=0
Sep 18 16:26:43.968550 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:43.968168    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk" event={"ID":"7dd084c0-f257-44bd-8197-18bc1aa1b0d9","Type":"ContainerDied","Data":"c06d2dc11c7be3c4dae4f31c32d4fd7345550adffa702fe50ab5220a38d55e6f"}
Sep 18 16:26:43.970141 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:43.970040    2580 generic.go:356] "Generic (PLEG): container finished" podID="a3f067c6-43a9-48de-9acb-7c1fcc44389e" containerID="5738df0a9a84707cc617d0d76f745dc7c73dfbe1870a323b891cecb5041ea102" exitCode=0
Sep 18 16:26:43.970141 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:43.970075    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl" event={"ID":"a3f067c6-43a9-48de-9acb-7c1fcc44389e","Type":"ContainerDied","Data":"5738df0a9a84707cc617d0d76f745dc7c73dfbe1870a323b891cecb5041ea102"}
Sep 18 16:26:47.995126 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:47.995092    2580 generic.go:356] "Generic (PLEG): container finished" podID="656d406f-8dff-43e6-95f5-4908cfcc3179" containerID="772b37a5fa8b37d69ea46b40b9d14b7cb1d73120196c41b54ce53e18b70eb845" exitCode=0
Sep 18 16:26:47.995546 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:47.995131    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd" event={"ID":"656d406f-8dff-43e6-95f5-4908cfcc3179","Type":"ContainerDied","Data":"772b37a5fa8b37d69ea46b40b9d14b7cb1d73120196c41b54ce53e18b70eb845"}
Sep 18 16:26:56.059955 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:56.059919    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk" event={"ID":"7dd084c0-f257-44bd-8197-18bc1aa1b0d9","Type":"ContainerStarted","Data":"f7f68ff4eef4108a3ff613b971383b0eb8eea45fd1fc43bb7ca866b0e52acbbf"}
Sep 18 16:26:56.063346 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:56.063276    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl" event={"ID":"a3f067c6-43a9-48de-9acb-7c1fcc44389e","Type":"ContainerStarted","Data":"c423881f0a26c2dcebde7569c107c160e8d74c6243b9557834376de5745f201a"}
Sep 18 16:26:56.068607 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:26:56.068576    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6" event={"ID":"3cdefa0f-50a9-40a3-8571-c90a98531b0e","Type":"ContainerStarted","Data":"e099e128bce9ce2b961ca7977fd0a66643fc0bf1b4b04029991510778134aa1f"}
Sep 18 16:27:00.097881 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:00.097789    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl" event={"ID":"a3f067c6-43a9-48de-9acb-7c1fcc44389e","Type":"ContainerStarted","Data":"94633ac90032a18512ab768330bcdf61cb99e1583297390cd3991c5d7b9abaee"}
Sep 18 16:27:00.098343 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:00.097966    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"
Sep 18 16:27:00.099973 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:00.099923    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6" event={"ID":"3cdefa0f-50a9-40a3-8571-c90a98531b0e","Type":"ContainerStarted","Data":"d38e56c13c69d7aa8f6182a652343529019b6b17311d1a630c63df6807d0138c"}
Sep 18 16:27:00.100078 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:00.099991    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6"
Sep 18 16:27:00.100078 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:00.100022    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6"
Sep 18 16:27:00.101713 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:00.101675    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6" podUID="3cdefa0f-50a9-40a3-8571-c90a98531b0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 16:27:00.109525 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:00.109497    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk" event={"ID":"7dd084c0-f257-44bd-8197-18bc1aa1b0d9","Type":"ContainerStarted","Data":"09be7a5f914565465ee6fd943dabb49d72d4d7eb8c7ebe2ee2674198761855b2"}
Sep 18 16:27:00.109717 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:00.109694    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"
Sep 18 16:27:00.109825 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:00.109734    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"
Sep 18 16:27:00.110997 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:00.110973    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk" podUID="7dd084c0-f257-44bd-8197-18bc1aa1b0d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 16:27:00.120492 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:00.120350    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl" podStartSLOduration=3.416738805 podStartE2EDuration="23.120330377s" totalImagesPullingTime="15.901737087s" totalInitContainerRuntime="3.801854485s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:26:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:27:00.119211547 +0000 UTC m=+666.207527189" watchObservedRunningTime="2026-09-18 16:27:00.120330377 +0000 UTC m=+666.208646017"
Sep 18 16:27:00.137455 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:00.137399    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6" podStartSLOduration=1.872469924 podStartE2EDuration="23.137378145s" totalImagesPullingTime="21.264908221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:26:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:27:00.136813006 +0000 UTC m=+666.225128682" watchObservedRunningTime="2026-09-18 16:27:00.137378145 +0000 UTC m=+666.225693787"
Sep 18 16:27:00.158987 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:00.158932    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk" podStartSLOduration=3.111020084 podStartE2EDuration="23.158916384s" totalImagesPullingTime="15.889676568s" totalInitContainerRuntime="4.158219732s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:26:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:27:00.15641581 +0000 UTC m=+666.244731460" watchObservedRunningTime="2026-09-18 16:27:00.158916384 +0000 UTC m=+666.247232024"
Sep 18 16:27:01.116251 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:01.116013    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk" podUID="7dd084c0-f257-44bd-8197-18bc1aa1b0d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 16:27:01.116730 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:01.116446    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6" podUID="3cdefa0f-50a9-40a3-8571-c90a98531b0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 16:27:01.116730 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:01.116556    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"
Sep 18 16:27:01.117383 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:01.117349    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl" podUID="a3f067c6-43a9-48de-9acb-7c1fcc44389e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 16:27:02.121062 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:02.121021    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl" podUID="a3f067c6-43a9-48de-9acb-7c1fcc44389e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 16:27:06.124429 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:06.124401    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"
Sep 18 16:27:06.124883 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:06.124474    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6"
Sep 18 16:27:06.124959 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:06.124891    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk" podUID="7dd084c0-f257-44bd-8197-18bc1aa1b0d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 16:27:06.124959 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:06.124891    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6" podUID="3cdefa0f-50a9-40a3-8571-c90a98531b0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 16:27:07.126414 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:07.126379    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"
Sep 18 16:27:07.127040 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:07.127012    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl" podUID="a3f067c6-43a9-48de-9acb-7c1fcc44389e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 16:27:16.125012 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:16.124971    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk" podUID="7dd084c0-f257-44bd-8197-18bc1aa1b0d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 16:27:16.125561 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:16.124972    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6" podUID="3cdefa0f-50a9-40a3-8571-c90a98531b0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 16:27:17.127669 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:17.127633    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl" podUID="a3f067c6-43a9-48de-9acb-7c1fcc44389e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 16:27:17.199751 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:17.199722    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd" event={"ID":"656d406f-8dff-43e6-95f5-4908cfcc3179","Type":"ContainerStarted","Data":"b5713c1738409dbf116eef4379e09df6fa3ec521f7c51a37cb04370a2ddbac9e"}
Sep 18 16:27:18.206126 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:18.206087    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd" event={"ID":"656d406f-8dff-43e6-95f5-4908cfcc3179","Type":"ContainerStarted","Data":"43281943330c7b12857412b502a9b8740a7c60149be6a3f89b810c8f90fcc8c2"}
Sep 18 16:27:18.206632 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:18.206158    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"
Sep 18 16:27:18.206632 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:18.206318    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"
Sep 18 16:27:18.207702 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:18.207678    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd" podUID="656d406f-8dff-43e6-95f5-4908cfcc3179" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 16:27:18.227437 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:18.227384    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd" podStartSLOduration=7.9901154420000005 podStartE2EDuration="41.227367766s" totalImagesPullingTime="29.029725598s" totalInitContainerRuntime="4.207526726s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:27:18.225147777 +0000 UTC m=+684.313463440" watchObservedRunningTime="2026-09-18 16:27:18.227367766 +0000 UTC m=+684.315683404"
Sep 18 16:27:19.211941 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:19.211897    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd" podUID="656d406f-8dff-43e6-95f5-4908cfcc3179" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 16:27:24.218181 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:24.218139    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"
Sep 18 16:27:24.218846 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:24.218810    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd" podUID="656d406f-8dff-43e6-95f5-4908cfcc3179" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 16:27:26.125344 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:26.125292    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk" podUID="7dd084c0-f257-44bd-8197-18bc1aa1b0d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 16:27:26.125735 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:26.125293    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6" podUID="3cdefa0f-50a9-40a3-8571-c90a98531b0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 16:27:27.127614 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:27.127577    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl" podUID="a3f067c6-43a9-48de-9acb-7c1fcc44389e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 16:27:34.219212 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:34.219163    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd" podUID="656d406f-8dff-43e6-95f5-4908cfcc3179" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 16:27:36.125099 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:36.125065    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk" podUID="7dd084c0-f257-44bd-8197-18bc1aa1b0d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 16:27:36.125479 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:36.125065    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6" podUID="3cdefa0f-50a9-40a3-8571-c90a98531b0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 16:27:37.127525 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:37.127486    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl" podUID="a3f067c6-43a9-48de-9acb-7c1fcc44389e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 16:27:44.219573 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:44.219462    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd" podUID="656d406f-8dff-43e6-95f5-4908cfcc3179" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 16:27:46.125148 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:46.125104    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6" podUID="3cdefa0f-50a9-40a3-8571-c90a98531b0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 16:27:46.125608 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:46.125107    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk" podUID="7dd084c0-f257-44bd-8197-18bc1aa1b0d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 16:27:47.127753 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:47.127719    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl" podUID="a3f067c6-43a9-48de-9acb-7c1fcc44389e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 16:27:54.219480 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:54.219429    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd" podUID="656d406f-8dff-43e6-95f5-4908cfcc3179" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 16:27:56.125764 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:56.125722    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk" podUID="7dd084c0-f257-44bd-8197-18bc1aa1b0d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 16:27:56.126450 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:56.126428    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6"
Sep 18 16:27:57.127188 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:27:57.127151    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl" podUID="a3f067c6-43a9-48de-9acb-7c1fcc44389e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 16:28:04.219512 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:04.219473    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd" podUID="656d406f-8dff-43e6-95f5-4908cfcc3179" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 16:28:06.124882 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:06.124842    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk" podUID="7dd084c0-f257-44bd-8197-18bc1aa1b0d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 16:28:07.127239 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:07.127205    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl" podUID="a3f067c6-43a9-48de-9acb-7c1fcc44389e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 16:28:07.173498 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:07.173468    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:28:07.838236 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:07.838196    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:28:09.116926 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:09.116876    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk" podUID="7dd084c0-f257-44bd-8197-18bc1aa1b0d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 16:28:13.117252 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:13.117208    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl" podUID="a3f067c6-43a9-48de-9acb-7c1fcc44389e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 16:28:14.219260 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:14.219223    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd" podUID="656d406f-8dff-43e6-95f5-4908cfcc3179" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 16:28:18.314520 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:18.314489    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:28:18.428071 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:18.428038    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6"]
Sep 18 16:28:18.428281 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:18.428259    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6" podUID="3cdefa0f-50a9-40a3-8571-c90a98531b0e" containerName="kserve-container" containerID="cri-o://e099e128bce9ce2b961ca7977fd0a66643fc0bf1b4b04029991510778134aa1f" gracePeriod=30
Sep 18 16:28:18.428388 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:18.428332    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6" podUID="3cdefa0f-50a9-40a3-8571-c90a98531b0e" containerName="kube-rbac-proxy" containerID="cri-o://d38e56c13c69d7aa8f6182a652343529019b6b17311d1a630c63df6807d0138c" gracePeriod=30
Sep 18 16:28:18.455147 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:18.455115    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:28:18.565296 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:18.565220    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:28:18.584575 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:18.584543    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg"]
Sep 18 16:28:18.586973 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:18.586956    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg"
Sep 18 16:28:18.589347 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:18.589323    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8aa04-predictor-serving-cert\""
Sep 18 16:28:18.589620 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:18.589603    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8aa04-kube-rbac-proxy-sar-config\""
Sep 18 16:28:18.598420 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:18.598398    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg"]
Sep 18 16:28:18.656508 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:18.656479    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:28:18.722097 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:18.722068    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:28:18.734372 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:18.734347    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3354167c-5adc-4697-a501-a3985a5aa4c6-proxy-tls\") pod \"success-200-isvc-8aa04-predictor-b58b89b85-5tmrg\" (UID: \"3354167c-5adc-4697-a501-a3985a5aa4c6\") " pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg"
Sep 18 16:28:18.734480 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:18.734406    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3354167c-5adc-4697-a501-a3985a5aa4c6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8aa04-predictor-b58b89b85-5tmrg\" (UID: \"3354167c-5adc-4697-a501-a3985a5aa4c6\") " pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg"
Sep 18 16:28:18.734526 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:18.734494    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpdj8\" (UniqueName: \"kubernetes.io/projected/3354167c-5adc-4697-a501-a3985a5aa4c6-kube-api-access-jpdj8\") pod \"success-200-isvc-8aa04-predictor-b58b89b85-5tmrg\" (UID: \"3354167c-5adc-4697-a501-a3985a5aa4c6\") " pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg"
Sep 18 16:28:18.822492 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:18.822417    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:28:18.835603 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:18.835581    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3354167c-5adc-4697-a501-a3985a5aa4c6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8aa04-predictor-b58b89b85-5tmrg\" (UID: \"3354167c-5adc-4697-a501-a3985a5aa4c6\") " pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg"
Sep 18 16:28:18.835677 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:18.835648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpdj8\" (UniqueName: \"kubernetes.io/projected/3354167c-5adc-4697-a501-a3985a5aa4c6-kube-api-access-jpdj8\") pod \"success-200-isvc-8aa04-predictor-b58b89b85-5tmrg\" (UID: \"3354167c-5adc-4697-a501-a3985a5aa4c6\") " pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg"
Sep 18 16:28:18.835716 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:18.835678    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3354167c-5adc-4697-a501-a3985a5aa4c6-proxy-tls\") pod \"success-200-isvc-8aa04-predictor-b58b89b85-5tmrg\" (UID: \"3354167c-5adc-4697-a501-a3985a5aa4c6\") " pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg"
Sep 18 16:28:18.836208 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:18.836191    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3354167c-5adc-4697-a501-a3985a5aa4c6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8aa04-predictor-b58b89b85-5tmrg\" (UID: \"3354167c-5adc-4697-a501-a3985a5aa4c6\") " pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg"
Sep 18 16:28:18.837981 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:18.837962    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3354167c-5adc-4697-a501-a3985a5aa4c6-proxy-tls\") pod \"success-200-isvc-8aa04-predictor-b58b89b85-5tmrg\" (UID: \"3354167c-5adc-4697-a501-a3985a5aa4c6\") " pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg"
Sep 18 16:28:18.846850 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:18.846825    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpdj8\" (UniqueName: \"kubernetes.io/projected/3354167c-5adc-4697-a501-a3985a5aa4c6-kube-api-access-jpdj8\") pod \"success-200-isvc-8aa04-predictor-b58b89b85-5tmrg\" (UID: \"3354167c-5adc-4697-a501-a3985a5aa4c6\") " pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg"
Sep 18 16:28:18.897271 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:18.897244    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg"
Sep 18 16:28:18.923509 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:18.923479    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:28:19.117347 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:19.117225    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk" podUID="7dd084c0-f257-44bd-8197-18bc1aa1b0d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 16:28:19.174783 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:19.174761    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg"]
Sep 18 16:28:19.177348 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:28:19.177316    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3354167c_5adc_4697_a501_a3985a5aa4c6.slice/crio-6376207f8860cecd5bcd01e5b134ff7ba925ad887f02a35ac5c63c94c594446c WatchSource:0}: Error finding container 6376207f8860cecd5bcd01e5b134ff7ba925ad887f02a35ac5c63c94c594446c: Status 404 returned error can't find the container with id 6376207f8860cecd5bcd01e5b134ff7ba925ad887f02a35ac5c63c94c594446c
Sep 18 16:28:19.503341 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:19.503288    2580 generic.go:356] "Generic (PLEG): container finished" podID="3cdefa0f-50a9-40a3-8571-c90a98531b0e" containerID="d38e56c13c69d7aa8f6182a652343529019b6b17311d1a630c63df6807d0138c" exitCode=2
Sep 18 16:28:19.503727 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:19.503337    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6" event={"ID":"3cdefa0f-50a9-40a3-8571-c90a98531b0e","Type":"ContainerDied","Data":"d38e56c13c69d7aa8f6182a652343529019b6b17311d1a630c63df6807d0138c"}
Sep 18 16:28:19.505160 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:19.505138    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg" event={"ID":"3354167c-5adc-4697-a501-a3985a5aa4c6","Type":"ContainerStarted","Data":"ad18fce5db1de1cc61f329ff4970912697d5fd8308c9601355c69585cd3f4e74"}
Sep 18 16:28:19.505271 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:19.505212    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg"
Sep 18 16:28:19.505271 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:19.505234    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg"
Sep 18 16:28:19.505271 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:19.505252    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg" event={"ID":"3354167c-5adc-4697-a501-a3985a5aa4c6","Type":"ContainerStarted","Data":"4239f5ec148950d9b4fe1b0c7da1457bde9c4958c89c30dd0d25bf686ef6c6df"}
Sep 18 16:28:19.505271 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:19.505269    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg" event={"ID":"3354167c-5adc-4697-a501-a3985a5aa4c6","Type":"ContainerStarted","Data":"6376207f8860cecd5bcd01e5b134ff7ba925ad887f02a35ac5c63c94c594446c"}
Sep 18 16:28:19.506542 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:19.506484    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg" podUID="3354167c-5adc-4697-a501-a3985a5aa4c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 16:28:19.523779 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:19.523726    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg" podStartSLOduration=1.523712298 podStartE2EDuration="1.523712298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:28:19.521653752 +0000 UTC m=+745.609969384" watchObservedRunningTime="2026-09-18 16:28:19.523712298 +0000 UTC m=+745.612028006"
Sep 18 16:28:20.510483 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:20.510448    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg" podUID="3354167c-5adc-4697-a501-a3985a5aa4c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 16:28:21.116157 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:21.116106    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6" podUID="3cdefa0f-50a9-40a3-8571-c90a98531b0e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 18 16:28:22.263892 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:22.263868    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6"
Sep 18 16:28:22.369656 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:22.369619    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cdefa0f-50a9-40a3-8571-c90a98531b0e-proxy-tls\") pod \"3cdefa0f-50a9-40a3-8571-c90a98531b0e\" (UID: \"3cdefa0f-50a9-40a3-8571-c90a98531b0e\") "
Sep 18 16:28:22.369832 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:22.369726    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cdefa0f-50a9-40a3-8571-c90a98531b0e-kube-rbac-proxy-sar-config\") pod \"3cdefa0f-50a9-40a3-8571-c90a98531b0e\" (UID: \"3cdefa0f-50a9-40a3-8571-c90a98531b0e\") "
Sep 18 16:28:22.369832 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:22.369799    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwf5s\" (UniqueName: \"kubernetes.io/projected/3cdefa0f-50a9-40a3-8571-c90a98531b0e-kube-api-access-xwf5s\") pod \"3cdefa0f-50a9-40a3-8571-c90a98531b0e\" (UID: \"3cdefa0f-50a9-40a3-8571-c90a98531b0e\") "
Sep 18 16:28:22.370128 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:22.370096    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cdefa0f-50a9-40a3-8571-c90a98531b0e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3cdefa0f-50a9-40a3-8571-c90a98531b0e" (UID: "3cdefa0f-50a9-40a3-8571-c90a98531b0e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:28:22.372111 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:22.372042    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cdefa0f-50a9-40a3-8571-c90a98531b0e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3cdefa0f-50a9-40a3-8571-c90a98531b0e" (UID: "3cdefa0f-50a9-40a3-8571-c90a98531b0e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:28:22.372261 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:22.372223    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cdefa0f-50a9-40a3-8571-c90a98531b0e-kube-api-access-xwf5s" (OuterVolumeSpecName: "kube-api-access-xwf5s") pod "3cdefa0f-50a9-40a3-8571-c90a98531b0e" (UID: "3cdefa0f-50a9-40a3-8571-c90a98531b0e"). InnerVolumeSpecName "kube-api-access-xwf5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:28:22.471589 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:22.471559    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cdefa0f-50a9-40a3-8571-c90a98531b0e-proxy-tls\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:28:22.471589 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:22.471586    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cdefa0f-50a9-40a3-8571-c90a98531b0e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:28:22.471785 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:22.471600    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwf5s\" (UniqueName: \"kubernetes.io/projected/3cdefa0f-50a9-40a3-8571-c90a98531b0e-kube-api-access-xwf5s\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:28:22.524260 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:22.524221    2580 generic.go:356] "Generic (PLEG): container finished" podID="3cdefa0f-50a9-40a3-8571-c90a98531b0e" containerID="e099e128bce9ce2b961ca7977fd0a66643fc0bf1b4b04029991510778134aa1f" exitCode=0
Sep 18 16:28:22.524414 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:22.524269    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6"
Sep 18 16:28:22.524414 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:22.524326    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6" event={"ID":"3cdefa0f-50a9-40a3-8571-c90a98531b0e","Type":"ContainerDied","Data":"e099e128bce9ce2b961ca7977fd0a66643fc0bf1b4b04029991510778134aa1f"}
Sep 18 16:28:22.524414 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:22.524392    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6" event={"ID":"3cdefa0f-50a9-40a3-8571-c90a98531b0e","Type":"ContainerDied","Data":"4488359f1642470bead5deddc60788b30c68f713fb38852b31bac169e08c7263"}
Sep 18 16:28:22.524539 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:22.524424    2580 scope.go:117] "RemoveContainer" containerID="d38e56c13c69d7aa8f6182a652343529019b6b17311d1a630c63df6807d0138c"
Sep 18 16:28:22.533450 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:22.533433    2580 scope.go:117] "RemoveContainer" containerID="e099e128bce9ce2b961ca7977fd0a66643fc0bf1b4b04029991510778134aa1f"
Sep 18 16:28:22.541148 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:22.541130    2580 scope.go:117] "RemoveContainer" containerID="d38e56c13c69d7aa8f6182a652343529019b6b17311d1a630c63df6807d0138c"
Sep 18 16:28:22.541441 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:28:22.541419    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d38e56c13c69d7aa8f6182a652343529019b6b17311d1a630c63df6807d0138c\": container with ID starting with d38e56c13c69d7aa8f6182a652343529019b6b17311d1a630c63df6807d0138c not found: ID does not exist" containerID="d38e56c13c69d7aa8f6182a652343529019b6b17311d1a630c63df6807d0138c"
Sep 18 16:28:22.541505 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:22.541449    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d38e56c13c69d7aa8f6182a652343529019b6b17311d1a630c63df6807d0138c"} err="failed to get container status \"d38e56c13c69d7aa8f6182a652343529019b6b17311d1a630c63df6807d0138c\": rpc error: code = NotFound desc = could not find container \"d38e56c13c69d7aa8f6182a652343529019b6b17311d1a630c63df6807d0138c\": container with ID starting with d38e56c13c69d7aa8f6182a652343529019b6b17311d1a630c63df6807d0138c not found: ID does not exist"
Sep 18 16:28:22.541505 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:22.541467    2580 scope.go:117] "RemoveContainer" containerID="e099e128bce9ce2b961ca7977fd0a66643fc0bf1b4b04029991510778134aa1f"
Sep 18 16:28:22.541720 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:28:22.541706    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e099e128bce9ce2b961ca7977fd0a66643fc0bf1b4b04029991510778134aa1f\": container with ID starting with e099e128bce9ce2b961ca7977fd0a66643fc0bf1b4b04029991510778134aa1f not found: ID does not exist" containerID="e099e128bce9ce2b961ca7977fd0a66643fc0bf1b4b04029991510778134aa1f"
Sep 18 16:28:22.541754 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:22.541724    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e099e128bce9ce2b961ca7977fd0a66643fc0bf1b4b04029991510778134aa1f"} err="failed to get container status \"e099e128bce9ce2b961ca7977fd0a66643fc0bf1b4b04029991510778134aa1f\": rpc error: code = NotFound desc = could not find container \"e099e128bce9ce2b961ca7977fd0a66643fc0bf1b4b04029991510778134aa1f\": container with ID starting with e099e128bce9ce2b961ca7977fd0a66643fc0bf1b4b04029991510778134aa1f not found: ID does not exist"
Sep 18 16:28:22.546961 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:22.546937    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6"]
Sep 18 16:28:22.556449 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:22.556429    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-191f9-predictor-6f8b8449b5-shdc6"]
Sep 18 16:28:23.118042 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:23.118002    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"
Sep 18 16:28:24.123891 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:24.123856    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cdefa0f-50a9-40a3-8571-c90a98531b0e" path="/var/lib/kubelet/pods/3cdefa0f-50a9-40a3-8571-c90a98531b0e/volumes"
Sep 18 16:28:24.219599 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:24.219554    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd" podUID="656d406f-8dff-43e6-95f5-4908cfcc3179" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 16:28:24.481403 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:24.481371    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:28:25.514934 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:25.514900    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg"
Sep 18 16:28:25.515436 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:25.515343    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg" podUID="3354167c-5adc-4697-a501-a3985a5aa4c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 16:28:29.117389 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:29.117355    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"
Sep 18 16:28:34.220052 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:34.220023    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"
Sep 18 16:28:35.516198 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:35.516165    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg" podUID="3354167c-5adc-4697-a501-a3985a5aa4c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 16:28:45.516328 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:45.516273    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg" podUID="3354167c-5adc-4697-a501-a3985a5aa4c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 16:28:47.248698 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:47.248657    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:28:47.428932 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:47.428899    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:28:51.559806 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:51.559772    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:28:51.660462 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:51.660430    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:28:55.516057 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:55.516009    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg" podUID="3354167c-5adc-4697-a501-a3985a5aa4c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 16:28:58.833756 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:58.833709    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:28:58.903581 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:58.903550    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"]
Sep 18 16:28:58.903853 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:58.903828    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk" podUID="7dd084c0-f257-44bd-8197-18bc1aa1b0d9" containerName="kserve-container" containerID="cri-o://f7f68ff4eef4108a3ff613b971383b0eb8eea45fd1fc43bb7ca866b0e52acbbf" gracePeriod=30
Sep 18 16:28:58.903927 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:58.903853    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk" podUID="7dd084c0-f257-44bd-8197-18bc1aa1b0d9" containerName="kube-rbac-proxy" containerID="cri-o://09be7a5f914565465ee6fd943dabb49d72d4d7eb8c7ebe2ee2674198761855b2" gracePeriod=30
Sep 18 16:28:58.938795 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:58.938760    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:28:58.943426 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:58.943402    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"]
Sep 18 16:28:58.943626 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:58.943605    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl" podUID="a3f067c6-43a9-48de-9acb-7c1fcc44389e" containerName="kserve-container" containerID="cri-o://c423881f0a26c2dcebde7569c107c160e8d74c6243b9557834376de5745f201a" gracePeriod=30
Sep 18 16:28:58.943711 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:58.943648    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl" podUID="a3f067c6-43a9-48de-9acb-7c1fcc44389e" containerName="kube-rbac-proxy" containerID="cri-o://94633ac90032a18512ab768330bcdf61cb99e1583297390cd3991c5d7b9abaee" gracePeriod=30
Sep 18 16:28:58.977247 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:58.977221    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"]
Sep 18 16:28:58.977487 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:58.977466    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd" podUID="656d406f-8dff-43e6-95f5-4908cfcc3179" containerName="kserve-container" containerID="cri-o://b5713c1738409dbf116eef4379e09df6fa3ec521f7c51a37cb04370a2ddbac9e" gracePeriod=30
Sep 18 16:28:58.977569 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:58.977544    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd" podUID="656d406f-8dff-43e6-95f5-4908cfcc3179" containerName="kube-rbac-proxy" containerID="cri-o://43281943330c7b12857412b502a9b8740a7c60149be6a3f89b810c8f90fcc8c2" gracePeriod=30
Sep 18 16:28:59.040105 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.040077    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:28:59.073632 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.073600    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r"]
Sep 18 16:28:59.074005 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.073992    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cdefa0f-50a9-40a3-8571-c90a98531b0e" containerName="kserve-container"
Sep 18 16:28:59.074048 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.074010    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cdefa0f-50a9-40a3-8571-c90a98531b0e" containerName="kserve-container"
Sep 18 16:28:59.074048 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.074037    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cdefa0f-50a9-40a3-8571-c90a98531b0e" containerName="kube-rbac-proxy"
Sep 18 16:28:59.074048 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.074043    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cdefa0f-50a9-40a3-8571-c90a98531b0e" containerName="kube-rbac-proxy"
Sep 18 16:28:59.074180 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.074106    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cdefa0f-50a9-40a3-8571-c90a98531b0e" containerName="kserve-container"
Sep 18 16:28:59.074180 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.074122    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cdefa0f-50a9-40a3-8571-c90a98531b0e" containerName="kube-rbac-proxy"
Sep 18 16:28:59.076276 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.076262    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r"
Sep 18 16:28:59.078918 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.078900    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8d780-kube-rbac-proxy-sar-config\""
Sep 18 16:28:59.079027 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.078960    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8d780-predictor-serving-cert\""
Sep 18 16:28:59.097522 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.097497    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r"]
Sep 18 16:28:59.116591 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.116561    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk" podUID="7dd084c0-f257-44bd-8197-18bc1aa1b0d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 16:28:59.162763 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.162730    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:28:59.198494 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.198467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bd6b51b-e04b-4862-9519-14b4e16b9a70-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8d780-predictor-577499d7ff-2nw5r\" (UID: \"0bd6b51b-e04b-4862-9519-14b4e16b9a70\") " pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r"
Sep 18 16:28:59.198613 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.198496    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq7ln\" (UniqueName: \"kubernetes.io/projected/0bd6b51b-e04b-4862-9519-14b4e16b9a70-kube-api-access-pq7ln\") pod \"success-200-isvc-8d780-predictor-577499d7ff-2nw5r\" (UID: \"0bd6b51b-e04b-4862-9519-14b4e16b9a70\") " pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r"
Sep 18 16:28:59.198613 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.198542    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bd6b51b-e04b-4862-9519-14b4e16b9a70-proxy-tls\") pod \"success-200-isvc-8d780-predictor-577499d7ff-2nw5r\" (UID: \"0bd6b51b-e04b-4862-9519-14b4e16b9a70\") " pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r"
Sep 18 16:28:59.212685 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.212654    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd" podUID="656d406f-8dff-43e6-95f5-4908cfcc3179" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 18 16:28:59.278054 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.278020    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:28:59.299407 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.299379    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bd6b51b-e04b-4862-9519-14b4e16b9a70-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8d780-predictor-577499d7ff-2nw5r\" (UID: \"0bd6b51b-e04b-4862-9519-14b4e16b9a70\") " pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r"
Sep 18 16:28:59.299407 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.299408    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq7ln\" (UniqueName: \"kubernetes.io/projected/0bd6b51b-e04b-4862-9519-14b4e16b9a70-kube-api-access-pq7ln\") pod \"success-200-isvc-8d780-predictor-577499d7ff-2nw5r\" (UID: \"0bd6b51b-e04b-4862-9519-14b4e16b9a70\") " pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r"
Sep 18 16:28:59.299617 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.299539    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bd6b51b-e04b-4862-9519-14b4e16b9a70-proxy-tls\") pod \"success-200-isvc-8d780-predictor-577499d7ff-2nw5r\" (UID: \"0bd6b51b-e04b-4862-9519-14b4e16b9a70\") " pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r"
Sep 18 16:28:59.300010 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.299983    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bd6b51b-e04b-4862-9519-14b4e16b9a70-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8d780-predictor-577499d7ff-2nw5r\" (UID: \"0bd6b51b-e04b-4862-9519-14b4e16b9a70\") " pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r"
Sep 18 16:28:59.301857 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.301835    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bd6b51b-e04b-4862-9519-14b4e16b9a70-proxy-tls\") pod \"success-200-isvc-8d780-predictor-577499d7ff-2nw5r\" (UID: \"0bd6b51b-e04b-4862-9519-14b4e16b9a70\") " pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r"
Sep 18 16:28:59.308498 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.308477    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq7ln\" (UniqueName: \"kubernetes.io/projected/0bd6b51b-e04b-4862-9519-14b4e16b9a70-kube-api-access-pq7ln\") pod \"success-200-isvc-8d780-predictor-577499d7ff-2nw5r\" (UID: \"0bd6b51b-e04b-4862-9519-14b4e16b9a70\") " pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r"
Sep 18 16:28:59.374961 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.374883    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:28:59.386850 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.386824    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r"
Sep 18 16:28:59.658716 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.658692    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r"]
Sep 18 16:28:59.660604 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:28:59.660581    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd6b51b_e04b_4862_9519_14b4e16b9a70.slice/crio-79fcbc46ba05d14f95aa97ff0409188f45cbeda7df15182ac46003de948cf44c WatchSource:0}: Error finding container 79fcbc46ba05d14f95aa97ff0409188f45cbeda7df15182ac46003de948cf44c: Status 404 returned error can't find the container with id 79fcbc46ba05d14f95aa97ff0409188f45cbeda7df15182ac46003de948cf44c
Sep 18 16:28:59.717833 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.717796    2580 generic.go:356] "Generic (PLEG): container finished" podID="a3f067c6-43a9-48de-9acb-7c1fcc44389e" containerID="94633ac90032a18512ab768330bcdf61cb99e1583297390cd3991c5d7b9abaee" exitCode=2
Sep 18 16:28:59.718032 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.717836    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl" event={"ID":"a3f067c6-43a9-48de-9acb-7c1fcc44389e","Type":"ContainerDied","Data":"94633ac90032a18512ab768330bcdf61cb99e1583297390cd3991c5d7b9abaee"}
Sep 18 16:28:59.720871 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.720839    2580 generic.go:356] "Generic (PLEG): container finished" podID="656d406f-8dff-43e6-95f5-4908cfcc3179" containerID="43281943330c7b12857412b502a9b8740a7c60149be6a3f89b810c8f90fcc8c2" exitCode=2
Sep 18 16:28:59.721028 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.720910    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd" event={"ID":"656d406f-8dff-43e6-95f5-4908cfcc3179","Type":"ContainerDied","Data":"43281943330c7b12857412b502a9b8740a7c60149be6a3f89b810c8f90fcc8c2"}
Sep 18 16:28:59.722814 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.722794    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r" event={"ID":"0bd6b51b-e04b-4862-9519-14b4e16b9a70","Type":"ContainerStarted","Data":"79fcbc46ba05d14f95aa97ff0409188f45cbeda7df15182ac46003de948cf44c"}
Sep 18 16:28:59.725007 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.724988    2580 generic.go:356] "Generic (PLEG): container finished" podID="7dd084c0-f257-44bd-8197-18bc1aa1b0d9" containerID="09be7a5f914565465ee6fd943dabb49d72d4d7eb8c7ebe2ee2674198761855b2" exitCode=2
Sep 18 16:28:59.725097 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:28:59.725037    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk" event={"ID":"7dd084c0-f257-44bd-8197-18bc1aa1b0d9","Type":"ContainerDied","Data":"09be7a5f914565465ee6fd943dabb49d72d4d7eb8c7ebe2ee2674198761855b2"}
Sep 18 16:29:00.731849 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:00.731805    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r" event={"ID":"0bd6b51b-e04b-4862-9519-14b4e16b9a70","Type":"ContainerStarted","Data":"f6af535beb88788c0be0a9dd5f5641f072ef756bab514f079cafc22d91a09477"}
Sep 18 16:29:00.732240 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:00.731867    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r" event={"ID":"0bd6b51b-e04b-4862-9519-14b4e16b9a70","Type":"ContainerStarted","Data":"e7bc4fa8578fe12ee2230b3b07cdd4e89230bcec577e3b5463a66e354462285c"}
Sep 18 16:29:00.732240 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:00.731895    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r"
Sep 18 16:29:00.755230 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:00.755174    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r" podStartSLOduration=1.755156052 podStartE2EDuration="1.755156052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:29:00.753787244 +0000 UTC m=+786.842102878" watchObservedRunningTime="2026-09-18 16:29:00.755156052 +0000 UTC m=+786.843471692"
Sep 18 16:29:01.116912 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:01.116806    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk" podUID="7dd084c0-f257-44bd-8197-18bc1aa1b0d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 18 16:29:01.736484 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:01.736455    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r"
Sep 18 16:29:01.737694 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:01.737670    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r" podUID="0bd6b51b-e04b-4862-9519-14b4e16b9a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 16:29:02.122107 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:02.122000    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl" podUID="a3f067c6-43a9-48de-9acb-7c1fcc44389e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Sep 18 16:29:02.741210 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:02.741173    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r" podUID="0bd6b51b-e04b-4862-9519-14b4e16b9a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 16:29:03.118013 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:03.117916    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl" podUID="a3f067c6-43a9-48de-9acb-7c1fcc44389e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 16:29:04.212661 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.212461    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd" podUID="656d406f-8dff-43e6-95f5-4908cfcc3179" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 18 16:29:04.218984 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.218942    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd" podUID="656d406f-8dff-43e6-95f5-4908cfcc3179" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 16:29:04.381651 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.381621    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"
Sep 18 16:29:04.555572 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.555536    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-kube-rbac-proxy-sar-config\") pod \"7dd084c0-f257-44bd-8197-18bc1aa1b0d9\" (UID: \"7dd084c0-f257-44bd-8197-18bc1aa1b0d9\") "
Sep 18 16:29:04.555770 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.555580    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-kserve-provision-location\") pod \"7dd084c0-f257-44bd-8197-18bc1aa1b0d9\" (UID: \"7dd084c0-f257-44bd-8197-18bc1aa1b0d9\") "
Sep 18 16:29:04.555770 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.555620    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-cabundle-cert\") pod \"7dd084c0-f257-44bd-8197-18bc1aa1b0d9\" (UID: \"7dd084c0-f257-44bd-8197-18bc1aa1b0d9\") "
Sep 18 16:29:04.555770 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.555709    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-proxy-tls\") pod \"7dd084c0-f257-44bd-8197-18bc1aa1b0d9\" (UID: \"7dd084c0-f257-44bd-8197-18bc1aa1b0d9\") "
Sep 18 16:29:04.555770 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.555733    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49r8t\" (UniqueName: \"kubernetes.io/projected/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-kube-api-access-49r8t\") pod \"7dd084c0-f257-44bd-8197-18bc1aa1b0d9\" (UID: \"7dd084c0-f257-44bd-8197-18bc1aa1b0d9\") "
Sep 18 16:29:04.555999 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.555936    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7dd084c0-f257-44bd-8197-18bc1aa1b0d9" (UID: "7dd084c0-f257-44bd-8197-18bc1aa1b0d9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:29:04.556049 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.555995    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7dd084c0-f257-44bd-8197-18bc1aa1b0d9" (UID: "7dd084c0-f257-44bd-8197-18bc1aa1b0d9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:29:04.556049 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.556022    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-kserve-provision-location\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:29:04.556049 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.556028    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7dd084c0-f257-44bd-8197-18bc1aa1b0d9" (UID: "7dd084c0-f257-44bd-8197-18bc1aa1b0d9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:29:04.557786 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.557766    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7dd084c0-f257-44bd-8197-18bc1aa1b0d9" (UID: "7dd084c0-f257-44bd-8197-18bc1aa1b0d9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:29:04.557881 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.557867    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-kube-api-access-49r8t" (OuterVolumeSpecName: "kube-api-access-49r8t") pod "7dd084c0-f257-44bd-8197-18bc1aa1b0d9" (UID: "7dd084c0-f257-44bd-8197-18bc1aa1b0d9"). InnerVolumeSpecName "kube-api-access-49r8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:29:04.657223 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.657144    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:29:04.657223 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.657176    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-cabundle-cert\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:29:04.657223 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.657188    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-proxy-tls\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:29:04.657223 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.657196    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49r8t\" (UniqueName: \"kubernetes.io/projected/7dd084c0-f257-44bd-8197-18bc1aa1b0d9-kube-api-access-49r8t\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:29:04.755601 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.755563    2580 generic.go:356] "Generic (PLEG): container finished" podID="656d406f-8dff-43e6-95f5-4908cfcc3179" containerID="b5713c1738409dbf116eef4379e09df6fa3ec521f7c51a37cb04370a2ddbac9e" exitCode=0
Sep 18 16:29:04.755790 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.755643    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd" event={"ID":"656d406f-8dff-43e6-95f5-4908cfcc3179","Type":"ContainerDied","Data":"b5713c1738409dbf116eef4379e09df6fa3ec521f7c51a37cb04370a2ddbac9e"}
Sep 18 16:29:04.757552 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.757521    2580 generic.go:356] "Generic (PLEG): container finished" podID="7dd084c0-f257-44bd-8197-18bc1aa1b0d9" containerID="f7f68ff4eef4108a3ff613b971383b0eb8eea45fd1fc43bb7ca866b0e52acbbf" exitCode=0
Sep 18 16:29:04.757672 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.757549    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk" event={"ID":"7dd084c0-f257-44bd-8197-18bc1aa1b0d9","Type":"ContainerDied","Data":"f7f68ff4eef4108a3ff613b971383b0eb8eea45fd1fc43bb7ca866b0e52acbbf"}
Sep 18 16:29:04.757672 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.757602    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"
Sep 18 16:29:04.757672 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.757619    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk" event={"ID":"7dd084c0-f257-44bd-8197-18bc1aa1b0d9","Type":"ContainerDied","Data":"a098f6400b5316fe3751f1cff5061170ec758db726b89c08117716c2863c63b5"}
Sep 18 16:29:04.757672 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.757660    2580 scope.go:117] "RemoveContainer" containerID="09be7a5f914565465ee6fd943dabb49d72d4d7eb8c7ebe2ee2674198761855b2"
Sep 18 16:29:04.767029 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.767013    2580 scope.go:117] "RemoveContainer" containerID="f7f68ff4eef4108a3ff613b971383b0eb8eea45fd1fc43bb7ca866b0e52acbbf"
Sep 18 16:29:04.777239 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.777214    2580 scope.go:117] "RemoveContainer" containerID="c06d2dc11c7be3c4dae4f31c32d4fd7345550adffa702fe50ab5220a38d55e6f"
Sep 18 16:29:04.783598 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.783572    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"]
Sep 18 16:29:04.787932 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.787333    2580 scope.go:117] "RemoveContainer" containerID="09be7a5f914565465ee6fd943dabb49d72d4d7eb8c7ebe2ee2674198761855b2"
Sep 18 16:29:04.788167 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:29:04.788115    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09be7a5f914565465ee6fd943dabb49d72d4d7eb8c7ebe2ee2674198761855b2\": container with ID starting with 09be7a5f914565465ee6fd943dabb49d72d4d7eb8c7ebe2ee2674198761855b2 not found: ID does not exist" containerID="09be7a5f914565465ee6fd943dabb49d72d4d7eb8c7ebe2ee2674198761855b2"
Sep 18 16:29:04.788167 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.788152    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09be7a5f914565465ee6fd943dabb49d72d4d7eb8c7ebe2ee2674198761855b2"} err="failed to get container status \"09be7a5f914565465ee6fd943dabb49d72d4d7eb8c7ebe2ee2674198761855b2\": rpc error: code = NotFound desc = could not find container \"09be7a5f914565465ee6fd943dabb49d72d4d7eb8c7ebe2ee2674198761855b2\": container with ID starting with 09be7a5f914565465ee6fd943dabb49d72d4d7eb8c7ebe2ee2674198761855b2 not found: ID does not exist"
Sep 18 16:29:04.788591 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.788181    2580 scope.go:117] "RemoveContainer" containerID="f7f68ff4eef4108a3ff613b971383b0eb8eea45fd1fc43bb7ca866b0e52acbbf"
Sep 18 16:29:04.788948 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:29:04.788929    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7f68ff4eef4108a3ff613b971383b0eb8eea45fd1fc43bb7ca866b0e52acbbf\": container with ID starting with f7f68ff4eef4108a3ff613b971383b0eb8eea45fd1fc43bb7ca866b0e52acbbf not found: ID does not exist" containerID="f7f68ff4eef4108a3ff613b971383b0eb8eea45fd1fc43bb7ca866b0e52acbbf"
Sep 18 16:29:04.789022 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.788954    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7f68ff4eef4108a3ff613b971383b0eb8eea45fd1fc43bb7ca866b0e52acbbf"} err="failed to get container status \"f7f68ff4eef4108a3ff613b971383b0eb8eea45fd1fc43bb7ca866b0e52acbbf\": rpc error: code = NotFound desc = could not find container \"f7f68ff4eef4108a3ff613b971383b0eb8eea45fd1fc43bb7ca866b0e52acbbf\": container with ID starting with f7f68ff4eef4108a3ff613b971383b0eb8eea45fd1fc43bb7ca866b0e52acbbf not found: ID does not exist"
Sep 18 16:29:04.789022 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.788969    2580 scope.go:117] "RemoveContainer" containerID="c06d2dc11c7be3c4dae4f31c32d4fd7345550adffa702fe50ab5220a38d55e6f"
Sep 18 16:29:04.789285 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:29:04.789263    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c06d2dc11c7be3c4dae4f31c32d4fd7345550adffa702fe50ab5220a38d55e6f\": container with ID starting with c06d2dc11c7be3c4dae4f31c32d4fd7345550adffa702fe50ab5220a38d55e6f not found: ID does not exist" containerID="c06d2dc11c7be3c4dae4f31c32d4fd7345550adffa702fe50ab5220a38d55e6f"
Sep 18 16:29:04.789364 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.789318    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c06d2dc11c7be3c4dae4f31c32d4fd7345550adffa702fe50ab5220a38d55e6f"} err="failed to get container status \"c06d2dc11c7be3c4dae4f31c32d4fd7345550adffa702fe50ab5220a38d55e6f\": rpc error: code = NotFound desc = could not find container \"c06d2dc11c7be3c4dae4f31c32d4fd7345550adffa702fe50ab5220a38d55e6f\": container with ID starting with c06d2dc11c7be3c4dae4f31c32d4fd7345550adffa702fe50ab5220a38d55e6f not found: ID does not exist"
Sep 18 16:29:04.790508 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.790490    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-b7ccc5f99-hx7fk"]
Sep 18 16:29:04.843475 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.843446    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"
Sep 18 16:29:04.959149 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.959115    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/656d406f-8dff-43e6-95f5-4908cfcc3179-kserve-provision-location\") pod \"656d406f-8dff-43e6-95f5-4908cfcc3179\" (UID: \"656d406f-8dff-43e6-95f5-4908cfcc3179\") "
Sep 18 16:29:04.959149 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.959150    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjmm8\" (UniqueName: \"kubernetes.io/projected/656d406f-8dff-43e6-95f5-4908cfcc3179-kube-api-access-qjmm8\") pod \"656d406f-8dff-43e6-95f5-4908cfcc3179\" (UID: \"656d406f-8dff-43e6-95f5-4908cfcc3179\") "
Sep 18 16:29:04.959446 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.959191    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/656d406f-8dff-43e6-95f5-4908cfcc3179-proxy-tls\") pod \"656d406f-8dff-43e6-95f5-4908cfcc3179\" (UID: \"656d406f-8dff-43e6-95f5-4908cfcc3179\") "
Sep 18 16:29:04.959446 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.959221    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/656d406f-8dff-43e6-95f5-4908cfcc3179-cabundle-cert\") pod \"656d406f-8dff-43e6-95f5-4908cfcc3179\" (UID: \"656d406f-8dff-43e6-95f5-4908cfcc3179\") "
Sep 18 16:29:04.959446 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.959275    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/656d406f-8dff-43e6-95f5-4908cfcc3179-kube-rbac-proxy-sar-config\") pod \"656d406f-8dff-43e6-95f5-4908cfcc3179\" (UID: \"656d406f-8dff-43e6-95f5-4908cfcc3179\") "
Sep 18 16:29:04.959603 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.959551    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/656d406f-8dff-43e6-95f5-4908cfcc3179-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "656d406f-8dff-43e6-95f5-4908cfcc3179" (UID: "656d406f-8dff-43e6-95f5-4908cfcc3179"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:29:04.959658 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.959640    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/656d406f-8dff-43e6-95f5-4908cfcc3179-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "656d406f-8dff-43e6-95f5-4908cfcc3179" (UID: "656d406f-8dff-43e6-95f5-4908cfcc3179"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:29:04.959715 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.959646    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/656d406f-8dff-43e6-95f5-4908cfcc3179-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "656d406f-8dff-43e6-95f5-4908cfcc3179" (UID: "656d406f-8dff-43e6-95f5-4908cfcc3179"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:29:04.961313 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.961287    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/656d406f-8dff-43e6-95f5-4908cfcc3179-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "656d406f-8dff-43e6-95f5-4908cfcc3179" (UID: "656d406f-8dff-43e6-95f5-4908cfcc3179"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:29:04.961376 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:04.961325    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/656d406f-8dff-43e6-95f5-4908cfcc3179-kube-api-access-qjmm8" (OuterVolumeSpecName: "kube-api-access-qjmm8") pod "656d406f-8dff-43e6-95f5-4908cfcc3179" (UID: "656d406f-8dff-43e6-95f5-4908cfcc3179"). InnerVolumeSpecName "kube-api-access-qjmm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:29:05.059910 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.059876    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/656d406f-8dff-43e6-95f5-4908cfcc3179-kserve-provision-location\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:29:05.059910 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.059906    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjmm8\" (UniqueName: \"kubernetes.io/projected/656d406f-8dff-43e6-95f5-4908cfcc3179-kube-api-access-qjmm8\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:29:05.059910 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.059916    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/656d406f-8dff-43e6-95f5-4908cfcc3179-proxy-tls\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:29:05.060135 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.059925    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/656d406f-8dff-43e6-95f5-4908cfcc3179-cabundle-cert\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:29:05.060135 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.059933    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/656d406f-8dff-43e6-95f5-4908cfcc3179-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:29:05.391443 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.391417    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"
Sep 18 16:29:05.515770 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.515664    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg" podUID="3354167c-5adc-4697-a501-a3985a5aa4c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 16:29:05.564900 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.564861    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3f067c6-43a9-48de-9acb-7c1fcc44389e-proxy-tls\") pod \"a3f067c6-43a9-48de-9acb-7c1fcc44389e\" (UID: \"a3f067c6-43a9-48de-9acb-7c1fcc44389e\") "
Sep 18 16:29:05.564900 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.564906    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z67hm\" (UniqueName: \"kubernetes.io/projected/a3f067c6-43a9-48de-9acb-7c1fcc44389e-kube-api-access-z67hm\") pod \"a3f067c6-43a9-48de-9acb-7c1fcc44389e\" (UID: \"a3f067c6-43a9-48de-9acb-7c1fcc44389e\") "
Sep 18 16:29:05.565144 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.564936    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3f067c6-43a9-48de-9acb-7c1fcc44389e-kube-rbac-proxy-sar-config\") pod \"a3f067c6-43a9-48de-9acb-7c1fcc44389e\" (UID: \"a3f067c6-43a9-48de-9acb-7c1fcc44389e\") "
Sep 18 16:29:05.565144 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.564964    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3f067c6-43a9-48de-9acb-7c1fcc44389e-kserve-provision-location\") pod \"a3f067c6-43a9-48de-9acb-7c1fcc44389e\" (UID: \"a3f067c6-43a9-48de-9acb-7c1fcc44389e\") "
Sep 18 16:29:05.565144 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.565023    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3f067c6-43a9-48de-9acb-7c1fcc44389e-cabundle-cert\") pod \"a3f067c6-43a9-48de-9acb-7c1fcc44389e\" (UID: \"a3f067c6-43a9-48de-9acb-7c1fcc44389e\") "
Sep 18 16:29:05.565346 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.565286    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f067c6-43a9-48de-9acb-7c1fcc44389e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a3f067c6-43a9-48de-9acb-7c1fcc44389e" (UID: "a3f067c6-43a9-48de-9acb-7c1fcc44389e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:29:05.565406 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.565335    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3f067c6-43a9-48de-9acb-7c1fcc44389e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a3f067c6-43a9-48de-9acb-7c1fcc44389e" (UID: "a3f067c6-43a9-48de-9acb-7c1fcc44389e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:29:05.565468 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.565449    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3f067c6-43a9-48de-9acb-7c1fcc44389e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a3f067c6-43a9-48de-9acb-7c1fcc44389e" (UID: "a3f067c6-43a9-48de-9acb-7c1fcc44389e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:29:05.567103 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.567081    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3f067c6-43a9-48de-9acb-7c1fcc44389e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a3f067c6-43a9-48de-9acb-7c1fcc44389e" (UID: "a3f067c6-43a9-48de-9acb-7c1fcc44389e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:29:05.567103 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.567094    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f067c6-43a9-48de-9acb-7c1fcc44389e-kube-api-access-z67hm" (OuterVolumeSpecName: "kube-api-access-z67hm") pod "a3f067c6-43a9-48de-9acb-7c1fcc44389e" (UID: "a3f067c6-43a9-48de-9acb-7c1fcc44389e"). InnerVolumeSpecName "kube-api-access-z67hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:29:05.666390 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.666337    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3f067c6-43a9-48de-9acb-7c1fcc44389e-proxy-tls\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:29:05.666390 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.666383    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z67hm\" (UniqueName: \"kubernetes.io/projected/a3f067c6-43a9-48de-9acb-7c1fcc44389e-kube-api-access-z67hm\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:29:05.666390 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.666393    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3f067c6-43a9-48de-9acb-7c1fcc44389e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:29:05.666390 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.666401    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3f067c6-43a9-48de-9acb-7c1fcc44389e-kserve-provision-location\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:29:05.666390 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.666410    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3f067c6-43a9-48de-9acb-7c1fcc44389e-cabundle-cert\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:29:05.764378 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.764347    2580 generic.go:356] "Generic (PLEG): container finished" podID="a3f067c6-43a9-48de-9acb-7c1fcc44389e" containerID="c423881f0a26c2dcebde7569c107c160e8d74c6243b9557834376de5745f201a" exitCode=0
Sep 18 16:29:05.764557 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.764406    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"
Sep 18 16:29:05.764557 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.764442    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl" event={"ID":"a3f067c6-43a9-48de-9acb-7c1fcc44389e","Type":"ContainerDied","Data":"c423881f0a26c2dcebde7569c107c160e8d74c6243b9557834376de5745f201a"}
Sep 18 16:29:05.764557 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.764503    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl" event={"ID":"a3f067c6-43a9-48de-9acb-7c1fcc44389e","Type":"ContainerDied","Data":"c9bee4a3f1e5de7ab92984e7ffa76fd7341978dd04d5871397af4759ce646651"}
Sep 18 16:29:05.764557 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.764535    2580 scope.go:117] "RemoveContainer" containerID="94633ac90032a18512ab768330bcdf61cb99e1583297390cd3991c5d7b9abaee"
Sep 18 16:29:05.766700 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.766634    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd" event={"ID":"656d406f-8dff-43e6-95f5-4908cfcc3179","Type":"ContainerDied","Data":"b4387263d3d2b4cb420cd1d7b894c60d69f3545280a5503582452fb07a02e0b0"}
Sep 18 16:29:05.766700 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.766655    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"
Sep 18 16:29:05.774998 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.774330    2580 scope.go:117] "RemoveContainer" containerID="c423881f0a26c2dcebde7569c107c160e8d74c6243b9557834376de5745f201a"
Sep 18 16:29:05.782778 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.782756    2580 scope.go:117] "RemoveContainer" containerID="5738df0a9a84707cc617d0d76f745dc7c73dfbe1870a323b891cecb5041ea102"
Sep 18 16:29:05.789971 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.789954    2580 scope.go:117] "RemoveContainer" containerID="94633ac90032a18512ab768330bcdf61cb99e1583297390cd3991c5d7b9abaee"
Sep 18 16:29:05.790499 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:29:05.790453    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94633ac90032a18512ab768330bcdf61cb99e1583297390cd3991c5d7b9abaee\": container with ID starting with 94633ac90032a18512ab768330bcdf61cb99e1583297390cd3991c5d7b9abaee not found: ID does not exist" containerID="94633ac90032a18512ab768330bcdf61cb99e1583297390cd3991c5d7b9abaee"
Sep 18 16:29:05.790656 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.790630    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94633ac90032a18512ab768330bcdf61cb99e1583297390cd3991c5d7b9abaee"} err="failed to get container status \"94633ac90032a18512ab768330bcdf61cb99e1583297390cd3991c5d7b9abaee\": rpc error: code = NotFound desc = could not find container \"94633ac90032a18512ab768330bcdf61cb99e1583297390cd3991c5d7b9abaee\": container with ID starting with 94633ac90032a18512ab768330bcdf61cb99e1583297390cd3991c5d7b9abaee not found: ID does not exist"
Sep 18 16:29:05.790761 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.790750    2580 scope.go:117] "RemoveContainer" containerID="c423881f0a26c2dcebde7569c107c160e8d74c6243b9557834376de5745f201a"
Sep 18 16:29:05.791145 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:29:05.791125    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c423881f0a26c2dcebde7569c107c160e8d74c6243b9557834376de5745f201a\": container with ID starting with c423881f0a26c2dcebde7569c107c160e8d74c6243b9557834376de5745f201a not found: ID does not exist" containerID="c423881f0a26c2dcebde7569c107c160e8d74c6243b9557834376de5745f201a"
Sep 18 16:29:05.791229 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.791149    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c423881f0a26c2dcebde7569c107c160e8d74c6243b9557834376de5745f201a"} err="failed to get container status \"c423881f0a26c2dcebde7569c107c160e8d74c6243b9557834376de5745f201a\": rpc error: code = NotFound desc = could not find container \"c423881f0a26c2dcebde7569c107c160e8d74c6243b9557834376de5745f201a\": container with ID starting with c423881f0a26c2dcebde7569c107c160e8d74c6243b9557834376de5745f201a not found: ID does not exist"
Sep 18 16:29:05.791229 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.791167    2580 scope.go:117] "RemoveContainer" containerID="5738df0a9a84707cc617d0d76f745dc7c73dfbe1870a323b891cecb5041ea102"
Sep 18 16:29:05.791802 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:29:05.791438    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5738df0a9a84707cc617d0d76f745dc7c73dfbe1870a323b891cecb5041ea102\": container with ID starting with 5738df0a9a84707cc617d0d76f745dc7c73dfbe1870a323b891cecb5041ea102 not found: ID does not exist" containerID="5738df0a9a84707cc617d0d76f745dc7c73dfbe1870a323b891cecb5041ea102"
Sep 18 16:29:05.791802 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.791458    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5738df0a9a84707cc617d0d76f745dc7c73dfbe1870a323b891cecb5041ea102"} err="failed to get container status \"5738df0a9a84707cc617d0d76f745dc7c73dfbe1870a323b891cecb5041ea102\": rpc error: code = NotFound desc = could not find container \"5738df0a9a84707cc617d0d76f745dc7c73dfbe1870a323b891cecb5041ea102\": container with ID starting with 5738df0a9a84707cc617d0d76f745dc7c73dfbe1870a323b891cecb5041ea102 not found: ID does not exist"
Sep 18 16:29:05.791802 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.791477    2580 scope.go:117] "RemoveContainer" containerID="43281943330c7b12857412b502a9b8740a7c60149be6a3f89b810c8f90fcc8c2"
Sep 18 16:29:05.793116 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.793095    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"]
Sep 18 16:29:05.798157 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.798137    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-66bf4f5cc4-lj7dl"]
Sep 18 16:29:05.799581 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.799564    2580 scope.go:117] "RemoveContainer" containerID="b5713c1738409dbf116eef4379e09df6fa3ec521f7c51a37cb04370a2ddbac9e"
Sep 18 16:29:05.806806 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.806789    2580 scope.go:117] "RemoveContainer" containerID="772b37a5fa8b37d69ea46b40b9d14b7cb1d73120196c41b54ce53e18b70eb845"
Sep 18 16:29:05.812645 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.812542    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"]
Sep 18 16:29:05.815050 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:05.815029    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-jp7sd"]
Sep 18 16:29:06.123798 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:06.123726    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="656d406f-8dff-43e6-95f5-4908cfcc3179" path="/var/lib/kubelet/pods/656d406f-8dff-43e6-95f5-4908cfcc3179/volumes"
Sep 18 16:29:06.124222 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:06.124210    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dd084c0-f257-44bd-8197-18bc1aa1b0d9" path="/var/lib/kubelet/pods/7dd084c0-f257-44bd-8197-18bc1aa1b0d9/volumes"
Sep 18 16:29:06.124987 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:06.124972    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3f067c6-43a9-48de-9acb-7c1fcc44389e" path="/var/lib/kubelet/pods/a3f067c6-43a9-48de-9acb-7c1fcc44389e/volumes"
Sep 18 16:29:07.746446 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:07.746420    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r"
Sep 18 16:29:07.747036 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:07.747012    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r" podUID="0bd6b51b-e04b-4862-9519-14b4e16b9a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 16:29:15.516329 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:15.516272    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg"
Sep 18 16:29:17.747572 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:17.747529    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r" podUID="0bd6b51b-e04b-4862-9519-14b4e16b9a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 16:29:27.747942 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:27.747898    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r" podUID="0bd6b51b-e04b-4862-9519-14b4e16b9a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 16:29:28.594385 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:28.594354    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:29:29.150681 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:29.150640    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:29:37.747104 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:37.747067    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r" podUID="0bd6b51b-e04b-4862-9519-14b4e16b9a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 16:29:43.129116 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:43.129080    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:29:47.747423 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:47.747389    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r" podUID="0bd6b51b-e04b-4862-9519-14b4e16b9a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 16:29:57.748349 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:29:57.748314    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r"
Sep 18 16:30:09.029856 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:30:09.029823    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:30:09.149801 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:30:09.149773    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:28.718403 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:31:28.718359    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:30.158312 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:31:30.158271    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:35:20.862831 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:35:20.862748    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:37:43.455768 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.455734    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:37:43.530409 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.530373    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg"]
Sep 18 16:37:43.530699 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.530669    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg" podUID="3354167c-5adc-4697-a501-a3985a5aa4c6" containerName="kserve-container" containerID="cri-o://4239f5ec148950d9b4fe1b0c7da1457bde9c4958c89c30dd0d25bf686ef6c6df" gracePeriod=30
Sep 18 16:37:43.530835 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.530703    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg" podUID="3354167c-5adc-4697-a501-a3985a5aa4c6" containerName="kube-rbac-proxy" containerID="cri-o://ad18fce5db1de1cc61f329ff4970912697d5fd8308c9601355c69585cd3f4e74" gracePeriod=30
Sep 18 16:37:43.674450 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.674417    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb"]
Sep 18 16:37:43.674837 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.674823    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dd084c0-f257-44bd-8197-18bc1aa1b0d9" containerName="kube-rbac-proxy"
Sep 18 16:37:43.674889 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.674839    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dd084c0-f257-44bd-8197-18bc1aa1b0d9" containerName="kube-rbac-proxy"
Sep 18 16:37:43.674889 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.674849    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3f067c6-43a9-48de-9acb-7c1fcc44389e" containerName="kube-rbac-proxy"
Sep 18 16:37:43.674889 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.674855    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3f067c6-43a9-48de-9acb-7c1fcc44389e" containerName="kube-rbac-proxy"
Sep 18 16:37:43.674889 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.674871    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3f067c6-43a9-48de-9acb-7c1fcc44389e" containerName="storage-initializer"
Sep 18 16:37:43.674889 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.674878    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3f067c6-43a9-48de-9acb-7c1fcc44389e" containerName="storage-initializer"
Sep 18 16:37:43.674889 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.674884    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="656d406f-8dff-43e6-95f5-4908cfcc3179" containerName="storage-initializer"
Sep 18 16:37:43.674889 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.674890    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="656d406f-8dff-43e6-95f5-4908cfcc3179" containerName="storage-initializer"
Sep 18 16:37:43.675103 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.674896    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3f067c6-43a9-48de-9acb-7c1fcc44389e" containerName="kserve-container"
Sep 18 16:37:43.675103 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.674902    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3f067c6-43a9-48de-9acb-7c1fcc44389e" containerName="kserve-container"
Sep 18 16:37:43.675103 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.674908    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="656d406f-8dff-43e6-95f5-4908cfcc3179" containerName="kserve-container"
Sep 18 16:37:43.675103 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.674914    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="656d406f-8dff-43e6-95f5-4908cfcc3179" containerName="kserve-container"
Sep 18 16:37:43.675103 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.674921    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="656d406f-8dff-43e6-95f5-4908cfcc3179" containerName="kube-rbac-proxy"
Sep 18 16:37:43.675103 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.674926    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="656d406f-8dff-43e6-95f5-4908cfcc3179" containerName="kube-rbac-proxy"
Sep 18 16:37:43.675103 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.674939    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dd084c0-f257-44bd-8197-18bc1aa1b0d9" containerName="storage-initializer"
Sep 18 16:37:43.675103 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.674944    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dd084c0-f257-44bd-8197-18bc1aa1b0d9" containerName="storage-initializer"
Sep 18 16:37:43.675103 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.674951    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dd084c0-f257-44bd-8197-18bc1aa1b0d9" containerName="kserve-container"
Sep 18 16:37:43.675103 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.674956    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dd084c0-f257-44bd-8197-18bc1aa1b0d9" containerName="kserve-container"
Sep 18 16:37:43.675103 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.675009    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dd084c0-f257-44bd-8197-18bc1aa1b0d9" containerName="kube-rbac-proxy"
Sep 18 16:37:43.675103 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.675018    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3f067c6-43a9-48de-9acb-7c1fcc44389e" containerName="kserve-container"
Sep 18 16:37:43.675103 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.675024    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3f067c6-43a9-48de-9acb-7c1fcc44389e" containerName="kube-rbac-proxy"
Sep 18 16:37:43.675103 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.675030    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dd084c0-f257-44bd-8197-18bc1aa1b0d9" containerName="kserve-container"
Sep 18 16:37:43.675103 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.675035    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="656d406f-8dff-43e6-95f5-4908cfcc3179" containerName="kube-rbac-proxy"
Sep 18 16:37:43.675103 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.675042    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="656d406f-8dff-43e6-95f5-4908cfcc3179" containerName="kserve-container"
Sep 18 16:37:43.678128 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.678108    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb"
Sep 18 16:37:43.681207 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.681185    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6304f-predictor-serving-cert\""
Sep 18 16:37:43.681366 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.681219    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6304f-kube-rbac-proxy-sar-config\""
Sep 18 16:37:43.687476 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.687452    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb"]
Sep 18 16:37:43.821888 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.821807    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/613372f2-24b2-4f97-950d-198bc835eeb1-proxy-tls\") pod \"success-200-isvc-6304f-predictor-9c8948987-8lbnb\" (UID: \"613372f2-24b2-4f97-950d-198bc835eeb1\") " pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb"
Sep 18 16:37:43.822031 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.822009    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/613372f2-24b2-4f97-950d-198bc835eeb1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6304f-predictor-9c8948987-8lbnb\" (UID: \"613372f2-24b2-4f97-950d-198bc835eeb1\") " pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb"
Sep 18 16:37:43.822073 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.822055    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm9ls\" (UniqueName: \"kubernetes.io/projected/613372f2-24b2-4f97-950d-198bc835eeb1-kube-api-access-gm9ls\") pod \"success-200-isvc-6304f-predictor-9c8948987-8lbnb\" (UID: \"613372f2-24b2-4f97-950d-198bc835eeb1\") " pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb"
Sep 18 16:37:43.881952 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.881919    2580 generic.go:356] "Generic (PLEG): container finished" podID="3354167c-5adc-4697-a501-a3985a5aa4c6" containerID="ad18fce5db1de1cc61f329ff4970912697d5fd8308c9601355c69585cd3f4e74" exitCode=2
Sep 18 16:37:43.882104 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.881974    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg" event={"ID":"3354167c-5adc-4697-a501-a3985a5aa4c6","Type":"ContainerDied","Data":"ad18fce5db1de1cc61f329ff4970912697d5fd8308c9601355c69585cd3f4e74"}
Sep 18 16:37:43.922700 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.922667    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm9ls\" (UniqueName: \"kubernetes.io/projected/613372f2-24b2-4f97-950d-198bc835eeb1-kube-api-access-gm9ls\") pod \"success-200-isvc-6304f-predictor-9c8948987-8lbnb\" (UID: \"613372f2-24b2-4f97-950d-198bc835eeb1\") " pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb"
Sep 18 16:37:43.922874 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.922734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/613372f2-24b2-4f97-950d-198bc835eeb1-proxy-tls\") pod \"success-200-isvc-6304f-predictor-9c8948987-8lbnb\" (UID: \"613372f2-24b2-4f97-950d-198bc835eeb1\") " pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb"
Sep 18 16:37:43.922874 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.922814    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/613372f2-24b2-4f97-950d-198bc835eeb1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6304f-predictor-9c8948987-8lbnb\" (UID: \"613372f2-24b2-4f97-950d-198bc835eeb1\") " pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb"
Sep 18 16:37:43.922994 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:37:43.922910    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-6304f-predictor-serving-cert: secret "success-200-isvc-6304f-predictor-serving-cert" not found
Sep 18 16:37:43.922994 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:37:43.922983    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/613372f2-24b2-4f97-950d-198bc835eeb1-proxy-tls podName:613372f2-24b2-4f97-950d-198bc835eeb1 nodeName:}" failed. No retries permitted until 2026-09-18 16:37:44.422959936 +0000 UTC m=+1310.511275570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/613372f2-24b2-4f97-950d-198bc835eeb1-proxy-tls") pod "success-200-isvc-6304f-predictor-9c8948987-8lbnb" (UID: "613372f2-24b2-4f97-950d-198bc835eeb1") : secret "success-200-isvc-6304f-predictor-serving-cert" not found
Sep 18 16:37:43.923521 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.923503    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/613372f2-24b2-4f97-950d-198bc835eeb1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6304f-predictor-9c8948987-8lbnb\" (UID: \"613372f2-24b2-4f97-950d-198bc835eeb1\") " pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb"
Sep 18 16:37:43.931337 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:43.931295    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm9ls\" (UniqueName: \"kubernetes.io/projected/613372f2-24b2-4f97-950d-198bc835eeb1-kube-api-access-gm9ls\") pod \"success-200-isvc-6304f-predictor-9c8948987-8lbnb\" (UID: \"613372f2-24b2-4f97-950d-198bc835eeb1\") " pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb"
Sep 18 16:37:44.428950 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:44.428913    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/613372f2-24b2-4f97-950d-198bc835eeb1-proxy-tls\") pod \"success-200-isvc-6304f-predictor-9c8948987-8lbnb\" (UID: \"613372f2-24b2-4f97-950d-198bc835eeb1\") " pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb"
Sep 18 16:37:44.429128 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:37:44.429054    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-6304f-predictor-serving-cert: secret "success-200-isvc-6304f-predictor-serving-cert" not found
Sep 18 16:37:44.429128 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:37:44.429116    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/613372f2-24b2-4f97-950d-198bc835eeb1-proxy-tls podName:613372f2-24b2-4f97-950d-198bc835eeb1 nodeName:}" failed. No retries permitted until 2026-09-18 16:37:45.42910066 +0000 UTC m=+1311.517416278 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/613372f2-24b2-4f97-950d-198bc835eeb1-proxy-tls") pod "success-200-isvc-6304f-predictor-9c8948987-8lbnb" (UID: "613372f2-24b2-4f97-950d-198bc835eeb1") : secret "success-200-isvc-6304f-predictor-serving-cert" not found
Sep 18 16:37:44.658353 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:44.658323    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:37:45.438869 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:45.438829    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/613372f2-24b2-4f97-950d-198bc835eeb1-proxy-tls\") pod \"success-200-isvc-6304f-predictor-9c8948987-8lbnb\" (UID: \"613372f2-24b2-4f97-950d-198bc835eeb1\") " pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb"
Sep 18 16:37:45.441289 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:45.441254    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/613372f2-24b2-4f97-950d-198bc835eeb1-proxy-tls\") pod \"success-200-isvc-6304f-predictor-9c8948987-8lbnb\" (UID: \"613372f2-24b2-4f97-950d-198bc835eeb1\") " pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb"
Sep 18 16:37:45.488789 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:45.488752    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb"
Sep 18 16:37:45.510695 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:45.510661    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg" podUID="3354167c-5adc-4697-a501-a3985a5aa4c6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 18 16:37:45.515602 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:45.515570    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg" podUID="3354167c-5adc-4697-a501-a3985a5aa4c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 16:37:45.760029 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:45.760001    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb"]
Sep 18 16:37:45.762103 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:37:45.762064    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613372f2_24b2_4f97_950d_198bc835eeb1.slice/crio-16ffb7c85ff5c4cdd3a611c343a830d3c32f213c1fd21e273e80e2a84c74938f WatchSource:0}: Error finding container 16ffb7c85ff5c4cdd3a611c343a830d3c32f213c1fd21e273e80e2a84c74938f: Status 404 returned error can't find the container with id 16ffb7c85ff5c4cdd3a611c343a830d3c32f213c1fd21e273e80e2a84c74938f
Sep 18 16:37:45.891970 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:45.891943    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb" event={"ID":"613372f2-24b2-4f97-950d-198bc835eeb1","Type":"ContainerStarted","Data":"e715edfc756b9d49d9813d1b8b2a0b3f40d7525f041be54f9d67a0daa4e879a0"}
Sep 18 16:37:45.892066 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:45.892004    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb" event={"ID":"613372f2-24b2-4f97-950d-198bc835eeb1","Type":"ContainerStarted","Data":"16ffb7c85ff5c4cdd3a611c343a830d3c32f213c1fd21e273e80e2a84c74938f"}
Sep 18 16:37:46.898829 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:46.898791    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb" event={"ID":"613372f2-24b2-4f97-950d-198bc835eeb1","Type":"ContainerStarted","Data":"6aa962c36ffe3c88e19f70d3c4ee5ce53e27a8f234b7cd3df2a0ccaf4c2ba46b"}
Sep 18 16:37:46.899420 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:46.899342    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb"
Sep 18 16:37:46.899503 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:46.899487    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb"
Sep 18 16:37:46.900880 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:46.900852    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb" podUID="613372f2-24b2-4f97-950d-198bc835eeb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 16:37:46.901500 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:46.901480    2580 generic.go:356] "Generic (PLEG): container finished" podID="3354167c-5adc-4697-a501-a3985a5aa4c6" containerID="4239f5ec148950d9b4fe1b0c7da1457bde9c4958c89c30dd0d25bf686ef6c6df" exitCode=0
Sep 18 16:37:46.901614 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:46.901551    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg" event={"ID":"3354167c-5adc-4697-a501-a3985a5aa4c6","Type":"ContainerDied","Data":"4239f5ec148950d9b4fe1b0c7da1457bde9c4958c89c30dd0d25bf686ef6c6df"}
Sep 18 16:37:46.920810 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:46.920745    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb" podStartSLOduration=3.920721419 podStartE2EDuration="3.920721419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:37:46.918261525 +0000 UTC m=+1313.006577167" watchObservedRunningTime="2026-09-18 16:37:46.920721419 +0000 UTC m=+1313.009037060"
Sep 18 16:37:46.986665 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:46.986641    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg"
Sep 18 16:37:47.157497 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:47.157406    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3354167c-5adc-4697-a501-a3985a5aa4c6-kube-rbac-proxy-sar-config\") pod \"3354167c-5adc-4697-a501-a3985a5aa4c6\" (UID: \"3354167c-5adc-4697-a501-a3985a5aa4c6\") "
Sep 18 16:37:47.157497 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:47.157469    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpdj8\" (UniqueName: \"kubernetes.io/projected/3354167c-5adc-4697-a501-a3985a5aa4c6-kube-api-access-jpdj8\") pod \"3354167c-5adc-4697-a501-a3985a5aa4c6\" (UID: \"3354167c-5adc-4697-a501-a3985a5aa4c6\") "
Sep 18 16:37:47.157711 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:47.157534    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3354167c-5adc-4697-a501-a3985a5aa4c6-proxy-tls\") pod \"3354167c-5adc-4697-a501-a3985a5aa4c6\" (UID: \"3354167c-5adc-4697-a501-a3985a5aa4c6\") "
Sep 18 16:37:47.157790 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:47.157768    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3354167c-5adc-4697-a501-a3985a5aa4c6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3354167c-5adc-4697-a501-a3985a5aa4c6" (UID: "3354167c-5adc-4697-a501-a3985a5aa4c6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:37:47.159680 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:47.159646    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3354167c-5adc-4697-a501-a3985a5aa4c6-kube-api-access-jpdj8" (OuterVolumeSpecName: "kube-api-access-jpdj8") pod "3354167c-5adc-4697-a501-a3985a5aa4c6" (UID: "3354167c-5adc-4697-a501-a3985a5aa4c6"). InnerVolumeSpecName "kube-api-access-jpdj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:37:47.159680 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:47.159649    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3354167c-5adc-4697-a501-a3985a5aa4c6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3354167c-5adc-4697-a501-a3985a5aa4c6" (UID: "3354167c-5adc-4697-a501-a3985a5aa4c6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:37:47.258279 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:47.258240    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3354167c-5adc-4697-a501-a3985a5aa4c6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:37:47.258279 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:47.258272    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpdj8\" (UniqueName: \"kubernetes.io/projected/3354167c-5adc-4697-a501-a3985a5aa4c6-kube-api-access-jpdj8\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:37:47.258279 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:47.258284    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3354167c-5adc-4697-a501-a3985a5aa4c6-proxy-tls\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:37:47.907231 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:47.907192    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg" event={"ID":"3354167c-5adc-4697-a501-a3985a5aa4c6","Type":"ContainerDied","Data":"6376207f8860cecd5bcd01e5b134ff7ba925ad887f02a35ac5c63c94c594446c"}
Sep 18 16:37:47.907231 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:47.907227    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg"
Sep 18 16:37:47.907731 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:47.907271    2580 scope.go:117] "RemoveContainer" containerID="ad18fce5db1de1cc61f329ff4970912697d5fd8308c9601355c69585cd3f4e74"
Sep 18 16:37:47.907731 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:47.907495    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb" podUID="613372f2-24b2-4f97-950d-198bc835eeb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 16:37:47.916416 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:47.916400    2580 scope.go:117] "RemoveContainer" containerID="4239f5ec148950d9b4fe1b0c7da1457bde9c4958c89c30dd0d25bf686ef6c6df"
Sep 18 16:37:47.930937 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:47.930911    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg"]
Sep 18 16:37:47.937418 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:47.937397    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8aa04-predictor-b58b89b85-5tmrg"]
Sep 18 16:37:48.122213 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:48.122172    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3354167c-5adc-4697-a501-a3985a5aa4c6" path="/var/lib/kubelet/pods/3354167c-5adc-4697-a501-a3985a5aa4c6/volumes"
Sep 18 16:37:52.911821 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:52.911790    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb"
Sep 18 16:37:52.912436 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:37:52.912410    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb" podUID="613372f2-24b2-4f97-950d-198bc835eeb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 16:38:02.913347 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:02.913281    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb" podUID="613372f2-24b2-4f97-950d-198bc835eeb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 16:38:12.912914 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:12.912829    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb" podUID="613372f2-24b2-4f97-950d-198bc835eeb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 16:38:19.271278 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:19.271251    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:38:22.913148 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:22.913108    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb" podUID="613372f2-24b2-4f97-950d-198bc835eeb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 16:38:24.537613 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:24.537586    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:38:24.642142 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:24.642111    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r"]
Sep 18 16:38:24.642388 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:24.642354    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r" podUID="0bd6b51b-e04b-4862-9519-14b4e16b9a70" containerName="kserve-container" containerID="cri-o://f6af535beb88788c0be0a9dd5f5641f072ef756bab514f079cafc22d91a09477" gracePeriod=30
Sep 18 16:38:24.642526 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:24.642427    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r" podUID="0bd6b51b-e04b-4862-9519-14b4e16b9a70" containerName="kube-rbac-proxy" containerID="cri-o://e7bc4fa8578fe12ee2230b3b07cdd4e89230bcec577e3b5463a66e354462285c" gracePeriod=30
Sep 18 16:38:24.668782 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:24.668754    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:38:24.748494 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:24.748436    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:38:24.749163 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:24.749138    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f"]
Sep 18 16:38:24.749594 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:24.749578    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3354167c-5adc-4697-a501-a3985a5aa4c6" containerName="kube-rbac-proxy"
Sep 18 16:38:24.749687 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:24.749597    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3354167c-5adc-4697-a501-a3985a5aa4c6" containerName="kube-rbac-proxy"
Sep 18 16:38:24.749687 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:24.749612    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3354167c-5adc-4697-a501-a3985a5aa4c6" containerName="kserve-container"
Sep 18 16:38:24.749687 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:24.749621    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3354167c-5adc-4697-a501-a3985a5aa4c6" containerName="kserve-container"
Sep 18 16:38:24.749849 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:24.749707    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3354167c-5adc-4697-a501-a3985a5aa4c6" containerName="kube-rbac-proxy"
Sep 18 16:38:24.749849 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:24.749722    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3354167c-5adc-4697-a501-a3985a5aa4c6" containerName="kserve-container"
Sep 18 16:38:24.752880 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:24.752861    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f"
Sep 18 16:38:24.755255 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:24.755238    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7c9ed-predictor-serving-cert\""
Sep 18 16:38:24.755365 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:24.755283    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7c9ed-kube-rbac-proxy-sar-config\""
Sep 18 16:38:24.762088 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:24.762070    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f"]
Sep 18 16:38:24.864630 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:24.864555    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48zb9\" (UniqueName: \"kubernetes.io/projected/630df95e-f477-4d96-85a8-e21e8ceff156-kube-api-access-48zb9\") pod \"success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f\" (UID: \"630df95e-f477-4d96-85a8-e21e8ceff156\") " pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f"
Sep 18 16:38:24.864630 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:24.864618    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/630df95e-f477-4d96-85a8-e21e8ceff156-proxy-tls\") pod \"success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f\" (UID: \"630df95e-f477-4d96-85a8-e21e8ceff156\") " pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f"
Sep 18 16:38:24.864832 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:24.864701    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/630df95e-f477-4d96-85a8-e21e8ceff156-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f\" (UID: \"630df95e-f477-4d96-85a8-e21e8ceff156\") " pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f"
Sep 18 16:38:24.909455 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:24.909429    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:38:24.965288 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:24.965245    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/630df95e-f477-4d96-85a8-e21e8ceff156-proxy-tls\") pod \"success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f\" (UID: \"630df95e-f477-4d96-85a8-e21e8ceff156\") " pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f"
Sep 18 16:38:24.965448 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:24.965296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/630df95e-f477-4d96-85a8-e21e8ceff156-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f\" (UID: \"630df95e-f477-4d96-85a8-e21e8ceff156\") " pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f"
Sep 18 16:38:24.965448 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:24.965425    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48zb9\" (UniqueName: \"kubernetes.io/projected/630df95e-f477-4d96-85a8-e21e8ceff156-kube-api-access-48zb9\") pod \"success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f\" (UID: \"630df95e-f477-4d96-85a8-e21e8ceff156\") " pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f"
Sep 18 16:38:24.966165 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:24.966140    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/630df95e-f477-4d96-85a8-e21e8ceff156-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f\" (UID: \"630df95e-f477-4d96-85a8-e21e8ceff156\") " pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f"
Sep 18 16:38:24.967949 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:24.967917    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/630df95e-f477-4d96-85a8-e21e8ceff156-proxy-tls\") pod \"success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f\" (UID: \"630df95e-f477-4d96-85a8-e21e8ceff156\") " pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f"
Sep 18 16:38:24.973867 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:24.973840    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48zb9\" (UniqueName: \"kubernetes.io/projected/630df95e-f477-4d96-85a8-e21e8ceff156-kube-api-access-48zb9\") pod \"success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f\" (UID: \"630df95e-f477-4d96-85a8-e21e8ceff156\") " pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f"
Sep 18 16:38:25.001080 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:25.001059    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:38:25.063192 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:25.063167    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f"
Sep 18 16:38:25.068767 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:25.068746    2580 generic.go:356] "Generic (PLEG): container finished" podID="0bd6b51b-e04b-4862-9519-14b4e16b9a70" containerID="e7bc4fa8578fe12ee2230b3b07cdd4e89230bcec577e3b5463a66e354462285c" exitCode=2
Sep 18 16:38:25.068859 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:25.068778    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r" event={"ID":"0bd6b51b-e04b-4862-9519-14b4e16b9a70","Type":"ContainerDied","Data":"e7bc4fa8578fe12ee2230b3b07cdd4e89230bcec577e3b5463a66e354462285c"}
Sep 18 16:38:25.302380 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:25.302345    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:38:25.336132 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:25.336089    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f"]
Sep 18 16:38:25.338347 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:38:25.338317    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630df95e_f477_4d96_85a8_e21e8ceff156.slice/crio-24b8fc6fa60793ccae6f8999178c86f0f7a5562a0beb046d6cf16447955d6df6 WatchSource:0}: Error finding container 24b8fc6fa60793ccae6f8999178c86f0f7a5562a0beb046d6cf16447955d6df6: Status 404 returned error can't find the container with id 24b8fc6fa60793ccae6f8999178c86f0f7a5562a0beb046d6cf16447955d6df6
Sep 18 16:38:26.075435 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:26.075401    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f" event={"ID":"630df95e-f477-4d96-85a8-e21e8ceff156","Type":"ContainerStarted","Data":"8aa079c116649dfe69035734054138b7264d6ace20e3c9f8015259a3be11442f"}
Sep 18 16:38:26.075809 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:26.075456    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f" event={"ID":"630df95e-f477-4d96-85a8-e21e8ceff156","Type":"ContainerStarted","Data":"847d907863232b0a9bbf47a4db6dedf82547f875878202b143822926e13e70a7"}
Sep 18 16:38:26.075809 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:26.075477    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f" event={"ID":"630df95e-f477-4d96-85a8-e21e8ceff156","Type":"ContainerStarted","Data":"24b8fc6fa60793ccae6f8999178c86f0f7a5562a0beb046d6cf16447955d6df6"}
Sep 18 16:38:26.075809 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:26.075511    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f"
Sep 18 16:38:26.096433 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:26.096388    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f" podStartSLOduration=2.096373458 podStartE2EDuration="2.096373458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:38:26.09454237 +0000 UTC m=+1352.182858002" watchObservedRunningTime="2026-09-18 16:38:26.096373458 +0000 UTC m=+1352.184689097"
Sep 18 16:38:27.079450 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:27.079410    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f"
Sep 18 16:38:27.080616 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:27.080589    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f" podUID="630df95e-f477-4d96-85a8-e21e8ceff156" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 16:38:27.742090 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:27.742050    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r" podUID="0bd6b51b-e04b-4862-9519-14b4e16b9a70" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 18 16:38:27.747501 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:27.747478    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r" podUID="0bd6b51b-e04b-4862-9519-14b4e16b9a70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 16:38:28.085175 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:28.085145    2580 generic.go:356] "Generic (PLEG): container finished" podID="0bd6b51b-e04b-4862-9519-14b4e16b9a70" containerID="f6af535beb88788c0be0a9dd5f5641f072ef756bab514f079cafc22d91a09477" exitCode=0
Sep 18 16:38:28.085577 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:28.085220    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r" event={"ID":"0bd6b51b-e04b-4862-9519-14b4e16b9a70","Type":"ContainerDied","Data":"f6af535beb88788c0be0a9dd5f5641f072ef756bab514f079cafc22d91a09477"}
Sep 18 16:38:28.085577 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:28.085270    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r" event={"ID":"0bd6b51b-e04b-4862-9519-14b4e16b9a70","Type":"ContainerDied","Data":"79fcbc46ba05d14f95aa97ff0409188f45cbeda7df15182ac46003de948cf44c"}
Sep 18 16:38:28.085577 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:28.085319    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79fcbc46ba05d14f95aa97ff0409188f45cbeda7df15182ac46003de948cf44c"
Sep 18 16:38:28.085577 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:28.085534    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f" podUID="630df95e-f477-4d96-85a8-e21e8ceff156" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 16:38:28.085831 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:28.085812    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r"
Sep 18 16:38:28.190876 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:28.190846    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pq7ln\" (UniqueName: \"kubernetes.io/projected/0bd6b51b-e04b-4862-9519-14b4e16b9a70-kube-api-access-pq7ln\") pod \"0bd6b51b-e04b-4862-9519-14b4e16b9a70\" (UID: \"0bd6b51b-e04b-4862-9519-14b4e16b9a70\") "
Sep 18 16:38:28.191028 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:28.190891    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bd6b51b-e04b-4862-9519-14b4e16b9a70-kube-rbac-proxy-sar-config\") pod \"0bd6b51b-e04b-4862-9519-14b4e16b9a70\" (UID: \"0bd6b51b-e04b-4862-9519-14b4e16b9a70\") "
Sep 18 16:38:28.191028 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:28.190931    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bd6b51b-e04b-4862-9519-14b4e16b9a70-proxy-tls\") pod \"0bd6b51b-e04b-4862-9519-14b4e16b9a70\" (UID: \"0bd6b51b-e04b-4862-9519-14b4e16b9a70\") "
Sep 18 16:38:28.191324 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:28.191270    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bd6b51b-e04b-4862-9519-14b4e16b9a70-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0bd6b51b-e04b-4862-9519-14b4e16b9a70" (UID: "0bd6b51b-e04b-4862-9519-14b4e16b9a70"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:38:28.191618 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:28.191590    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bd6b51b-e04b-4862-9519-14b4e16b9a70-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:38:28.193468 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:28.193448    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd6b51b-e04b-4862-9519-14b4e16b9a70-kube-api-access-pq7ln" (OuterVolumeSpecName: "kube-api-access-pq7ln") pod "0bd6b51b-e04b-4862-9519-14b4e16b9a70" (UID: "0bd6b51b-e04b-4862-9519-14b4e16b9a70"). InnerVolumeSpecName "kube-api-access-pq7ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:38:28.193562 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:28.193449    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bd6b51b-e04b-4862-9519-14b4e16b9a70-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0bd6b51b-e04b-4862-9519-14b4e16b9a70" (UID: "0bd6b51b-e04b-4862-9519-14b4e16b9a70"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:38:28.292683 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:28.292602    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bd6b51b-e04b-4862-9519-14b4e16b9a70-proxy-tls\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:38:28.292683 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:28.292641    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq7ln\" (UniqueName: \"kubernetes.io/projected/0bd6b51b-e04b-4862-9519-14b4e16b9a70-kube-api-access-pq7ln\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:38:29.089707 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:29.089676    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r"
Sep 18 16:38:29.110078 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:29.110050    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r"]
Sep 18 16:38:29.112556 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:29.112531    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8d780-predictor-577499d7ff-2nw5r"]
Sep 18 16:38:30.123532 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:30.123503    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bd6b51b-e04b-4862-9519-14b4e16b9a70" path="/var/lib/kubelet/pods/0bd6b51b-e04b-4862-9519-14b4e16b9a70/volumes"
Sep 18 16:38:32.912641 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:32.912604    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb" podUID="613372f2-24b2-4f97-950d-198bc835eeb1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 16:38:33.094786 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:33.094757    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f"
Sep 18 16:38:33.095314 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:33.095269    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f" podUID="630df95e-f477-4d96-85a8-e21e8ceff156" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 16:38:42.913294 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:42.913266    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb"
Sep 18 16:38:43.096084 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:43.096042    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f" podUID="630df95e-f477-4d96-85a8-e21e8ceff156" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 16:38:53.095790 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:53.095749    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f" podUID="630df95e-f477-4d96-85a8-e21e8ceff156" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 16:38:53.668155 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:53.664531    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:38:53.953879 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:53.953850    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:38:56.115069 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:56.115036    2580 scope.go:117] "RemoveContainer" containerID="f6af535beb88788c0be0a9dd5f5641f072ef756bab514f079cafc22d91a09477"
Sep 18 16:38:56.124706 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:38:56.124687    2580 scope.go:117] "RemoveContainer" containerID="e7bc4fa8578fe12ee2230b3b07cdd4e89230bcec577e3b5463a66e354462285c"
Sep 18 16:39:03.095282 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:03.095241    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f" podUID="630df95e-f477-4d96-85a8-e21e8ceff156" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 16:39:04.273979 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.273940    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:04.348711 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.348668    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb"]
Sep 18 16:39:04.348932 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.348906    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb" podUID="613372f2-24b2-4f97-950d-198bc835eeb1" containerName="kserve-container" containerID="cri-o://e715edfc756b9d49d9813d1b8b2a0b3f40d7525f041be54f9d67a0daa4e879a0" gracePeriod=30
Sep 18 16:39:04.349001 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.348931    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb" podUID="613372f2-24b2-4f97-950d-198bc835eeb1" containerName="kube-rbac-proxy" containerID="cri-o://6aa962c36ffe3c88e19f70d3c4ee5ce53e27a8f234b7cd3df2a0ccaf4c2ba46b" gracePeriod=30
Sep 18 16:39:04.382537 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.382503    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:04.430778 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.430747    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x"]
Sep 18 16:39:04.431116 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.431105    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bd6b51b-e04b-4862-9519-14b4e16b9a70" containerName="kube-rbac-proxy"
Sep 18 16:39:04.431161 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.431118    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bd6b51b-e04b-4862-9519-14b4e16b9a70" containerName="kube-rbac-proxy"
Sep 18 16:39:04.431161 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.431128    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bd6b51b-e04b-4862-9519-14b4e16b9a70" containerName="kserve-container"
Sep 18 16:39:04.431161 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.431134    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bd6b51b-e04b-4862-9519-14b4e16b9a70" containerName="kserve-container"
Sep 18 16:39:04.431267 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.431199    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bd6b51b-e04b-4862-9519-14b4e16b9a70" containerName="kube-rbac-proxy"
Sep 18 16:39:04.431267 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.431207    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bd6b51b-e04b-4862-9519-14b4e16b9a70" containerName="kserve-container"
Sep 18 16:39:04.435691 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.435671    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x"
Sep 18 16:39:04.438107 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.438089    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-de495-kube-rbac-proxy-sar-config\""
Sep 18 16:39:04.438277 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.438091    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-de495-predictor-serving-cert\""
Sep 18 16:39:04.446365 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.446339    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x"]
Sep 18 16:39:04.482754 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.482727    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:04.497707 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.497672    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/687dbe5b-1b65-448e-9b49-577aa1a260cf-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-de495-predictor-6b9d94d467-z9z4x\" (UID: \"687dbe5b-1b65-448e-9b49-577aa1a260cf\") " pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x"
Sep 18 16:39:04.497875 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.497734    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/687dbe5b-1b65-448e-9b49-577aa1a260cf-proxy-tls\") pod \"success-200-isvc-de495-predictor-6b9d94d467-z9z4x\" (UID: \"687dbe5b-1b65-448e-9b49-577aa1a260cf\") " pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x"
Sep 18 16:39:04.497875 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.497759    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz4mm\" (UniqueName: \"kubernetes.io/projected/687dbe5b-1b65-448e-9b49-577aa1a260cf-kube-api-access-tz4mm\") pod \"success-200-isvc-de495-predictor-6b9d94d467-z9z4x\" (UID: \"687dbe5b-1b65-448e-9b49-577aa1a260cf\") " pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x"
Sep 18 16:39:04.553641 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.553551    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:04.598970 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.598932    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/687dbe5b-1b65-448e-9b49-577aa1a260cf-proxy-tls\") pod \"success-200-isvc-de495-predictor-6b9d94d467-z9z4x\" (UID: \"687dbe5b-1b65-448e-9b49-577aa1a260cf\") " pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x"
Sep 18 16:39:04.598970 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.598969    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz4mm\" (UniqueName: \"kubernetes.io/projected/687dbe5b-1b65-448e-9b49-577aa1a260cf-kube-api-access-tz4mm\") pod \"success-200-isvc-de495-predictor-6b9d94d467-z9z4x\" (UID: \"687dbe5b-1b65-448e-9b49-577aa1a260cf\") " pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x"
Sep 18 16:39:04.599180 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.599035    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/687dbe5b-1b65-448e-9b49-577aa1a260cf-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-de495-predictor-6b9d94d467-z9z4x\" (UID: \"687dbe5b-1b65-448e-9b49-577aa1a260cf\") " pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x"
Sep 18 16:39:04.599764 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.599745    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/687dbe5b-1b65-448e-9b49-577aa1a260cf-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-de495-predictor-6b9d94d467-z9z4x\" (UID: \"687dbe5b-1b65-448e-9b49-577aa1a260cf\") " pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x"
Sep 18 16:39:04.601312 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.601268    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/687dbe5b-1b65-448e-9b49-577aa1a260cf-proxy-tls\") pod \"success-200-isvc-de495-predictor-6b9d94d467-z9z4x\" (UID: \"687dbe5b-1b65-448e-9b49-577aa1a260cf\") " pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x"
Sep 18 16:39:04.608236 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.608217    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz4mm\" (UniqueName: \"kubernetes.io/projected/687dbe5b-1b65-448e-9b49-577aa1a260cf-kube-api-access-tz4mm\") pod \"success-200-isvc-de495-predictor-6b9d94d467-z9z4x\" (UID: \"687dbe5b-1b65-448e-9b49-577aa1a260cf\") " pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x"
Sep 18 16:39:04.622771 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.622749    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:04.752507 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.752474    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x"
Sep 18 16:39:04.993344 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:04.993312    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:05.021530 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:05.021508    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x"]
Sep 18 16:39:05.023246 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:39:05.023222    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod687dbe5b_1b65_448e_9b49_577aa1a260cf.slice/crio-d5f3804aadda369cfb94e1ddeb19ff09d97fbd77aeb7edd71011435d581f885b WatchSource:0}: Error finding container d5f3804aadda369cfb94e1ddeb19ff09d97fbd77aeb7edd71011435d581f885b: Status 404 returned error can't find the container with id d5f3804aadda369cfb94e1ddeb19ff09d97fbd77aeb7edd71011435d581f885b
Sep 18 16:39:05.240964 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:05.240935    2580 generic.go:356] "Generic (PLEG): container finished" podID="613372f2-24b2-4f97-950d-198bc835eeb1" containerID="6aa962c36ffe3c88e19f70d3c4ee5ce53e27a8f234b7cd3df2a0ccaf4c2ba46b" exitCode=2
Sep 18 16:39:05.241134 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:05.241012    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb" event={"ID":"613372f2-24b2-4f97-950d-198bc835eeb1","Type":"ContainerDied","Data":"6aa962c36ffe3c88e19f70d3c4ee5ce53e27a8f234b7cd3df2a0ccaf4c2ba46b"}
Sep 18 16:39:05.242678 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:05.242654    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x" event={"ID":"687dbe5b-1b65-448e-9b49-577aa1a260cf","Type":"ContainerStarted","Data":"9a20fa2b9ab4159f678d10c57f334348ac0073c74a16bd0ba239a6d2ea2dbca3"}
Sep 18 16:39:05.242794 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:05.242702    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x" event={"ID":"687dbe5b-1b65-448e-9b49-577aa1a260cf","Type":"ContainerStarted","Data":"b058cdd2e0857c060311f5064cd595fff04bb39a88099cec1da7c8038febb3a1"}
Sep 18 16:39:05.242794 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:05.242729    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x" event={"ID":"687dbe5b-1b65-448e-9b49-577aa1a260cf","Type":"ContainerStarted","Data":"d5f3804aadda369cfb94e1ddeb19ff09d97fbd77aeb7edd71011435d581f885b"}
Sep 18 16:39:05.242794 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:05.242765    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x"
Sep 18 16:39:05.261059 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:05.260972    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x" podStartSLOduration=1.2609538200000001 podStartE2EDuration="1.26095382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:39:05.26006752 +0000 UTC m=+1391.348383161" watchObservedRunningTime="2026-09-18 16:39:05.26095382 +0000 UTC m=+1391.349269461"
Sep 18 16:39:06.247074 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:06.247036    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x"
Sep 18 16:39:06.248372 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:06.248345    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x" podUID="687dbe5b-1b65-448e-9b49-577aa1a260cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 16:39:07.250833 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:07.250795    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x" podUID="687dbe5b-1b65-448e-9b49-577aa1a260cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 16:39:07.908254 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:07.908213    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb" podUID="613372f2-24b2-4f97-950d-198bc835eeb1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 18 16:39:08.257086 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:08.257054    2580 generic.go:356] "Generic (PLEG): container finished" podID="613372f2-24b2-4f97-950d-198bc835eeb1" containerID="e715edfc756b9d49d9813d1b8b2a0b3f40d7525f041be54f9d67a0daa4e879a0" exitCode=0
Sep 18 16:39:08.257438 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:08.257128    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb" event={"ID":"613372f2-24b2-4f97-950d-198bc835eeb1","Type":"ContainerDied","Data":"e715edfc756b9d49d9813d1b8b2a0b3f40d7525f041be54f9d67a0daa4e879a0"}
Sep 18 16:39:08.289135 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:08.289108    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb"
Sep 18 16:39:08.332088 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:08.332062    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gm9ls\" (UniqueName: \"kubernetes.io/projected/613372f2-24b2-4f97-950d-198bc835eeb1-kube-api-access-gm9ls\") pod \"613372f2-24b2-4f97-950d-198bc835eeb1\" (UID: \"613372f2-24b2-4f97-950d-198bc835eeb1\") "
Sep 18 16:39:08.332230 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:08.332114    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/613372f2-24b2-4f97-950d-198bc835eeb1-kube-rbac-proxy-sar-config\") pod \"613372f2-24b2-4f97-950d-198bc835eeb1\" (UID: \"613372f2-24b2-4f97-950d-198bc835eeb1\") "
Sep 18 16:39:08.332230 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:08.332179    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/613372f2-24b2-4f97-950d-198bc835eeb1-proxy-tls\") pod \"613372f2-24b2-4f97-950d-198bc835eeb1\" (UID: \"613372f2-24b2-4f97-950d-198bc835eeb1\") "
Sep 18 16:39:08.332534 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:08.332503    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613372f2-24b2-4f97-950d-198bc835eeb1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "613372f2-24b2-4f97-950d-198bc835eeb1" (UID: "613372f2-24b2-4f97-950d-198bc835eeb1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:39:08.334209 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:08.334188    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/613372f2-24b2-4f97-950d-198bc835eeb1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "613372f2-24b2-4f97-950d-198bc835eeb1" (UID: "613372f2-24b2-4f97-950d-198bc835eeb1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:39:08.334324 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:08.334205    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613372f2-24b2-4f97-950d-198bc835eeb1-kube-api-access-gm9ls" (OuterVolumeSpecName: "kube-api-access-gm9ls") pod "613372f2-24b2-4f97-950d-198bc835eeb1" (UID: "613372f2-24b2-4f97-950d-198bc835eeb1"). InnerVolumeSpecName "kube-api-access-gm9ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:39:08.432815 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:08.432794    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/613372f2-24b2-4f97-950d-198bc835eeb1-proxy-tls\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:39:08.432943 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:08.432816    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm9ls\" (UniqueName: \"kubernetes.io/projected/613372f2-24b2-4f97-950d-198bc835eeb1-kube-api-access-gm9ls\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:39:08.432943 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:08.432828    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/613372f2-24b2-4f97-950d-198bc835eeb1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:39:09.263132 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:09.263098    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb" event={"ID":"613372f2-24b2-4f97-950d-198bc835eeb1","Type":"ContainerDied","Data":"16ffb7c85ff5c4cdd3a611c343a830d3c32f213c1fd21e273e80e2a84c74938f"}
Sep 18 16:39:09.263608 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:09.263138    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb"
Sep 18 16:39:09.263608 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:09.263162    2580 scope.go:117] "RemoveContainer" containerID="6aa962c36ffe3c88e19f70d3c4ee5ce53e27a8f234b7cd3df2a0ccaf4c2ba46b"
Sep 18 16:39:09.272939 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:09.272921    2580 scope.go:117] "RemoveContainer" containerID="e715edfc756b9d49d9813d1b8b2a0b3f40d7525f041be54f9d67a0daa4e879a0"
Sep 18 16:39:09.286510 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:09.286486    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb"]
Sep 18 16:39:09.290092 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:09.290067    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6304f-predictor-9c8948987-8lbnb"]
Sep 18 16:39:10.122255 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:10.122222    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="613372f2-24b2-4f97-950d-198bc835eeb1" path="/var/lib/kubelet/pods/613372f2-24b2-4f97-950d-198bc835eeb1/volumes"
Sep 18 16:39:12.262039 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:12.262007    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x"
Sep 18 16:39:12.262676 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:12.262647    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x" podUID="687dbe5b-1b65-448e-9b49-577aa1a260cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 16:39:13.095578 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:13.095534    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f" podUID="630df95e-f477-4d96-85a8-e21e8ceff156" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 16:39:22.262593 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:22.262553    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x" podUID="687dbe5b-1b65-448e-9b49-577aa1a260cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 16:39:23.096504 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:23.096472    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f"
Sep 18 16:39:32.262866 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:32.262822    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x" podUID="687dbe5b-1b65-448e-9b49-577aa1a260cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 16:39:34.757902 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:34.757870    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:34.910179 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:34.910149    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:42.263331 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:42.263227    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x" podUID="687dbe5b-1b65-448e-9b49-577aa1a260cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 16:39:45.703328 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:45.703277    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:45.785733 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:45.785702    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f"]
Sep 18 16:39:45.785974 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:45.785917    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f" podUID="630df95e-f477-4d96-85a8-e21e8ceff156" containerName="kserve-container" containerID="cri-o://847d907863232b0a9bbf47a4db6dedf82547f875878202b143822926e13e70a7" gracePeriod=30
Sep 18 16:39:45.786073 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:45.785954    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f" podUID="630df95e-f477-4d96-85a8-e21e8ceff156" containerName="kube-rbac-proxy" containerID="cri-o://8aa079c116649dfe69035734054138b7264d6ace20e3c9f8015259a3be11442f" gracePeriod=30
Sep 18 16:39:45.815532 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:45.815503    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:45.909953 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:45.909927    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:45.914163 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:45.914133    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj"]
Sep 18 16:39:45.914540 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:45.914522    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="613372f2-24b2-4f97-950d-198bc835eeb1" containerName="kserve-container"
Sep 18 16:39:45.914540 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:45.914540    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="613372f2-24b2-4f97-950d-198bc835eeb1" containerName="kserve-container"
Sep 18 16:39:45.914663 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:45.914570    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="613372f2-24b2-4f97-950d-198bc835eeb1" containerName="kube-rbac-proxy"
Sep 18 16:39:45.914663 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:45.914576    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="613372f2-24b2-4f97-950d-198bc835eeb1" containerName="kube-rbac-proxy"
Sep 18 16:39:45.914663 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:45.914636    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="613372f2-24b2-4f97-950d-198bc835eeb1" containerName="kube-rbac-proxy"
Sep 18 16:39:45.914663 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:45.914645    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="613372f2-24b2-4f97-950d-198bc835eeb1" containerName="kserve-container"
Sep 18 16:39:45.917633 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:45.917612    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj"
Sep 18 16:39:45.920134 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:45.920113    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-630d7-predictor-serving-cert\""
Sep 18 16:39:45.920252 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:45.920131    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-630d7-kube-rbac-proxy-sar-config\""
Sep 18 16:39:45.927798 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:45.927777    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj"]
Sep 18 16:39:45.956821 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:45.956748    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s6lf\" (UniqueName: \"kubernetes.io/projected/8f93bdc0-729b-42b5-9d61-d7eff0a48438-kube-api-access-9s6lf\") pod \"success-200-isvc-630d7-predictor-76777f589-z5cbj\" (UID: \"8f93bdc0-729b-42b5-9d61-d7eff0a48438\") " pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj"
Sep 18 16:39:45.956946 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:45.956846    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f93bdc0-729b-42b5-9d61-d7eff0a48438-proxy-tls\") pod \"success-200-isvc-630d7-predictor-76777f589-z5cbj\" (UID: \"8f93bdc0-729b-42b5-9d61-d7eff0a48438\") " pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj"
Sep 18 16:39:45.956946 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:45.956916    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f93bdc0-729b-42b5-9d61-d7eff0a48438-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-630d7-predictor-76777f589-z5cbj\" (UID: \"8f93bdc0-729b-42b5-9d61-d7eff0a48438\") " pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj"
Sep 18 16:39:45.998365 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:45.998336    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:46.057962 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:46.057928    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s6lf\" (UniqueName: \"kubernetes.io/projected/8f93bdc0-729b-42b5-9d61-d7eff0a48438-kube-api-access-9s6lf\") pod \"success-200-isvc-630d7-predictor-76777f589-z5cbj\" (UID: \"8f93bdc0-729b-42b5-9d61-d7eff0a48438\") " pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj"
Sep 18 16:39:46.058143 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:46.057992    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f93bdc0-729b-42b5-9d61-d7eff0a48438-proxy-tls\") pod \"success-200-isvc-630d7-predictor-76777f589-z5cbj\" (UID: \"8f93bdc0-729b-42b5-9d61-d7eff0a48438\") " pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj"
Sep 18 16:39:46.058143 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:46.058033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f93bdc0-729b-42b5-9d61-d7eff0a48438-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-630d7-predictor-76777f589-z5cbj\" (UID: \"8f93bdc0-729b-42b5-9d61-d7eff0a48438\") " pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj"
Sep 18 16:39:46.058840 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:46.058776    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f93bdc0-729b-42b5-9d61-d7eff0a48438-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-630d7-predictor-76777f589-z5cbj\" (UID: \"8f93bdc0-729b-42b5-9d61-d7eff0a48438\") " pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj"
Sep 18 16:39:46.060843 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:46.060823    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f93bdc0-729b-42b5-9d61-d7eff0a48438-proxy-tls\") pod \"success-200-isvc-630d7-predictor-76777f589-z5cbj\" (UID: \"8f93bdc0-729b-42b5-9d61-d7eff0a48438\") " pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj"
Sep 18 16:39:46.068068 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:46.068046    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s6lf\" (UniqueName: \"kubernetes.io/projected/8f93bdc0-729b-42b5-9d61-d7eff0a48438-kube-api-access-9s6lf\") pod \"success-200-isvc-630d7-predictor-76777f589-z5cbj\" (UID: \"8f93bdc0-729b-42b5-9d61-d7eff0a48438\") " pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj"
Sep 18 16:39:46.097634 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:46.097610    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:46.230875 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:46.230780    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj"
Sep 18 16:39:46.398294 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:46.398260    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:46.415670 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:46.415638    2580 generic.go:356] "Generic (PLEG): container finished" podID="630df95e-f477-4d96-85a8-e21e8ceff156" containerID="8aa079c116649dfe69035734054138b7264d6ace20e3c9f8015259a3be11442f" exitCode=2
Sep 18 16:39:46.415819 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:46.415712    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f" event={"ID":"630df95e-f477-4d96-85a8-e21e8ceff156","Type":"ContainerDied","Data":"8aa079c116649dfe69035734054138b7264d6ace20e3c9f8015259a3be11442f"}
Sep 18 16:39:46.503853 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:46.503831    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj"]
Sep 18 16:39:46.505692 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:39:46.505669    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f93bdc0_729b_42b5_9d61_d7eff0a48438.slice/crio-9648c9eea31a37942185215ee25c6ed6a68af47b8ae89b118fdf345dd493ad34 WatchSource:0}: Error finding container 9648c9eea31a37942185215ee25c6ed6a68af47b8ae89b118fdf345dd493ad34: Status 404 returned error can't find the container with id 9648c9eea31a37942185215ee25c6ed6a68af47b8ae89b118fdf345dd493ad34
Sep 18 16:39:47.423791 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:47.423757    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj" event={"ID":"8f93bdc0-729b-42b5-9d61-d7eff0a48438","Type":"ContainerStarted","Data":"e37e2c00ce974c3cfe354909cdc527398bf9bcbeab7c71a54248ff66a414ebd3"}
Sep 18 16:39:47.424151 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:47.423816    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj" event={"ID":"8f93bdc0-729b-42b5-9d61-d7eff0a48438","Type":"ContainerStarted","Data":"40ee9908d146beb866802d98888e82c01976231a81b52a81615d11512eeec93e"}
Sep 18 16:39:47.424151 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:47.423837    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj" event={"ID":"8f93bdc0-729b-42b5-9d61-d7eff0a48438","Type":"ContainerStarted","Data":"9648c9eea31a37942185215ee25c6ed6a68af47b8ae89b118fdf345dd493ad34"}
Sep 18 16:39:47.424151 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:47.423860    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj"
Sep 18 16:39:47.424151 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:47.423880    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj"
Sep 18 16:39:47.425256 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:47.425228    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj" podUID="8f93bdc0-729b-42b5-9d61-d7eff0a48438" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 16:39:47.442848 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:47.442807    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj" podStartSLOduration=2.442793726 podStartE2EDuration="2.442793726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:39:47.440821095 +0000 UTC m=+1433.529136773" watchObservedRunningTime="2026-09-18 16:39:47.442793726 +0000 UTC m=+1433.531109365"
Sep 18 16:39:48.086826 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:48.086781    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f" podUID="630df95e-f477-4d96-85a8-e21e8ceff156" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 18 16:39:48.427990 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:48.427955    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj" podUID="8f93bdc0-729b-42b5-9d61-d7eff0a48438" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 16:39:49.434608 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:49.434581    2580 generic.go:356] "Generic (PLEG): container finished" podID="630df95e-f477-4d96-85a8-e21e8ceff156" containerID="847d907863232b0a9bbf47a4db6dedf82547f875878202b143822926e13e70a7" exitCode=0
Sep 18 16:39:49.434964 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:49.434654    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f" event={"ID":"630df95e-f477-4d96-85a8-e21e8ceff156","Type":"ContainerDied","Data":"847d907863232b0a9bbf47a4db6dedf82547f875878202b143822926e13e70a7"}
Sep 18 16:39:49.534154 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:49.534125    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f"
Sep 18 16:39:49.590085 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:49.590005    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/630df95e-f477-4d96-85a8-e21e8ceff156-proxy-tls\") pod \"630df95e-f477-4d96-85a8-e21e8ceff156\" (UID: \"630df95e-f477-4d96-85a8-e21e8ceff156\") "
Sep 18 16:39:49.590085 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:49.590063    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48zb9\" (UniqueName: \"kubernetes.io/projected/630df95e-f477-4d96-85a8-e21e8ceff156-kube-api-access-48zb9\") pod \"630df95e-f477-4d96-85a8-e21e8ceff156\" (UID: \"630df95e-f477-4d96-85a8-e21e8ceff156\") "
Sep 18 16:39:49.590264 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:49.590123    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/630df95e-f477-4d96-85a8-e21e8ceff156-kube-rbac-proxy-sar-config\") pod \"630df95e-f477-4d96-85a8-e21e8ceff156\" (UID: \"630df95e-f477-4d96-85a8-e21e8ceff156\") "
Sep 18 16:39:49.590501 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:49.590472    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/630df95e-f477-4d96-85a8-e21e8ceff156-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "630df95e-f477-4d96-85a8-e21e8ceff156" (UID: "630df95e-f477-4d96-85a8-e21e8ceff156"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:39:49.592124 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:49.592105    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630df95e-f477-4d96-85a8-e21e8ceff156-kube-api-access-48zb9" (OuterVolumeSpecName: "kube-api-access-48zb9") pod "630df95e-f477-4d96-85a8-e21e8ceff156" (UID: "630df95e-f477-4d96-85a8-e21e8ceff156"). InnerVolumeSpecName "kube-api-access-48zb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:39:49.592222 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:49.592146    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/630df95e-f477-4d96-85a8-e21e8ceff156-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "630df95e-f477-4d96-85a8-e21e8ceff156" (UID: "630df95e-f477-4d96-85a8-e21e8ceff156"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:39:49.691724 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:49.691691    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/630df95e-f477-4d96-85a8-e21e8ceff156-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:39:49.691724 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:49.691721    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/630df95e-f477-4d96-85a8-e21e8ceff156-proxy-tls\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:39:49.691724 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:49.691731    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48zb9\" (UniqueName: \"kubernetes.io/projected/630df95e-f477-4d96-85a8-e21e8ceff156-kube-api-access-48zb9\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:39:50.440461 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:50.440426    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f" event={"ID":"630df95e-f477-4d96-85a8-e21e8ceff156","Type":"ContainerDied","Data":"24b8fc6fa60793ccae6f8999178c86f0f7a5562a0beb046d6cf16447955d6df6"}
Sep 18 16:39:50.440461 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:50.440451    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f"
Sep 18 16:39:50.440950 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:50.440503    2580 scope.go:117] "RemoveContainer" containerID="8aa079c116649dfe69035734054138b7264d6ace20e3c9f8015259a3be11442f"
Sep 18 16:39:50.448398 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:50.448378    2580 scope.go:117] "RemoveContainer" containerID="847d907863232b0a9bbf47a4db6dedf82547f875878202b143822926e13e70a7"
Sep 18 16:39:50.459027 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:50.459002    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f"]
Sep 18 16:39:50.467333 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:50.465188    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7c9ed-predictor-7fbffd8ccc-xzt2f"]
Sep 18 16:39:52.122459 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:52.122424    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="630df95e-f477-4d96-85a8-e21e8ceff156" path="/var/lib/kubelet/pods/630df95e-f477-4d96-85a8-e21e8ceff156/volumes"
Sep 18 16:39:52.263481 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:52.263443    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x" podUID="687dbe5b-1b65-448e-9b49-577aa1a260cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 16:39:53.431937 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:53.431911    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj"
Sep 18 16:39:53.432523 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:39:53.432491    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj" podUID="8f93bdc0-729b-42b5-9d61-d7eff0a48438" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 16:40:02.263448 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:40:02.263412    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x"
Sep 18 16:40:03.432858 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:40:03.432815    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj" podUID="8f93bdc0-729b-42b5-9d61-d7eff0a48438" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 16:40:13.432647 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:40:13.432604    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj" podUID="8f93bdc0-729b-42b5-9d61-d7eff0a48438" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 16:40:14.506748 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:40:14.506713    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:40:14.835422 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:40:14.835340    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:40:23.433405 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:40:23.433352    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj" podUID="8f93bdc0-729b-42b5-9d61-d7eff0a48438" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 16:40:33.433137 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:40:33.433100    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj" podUID="8f93bdc0-729b-42b5-9d61-d7eff0a48438" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 16:40:43.433026 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:40:43.432993    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj"
Sep 18 16:40:56.709831 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:40:56.709793    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:40:56.775949 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:40:56.775924    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:42:58.448465 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:42:58.448431    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:43:42.216446 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:43:42.216416    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:45:40.078729 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:45:40.078654    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:46:50.011581 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:46:50.011531    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:48:29.550611 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:29.550575    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:48:29.627605 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:29.627571    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x"]
Sep 18 16:48:29.627894 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:29.627842    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x" podUID="687dbe5b-1b65-448e-9b49-577aa1a260cf" containerName="kserve-container" containerID="cri-o://b058cdd2e0857c060311f5064cd595fff04bb39a88099cec1da7c8038febb3a1" gracePeriod=30
Sep 18 16:48:29.627996 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:29.627895    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x" podUID="687dbe5b-1b65-448e-9b49-577aa1a260cf" containerName="kube-rbac-proxy" containerID="cri-o://9a20fa2b9ab4159f678d10c57f334348ac0073c74a16bd0ba239a6d2ea2dbca3" gracePeriod=30
Sep 18 16:48:29.713939 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:29.713900    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q"]
Sep 18 16:48:29.714399 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:29.714377    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="630df95e-f477-4d96-85a8-e21e8ceff156" containerName="kube-rbac-proxy"
Sep 18 16:48:29.714399 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:29.714400    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="630df95e-f477-4d96-85a8-e21e8ceff156" containerName="kube-rbac-proxy"
Sep 18 16:48:29.714586 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:29.714433    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="630df95e-f477-4d96-85a8-e21e8ceff156" containerName="kserve-container"
Sep 18 16:48:29.714586 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:29.714443    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="630df95e-f477-4d96-85a8-e21e8ceff156" containerName="kserve-container"
Sep 18 16:48:29.714586 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:29.714534    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="630df95e-f477-4d96-85a8-e21e8ceff156" containerName="kube-rbac-proxy"
Sep 18 16:48:29.714586 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:29.714548    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="630df95e-f477-4d96-85a8-e21e8ceff156" containerName="kserve-container"
Sep 18 16:48:29.717835 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:29.717807    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q"
Sep 18 16:48:29.720561 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:29.720525    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b3de6-kube-rbac-proxy-sar-config\""
Sep 18 16:48:29.720561 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:29.720527    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b3de6-predictor-serving-cert\""
Sep 18 16:48:29.734765 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:29.734738    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q"]
Sep 18 16:48:29.796870 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:29.796833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f9b2d18-1bfc-4ceb-a344-738bd027e59a-proxy-tls\") pod \"success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q\" (UID: \"1f9b2d18-1bfc-4ceb-a344-738bd027e59a\") " pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q"
Sep 18 16:48:29.797063 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:29.796892    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sff4\" (UniqueName: \"kubernetes.io/projected/1f9b2d18-1bfc-4ceb-a344-738bd027e59a-kube-api-access-7sff4\") pod \"success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q\" (UID: \"1f9b2d18-1bfc-4ceb-a344-738bd027e59a\") " pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q"
Sep 18 16:48:29.797063 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:29.796993    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f9b2d18-1bfc-4ceb-a344-738bd027e59a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q\" (UID: \"1f9b2d18-1bfc-4ceb-a344-738bd027e59a\") " pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q"
Sep 18 16:48:29.898157 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:29.898060    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f9b2d18-1bfc-4ceb-a344-738bd027e59a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q\" (UID: \"1f9b2d18-1bfc-4ceb-a344-738bd027e59a\") " pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q"
Sep 18 16:48:29.898157 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:29.898123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f9b2d18-1bfc-4ceb-a344-738bd027e59a-proxy-tls\") pod \"success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q\" (UID: \"1f9b2d18-1bfc-4ceb-a344-738bd027e59a\") " pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q"
Sep 18 16:48:29.898444 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:29.898176    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sff4\" (UniqueName: \"kubernetes.io/projected/1f9b2d18-1bfc-4ceb-a344-738bd027e59a-kube-api-access-7sff4\") pod \"success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q\" (UID: \"1f9b2d18-1bfc-4ceb-a344-738bd027e59a\") " pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q"
Sep 18 16:48:29.898875 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:29.898851    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f9b2d18-1bfc-4ceb-a344-738bd027e59a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q\" (UID: \"1f9b2d18-1bfc-4ceb-a344-738bd027e59a\") " pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q"
Sep 18 16:48:29.900632 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:29.900612    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f9b2d18-1bfc-4ceb-a344-738bd027e59a-proxy-tls\") pod \"success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q\" (UID: \"1f9b2d18-1bfc-4ceb-a344-738bd027e59a\") " pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q"
Sep 18 16:48:29.905548 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:29.905528    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sff4\" (UniqueName: \"kubernetes.io/projected/1f9b2d18-1bfc-4ceb-a344-738bd027e59a-kube-api-access-7sff4\") pod \"success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q\" (UID: \"1f9b2d18-1bfc-4ceb-a344-738bd027e59a\") " pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q"
Sep 18 16:48:30.029711 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:30.029666    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q"
Sep 18 16:48:30.303440 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:30.303411    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q"]
Sep 18 16:48:30.305468 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:48:30.305435    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f9b2d18_1bfc_4ceb_a344_738bd027e59a.slice/crio-166984defcc52f300c40b18f6f4800a405aaed58dbfd19ae357bbc8888a199f1 WatchSource:0}: Error finding container 166984defcc52f300c40b18f6f4800a405aaed58dbfd19ae357bbc8888a199f1: Status 404 returned error can't find the container with id 166984defcc52f300c40b18f6f4800a405aaed58dbfd19ae357bbc8888a199f1
Sep 18 16:48:30.610687 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:30.610590    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q" event={"ID":"1f9b2d18-1bfc-4ceb-a344-738bd027e59a","Type":"ContainerStarted","Data":"2bf804d80c258812b669df8683e58df16ba5d36edc6ad5e1f80333da5abb7cd5"}
Sep 18 16:48:30.611132 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:30.610767    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q"
Sep 18 16:48:30.611132 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:30.610824    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q"
Sep 18 16:48:30.611132 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:30.610851    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q" event={"ID":"1f9b2d18-1bfc-4ceb-a344-738bd027e59a","Type":"ContainerStarted","Data":"71ec1c913bfbfda7ea70467250f12737bd8207975100ab42cbdf4c165c3d463e"}
Sep 18 16:48:30.611132 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:30.610879    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q" event={"ID":"1f9b2d18-1bfc-4ceb-a344-738bd027e59a","Type":"ContainerStarted","Data":"166984defcc52f300c40b18f6f4800a405aaed58dbfd19ae357bbc8888a199f1"}
Sep 18 16:48:30.611874 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:30.611844    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q" podUID="1f9b2d18-1bfc-4ceb-a344-738bd027e59a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 16:48:30.612886 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:30.612856    2580 generic.go:356] "Generic (PLEG): container finished" podID="687dbe5b-1b65-448e-9b49-577aa1a260cf" containerID="9a20fa2b9ab4159f678d10c57f334348ac0073c74a16bd0ba239a6d2ea2dbca3" exitCode=2
Sep 18 16:48:30.613023 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:30.612930    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x" event={"ID":"687dbe5b-1b65-448e-9b49-577aa1a260cf","Type":"ContainerDied","Data":"9a20fa2b9ab4159f678d10c57f334348ac0073c74a16bd0ba239a6d2ea2dbca3"}
Sep 18 16:48:31.004270 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:31.004226    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:48:31.618733 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:31.618693    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q" podUID="1f9b2d18-1bfc-4ceb-a344-738bd027e59a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 16:48:32.251903 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:32.251859    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x" podUID="687dbe5b-1b65-448e-9b49-577aa1a260cf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 18 16:48:32.263074 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:32.263034    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x" podUID="687dbe5b-1b65-448e-9b49-577aa1a260cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 16:48:33.176575 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:33.176550    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x"
Sep 18 16:48:33.196238 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:33.196181    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q" podStartSLOduration=4.196165647 podStartE2EDuration="4.196165647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:48:30.63023876 +0000 UTC m=+1956.718554397" watchObservedRunningTime="2026-09-18 16:48:33.196165647 +0000 UTC m=+1959.284481288"
Sep 18 16:48:33.330841 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:33.330752    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/687dbe5b-1b65-448e-9b49-577aa1a260cf-proxy-tls\") pod \"687dbe5b-1b65-448e-9b49-577aa1a260cf\" (UID: \"687dbe5b-1b65-448e-9b49-577aa1a260cf\") "
Sep 18 16:48:33.330841 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:33.330798    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tz4mm\" (UniqueName: \"kubernetes.io/projected/687dbe5b-1b65-448e-9b49-577aa1a260cf-kube-api-access-tz4mm\") pod \"687dbe5b-1b65-448e-9b49-577aa1a260cf\" (UID: \"687dbe5b-1b65-448e-9b49-577aa1a260cf\") "
Sep 18 16:48:33.330841 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:33.330843    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/687dbe5b-1b65-448e-9b49-577aa1a260cf-kube-rbac-proxy-sar-config\") pod \"687dbe5b-1b65-448e-9b49-577aa1a260cf\" (UID: \"687dbe5b-1b65-448e-9b49-577aa1a260cf\") "
Sep 18 16:48:33.331257 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:33.331229    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/687dbe5b-1b65-448e-9b49-577aa1a260cf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "687dbe5b-1b65-448e-9b49-577aa1a260cf" (UID: "687dbe5b-1b65-448e-9b49-577aa1a260cf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:48:33.332909 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:33.332875    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/687dbe5b-1b65-448e-9b49-577aa1a260cf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "687dbe5b-1b65-448e-9b49-577aa1a260cf" (UID: "687dbe5b-1b65-448e-9b49-577aa1a260cf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:48:33.332995 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:33.332904    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687dbe5b-1b65-448e-9b49-577aa1a260cf-kube-api-access-tz4mm" (OuterVolumeSpecName: "kube-api-access-tz4mm") pod "687dbe5b-1b65-448e-9b49-577aa1a260cf" (UID: "687dbe5b-1b65-448e-9b49-577aa1a260cf"). InnerVolumeSpecName "kube-api-access-tz4mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:48:33.432111 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:33.432077    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/687dbe5b-1b65-448e-9b49-577aa1a260cf-proxy-tls\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:48:33.432111 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:33.432106    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz4mm\" (UniqueName: \"kubernetes.io/projected/687dbe5b-1b65-448e-9b49-577aa1a260cf-kube-api-access-tz4mm\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:48:33.432111 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:33.432117    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/687dbe5b-1b65-448e-9b49-577aa1a260cf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:48:33.629974 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:33.629888    2580 generic.go:356] "Generic (PLEG): container finished" podID="687dbe5b-1b65-448e-9b49-577aa1a260cf" containerID="b058cdd2e0857c060311f5064cd595fff04bb39a88099cec1da7c8038febb3a1" exitCode=0
Sep 18 16:48:33.629974 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:33.629957    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x"
Sep 18 16:48:33.630184 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:33.629976    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x" event={"ID":"687dbe5b-1b65-448e-9b49-577aa1a260cf","Type":"ContainerDied","Data":"b058cdd2e0857c060311f5064cd595fff04bb39a88099cec1da7c8038febb3a1"}
Sep 18 16:48:33.630184 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:33.630063    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x" event={"ID":"687dbe5b-1b65-448e-9b49-577aa1a260cf","Type":"ContainerDied","Data":"d5f3804aadda369cfb94e1ddeb19ff09d97fbd77aeb7edd71011435d581f885b"}
Sep 18 16:48:33.630184 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:33.630093    2580 scope.go:117] "RemoveContainer" containerID="9a20fa2b9ab4159f678d10c57f334348ac0073c74a16bd0ba239a6d2ea2dbca3"
Sep 18 16:48:33.639600 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:33.639585    2580 scope.go:117] "RemoveContainer" containerID="b058cdd2e0857c060311f5064cd595fff04bb39a88099cec1da7c8038febb3a1"
Sep 18 16:48:33.647130 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:33.647111    2580 scope.go:117] "RemoveContainer" containerID="9a20fa2b9ab4159f678d10c57f334348ac0073c74a16bd0ba239a6d2ea2dbca3"
Sep 18 16:48:33.647404 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:48:33.647384    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a20fa2b9ab4159f678d10c57f334348ac0073c74a16bd0ba239a6d2ea2dbca3\": container with ID starting with 9a20fa2b9ab4159f678d10c57f334348ac0073c74a16bd0ba239a6d2ea2dbca3 not found: ID does not exist" containerID="9a20fa2b9ab4159f678d10c57f334348ac0073c74a16bd0ba239a6d2ea2dbca3"
Sep 18 16:48:33.647474 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:33.647411    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a20fa2b9ab4159f678d10c57f334348ac0073c74a16bd0ba239a6d2ea2dbca3"} err="failed to get container status \"9a20fa2b9ab4159f678d10c57f334348ac0073c74a16bd0ba239a6d2ea2dbca3\": rpc error: code = NotFound desc = could not find container \"9a20fa2b9ab4159f678d10c57f334348ac0073c74a16bd0ba239a6d2ea2dbca3\": container with ID starting with 9a20fa2b9ab4159f678d10c57f334348ac0073c74a16bd0ba239a6d2ea2dbca3 not found: ID does not exist"
Sep 18 16:48:33.647474 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:33.647426    2580 scope.go:117] "RemoveContainer" containerID="b058cdd2e0857c060311f5064cd595fff04bb39a88099cec1da7c8038febb3a1"
Sep 18 16:48:33.647710 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:48:33.647690    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b058cdd2e0857c060311f5064cd595fff04bb39a88099cec1da7c8038febb3a1\": container with ID starting with b058cdd2e0857c060311f5064cd595fff04bb39a88099cec1da7c8038febb3a1 not found: ID does not exist" containerID="b058cdd2e0857c060311f5064cd595fff04bb39a88099cec1da7c8038febb3a1"
Sep 18 16:48:33.647760 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:33.647715    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b058cdd2e0857c060311f5064cd595fff04bb39a88099cec1da7c8038febb3a1"} err="failed to get container status \"b058cdd2e0857c060311f5064cd595fff04bb39a88099cec1da7c8038febb3a1\": rpc error: code = NotFound desc = could not find container \"b058cdd2e0857c060311f5064cd595fff04bb39a88099cec1da7c8038febb3a1\": container with ID starting with b058cdd2e0857c060311f5064cd595fff04bb39a88099cec1da7c8038febb3a1 not found: ID does not exist"
Sep 18 16:48:33.652381 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:33.652359    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x"]
Sep 18 16:48:33.657007 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:33.656986    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-de495-predictor-6b9d94d467-z9z4x"]
Sep 18 16:48:34.122381 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:34.122349    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="687dbe5b-1b65-448e-9b49-577aa1a260cf" path="/var/lib/kubelet/pods/687dbe5b-1b65-448e-9b49-577aa1a260cf/volumes"
Sep 18 16:48:36.623607 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:36.623579    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q"
Sep 18 16:48:36.624162 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:36.624136    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q" podUID="1f9b2d18-1bfc-4ceb-a344-738bd027e59a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 16:48:46.624830 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:46.624785    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q" podUID="1f9b2d18-1bfc-4ceb-a344-738bd027e59a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 16:48:56.624513 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:48:56.624428    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q" podUID="1f9b2d18-1bfc-4ceb-a344-738bd027e59a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 16:49:06.624174 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:06.624129    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q" podUID="1f9b2d18-1bfc-4ceb-a344-738bd027e59a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 16:49:10.826547 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:10.826504    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:10.924906 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:10.924867    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj"]
Sep 18 16:49:10.925157 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:10.925127    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj" podUID="8f93bdc0-729b-42b5-9d61-d7eff0a48438" containerName="kserve-container" containerID="cri-o://40ee9908d146beb866802d98888e82c01976231a81b52a81615d11512eeec93e" gracePeriod=30
Sep 18 16:49:10.925157 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:10.925147    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj" podUID="8f93bdc0-729b-42b5-9d61-d7eff0a48438" containerName="kube-rbac-proxy" containerID="cri-o://e37e2c00ce974c3cfe354909cdc527398bf9bcbeab7c71a54248ff66a414ebd3" gracePeriod=30
Sep 18 16:49:10.974695 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:10.974668    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:11.064724 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.064690    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:11.068591 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.068567    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26"]
Sep 18 16:49:11.068921 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.068909    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="687dbe5b-1b65-448e-9b49-577aa1a260cf" containerName="kserve-container"
Sep 18 16:49:11.068973 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.068923    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="687dbe5b-1b65-448e-9b49-577aa1a260cf" containerName="kserve-container"
Sep 18 16:49:11.068973 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.068945    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="687dbe5b-1b65-448e-9b49-577aa1a260cf" containerName="kube-rbac-proxy"
Sep 18 16:49:11.068973 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.068950    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="687dbe5b-1b65-448e-9b49-577aa1a260cf" containerName="kube-rbac-proxy"
Sep 18 16:49:11.069074 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.069001    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="687dbe5b-1b65-448e-9b49-577aa1a260cf" containerName="kserve-container"
Sep 18 16:49:11.069074 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.069011    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="687dbe5b-1b65-448e-9b49-577aa1a260cf" containerName="kube-rbac-proxy"
Sep 18 16:49:11.072089 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.072073    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26"
Sep 18 16:49:11.074670 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.074648    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a2696-predictor-serving-cert\""
Sep 18 16:49:11.075001 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.074980    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a2696-kube-rbac-proxy-sar-config\""
Sep 18 16:49:11.082328 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.082292    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26"]
Sep 18 16:49:11.162820 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.162791    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53c2ad28-054c-4eeb-9b70-1136a0c5cdfd-proxy-tls\") pod \"success-200-isvc-a2696-predictor-6d48d4c789-h4v26\" (UID: \"53c2ad28-054c-4eeb-9b70-1136a0c5cdfd\") " pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26"
Sep 18 16:49:11.162992 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.162836    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53c2ad28-054c-4eeb-9b70-1136a0c5cdfd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a2696-predictor-6d48d4c789-h4v26\" (UID: \"53c2ad28-054c-4eeb-9b70-1136a0c5cdfd\") " pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26"
Sep 18 16:49:11.162992 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.162855    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s75dt\" (UniqueName: \"kubernetes.io/projected/53c2ad28-054c-4eeb-9b70-1136a0c5cdfd-kube-api-access-s75dt\") pod \"success-200-isvc-a2696-predictor-6d48d4c789-h4v26\" (UID: \"53c2ad28-054c-4eeb-9b70-1136a0c5cdfd\") " pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26"
Sep 18 16:49:11.180943 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.180914    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:11.263915 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.263883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53c2ad28-054c-4eeb-9b70-1136a0c5cdfd-proxy-tls\") pod \"success-200-isvc-a2696-predictor-6d48d4c789-h4v26\" (UID: \"53c2ad28-054c-4eeb-9b70-1136a0c5cdfd\") " pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26"
Sep 18 16:49:11.264092 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.263931    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53c2ad28-054c-4eeb-9b70-1136a0c5cdfd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a2696-predictor-6d48d4c789-h4v26\" (UID: \"53c2ad28-054c-4eeb-9b70-1136a0c5cdfd\") " pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26"
Sep 18 16:49:11.264092 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.263952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s75dt\" (UniqueName: \"kubernetes.io/projected/53c2ad28-054c-4eeb-9b70-1136a0c5cdfd-kube-api-access-s75dt\") pod \"success-200-isvc-a2696-predictor-6d48d4c789-h4v26\" (UID: \"53c2ad28-054c-4eeb-9b70-1136a0c5cdfd\") " pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26"
Sep 18 16:49:11.264612 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.264581    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53c2ad28-054c-4eeb-9b70-1136a0c5cdfd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a2696-predictor-6d48d4c789-h4v26\" (UID: \"53c2ad28-054c-4eeb-9b70-1136a0c5cdfd\") " pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26"
Sep 18 16:49:11.266332 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.266315    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53c2ad28-054c-4eeb-9b70-1136a0c5cdfd-proxy-tls\") pod \"success-200-isvc-a2696-predictor-6d48d4c789-h4v26\" (UID: \"53c2ad28-054c-4eeb-9b70-1136a0c5cdfd\") " pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26"
Sep 18 16:49:11.272893 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.272873    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s75dt\" (UniqueName: \"kubernetes.io/projected/53c2ad28-054c-4eeb-9b70-1136a0c5cdfd-kube-api-access-s75dt\") pod \"success-200-isvc-a2696-predictor-6d48d4c789-h4v26\" (UID: \"53c2ad28-054c-4eeb-9b70-1136a0c5cdfd\") " pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26"
Sep 18 16:49:11.276005 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.275969    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:11.382637 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.382556    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26"
Sep 18 16:49:11.413998 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.413963    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:11.653715 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.653637    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26"]
Sep 18 16:49:11.656751 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:49:11.656714    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c2ad28_054c_4eeb_9b70_1136a0c5cdfd.slice/crio-ffcec94e2b49a86ed6d51a665ae59fa4eda4ab5fe303d0d9106ccbf683db10bc WatchSource:0}: Error finding container ffcec94e2b49a86ed6d51a665ae59fa4eda4ab5fe303d0d9106ccbf683db10bc: Status 404 returned error can't find the container with id ffcec94e2b49a86ed6d51a665ae59fa4eda4ab5fe303d0d9106ccbf683db10bc
Sep 18 16:49:11.796180 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.796144    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26" event={"ID":"53c2ad28-054c-4eeb-9b70-1136a0c5cdfd","Type":"ContainerStarted","Data":"262e191480634c7cd11cefba5b17786818bb495ddd51159c88f11d885deda619"}
Sep 18 16:49:11.796338 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.796214    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26" event={"ID":"53c2ad28-054c-4eeb-9b70-1136a0c5cdfd","Type":"ContainerStarted","Data":"fcf870ad04a8d2d155d2c23c1051a5648c175b45cdbe56a5bb350f152cb75d02"}
Sep 18 16:49:11.796338 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.796245    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26" event={"ID":"53c2ad28-054c-4eeb-9b70-1136a0c5cdfd","Type":"ContainerStarted","Data":"ffcec94e2b49a86ed6d51a665ae59fa4eda4ab5fe303d0d9106ccbf683db10bc"}
Sep 18 16:49:11.796338 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.796278    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26"
Sep 18 16:49:11.796338 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.796329    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26"
Sep 18 16:49:11.797923 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.797898    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26" podUID="53c2ad28-054c-4eeb-9b70-1136a0c5cdfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 16:49:11.798053 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.798030    2580 generic.go:356] "Generic (PLEG): container finished" podID="8f93bdc0-729b-42b5-9d61-d7eff0a48438" containerID="e37e2c00ce974c3cfe354909cdc527398bf9bcbeab7c71a54248ff66a414ebd3" exitCode=2
Sep 18 16:49:11.798110 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.798100    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj" event={"ID":"8f93bdc0-729b-42b5-9d61-d7eff0a48438","Type":"ContainerDied","Data":"e37e2c00ce974c3cfe354909cdc527398bf9bcbeab7c71a54248ff66a414ebd3"}
Sep 18 16:49:11.815992 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:11.815654    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26" podStartSLOduration=0.815633141 podStartE2EDuration="815.633141ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:49:11.815602994 +0000 UTC m=+1997.903918637" watchObservedRunningTime="2026-09-18 16:49:11.815633141 +0000 UTC m=+1997.903948795"
Sep 18 16:49:12.807682 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:12.807629    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26" podUID="53c2ad28-054c-4eeb-9b70-1136a0c5cdfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 16:49:13.428536 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:13.428494    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj" podUID="8f93bdc0-729b-42b5-9d61-d7eff0a48438" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 18 16:49:13.432843 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:13.432814    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj" podUID="8f93bdc0-729b-42b5-9d61-d7eff0a48438" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 16:49:14.468777 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:14.468748    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj"
Sep 18 16:49:14.596010 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:14.595924    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f93bdc0-729b-42b5-9d61-d7eff0a48438-kube-rbac-proxy-sar-config\") pod \"8f93bdc0-729b-42b5-9d61-d7eff0a48438\" (UID: \"8f93bdc0-729b-42b5-9d61-d7eff0a48438\") "
Sep 18 16:49:14.596146 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:14.596039    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f93bdc0-729b-42b5-9d61-d7eff0a48438-proxy-tls\") pod \"8f93bdc0-729b-42b5-9d61-d7eff0a48438\" (UID: \"8f93bdc0-729b-42b5-9d61-d7eff0a48438\") "
Sep 18 16:49:14.596146 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:14.596108    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9s6lf\" (UniqueName: \"kubernetes.io/projected/8f93bdc0-729b-42b5-9d61-d7eff0a48438-kube-api-access-9s6lf\") pod \"8f93bdc0-729b-42b5-9d61-d7eff0a48438\" (UID: \"8f93bdc0-729b-42b5-9d61-d7eff0a48438\") "
Sep 18 16:49:14.596341 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:14.596295    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f93bdc0-729b-42b5-9d61-d7eff0a48438-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8f93bdc0-729b-42b5-9d61-d7eff0a48438" (UID: "8f93bdc0-729b-42b5-9d61-d7eff0a48438"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:49:14.596421 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:14.596396    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f93bdc0-729b-42b5-9d61-d7eff0a48438-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:49:14.598033 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:14.598009    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f93bdc0-729b-42b5-9d61-d7eff0a48438-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8f93bdc0-729b-42b5-9d61-d7eff0a48438" (UID: "8f93bdc0-729b-42b5-9d61-d7eff0a48438"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:49:14.598244 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:14.598223    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f93bdc0-729b-42b5-9d61-d7eff0a48438-kube-api-access-9s6lf" (OuterVolumeSpecName: "kube-api-access-9s6lf") pod "8f93bdc0-729b-42b5-9d61-d7eff0a48438" (UID: "8f93bdc0-729b-42b5-9d61-d7eff0a48438"). InnerVolumeSpecName "kube-api-access-9s6lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:49:14.697831 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:14.697794    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f93bdc0-729b-42b5-9d61-d7eff0a48438-proxy-tls\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:49:14.697831 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:14.697823    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s6lf\" (UniqueName: \"kubernetes.io/projected/8f93bdc0-729b-42b5-9d61-d7eff0a48438-kube-api-access-9s6lf\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:49:14.815194 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:14.815157    2580 generic.go:356] "Generic (PLEG): container finished" podID="8f93bdc0-729b-42b5-9d61-d7eff0a48438" containerID="40ee9908d146beb866802d98888e82c01976231a81b52a81615d11512eeec93e" exitCode=0
Sep 18 16:49:14.815385 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:14.815219    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj" event={"ID":"8f93bdc0-729b-42b5-9d61-d7eff0a48438","Type":"ContainerDied","Data":"40ee9908d146beb866802d98888e82c01976231a81b52a81615d11512eeec93e"}
Sep 18 16:49:14.815385 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:14.815270    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj" event={"ID":"8f93bdc0-729b-42b5-9d61-d7eff0a48438","Type":"ContainerDied","Data":"9648c9eea31a37942185215ee25c6ed6a68af47b8ae89b118fdf345dd493ad34"}
Sep 18 16:49:14.815385 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:14.815314    2580 scope.go:117] "RemoveContainer" containerID="e37e2c00ce974c3cfe354909cdc527398bf9bcbeab7c71a54248ff66a414ebd3"
Sep 18 16:49:14.815385 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:14.815218    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj"
Sep 18 16:49:14.824471 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:14.824451    2580 scope.go:117] "RemoveContainer" containerID="40ee9908d146beb866802d98888e82c01976231a81b52a81615d11512eeec93e"
Sep 18 16:49:14.831864 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:14.831844    2580 scope.go:117] "RemoveContainer" containerID="e37e2c00ce974c3cfe354909cdc527398bf9bcbeab7c71a54248ff66a414ebd3"
Sep 18 16:49:14.832127 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:49:14.832103    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e37e2c00ce974c3cfe354909cdc527398bf9bcbeab7c71a54248ff66a414ebd3\": container with ID starting with e37e2c00ce974c3cfe354909cdc527398bf9bcbeab7c71a54248ff66a414ebd3 not found: ID does not exist" containerID="e37e2c00ce974c3cfe354909cdc527398bf9bcbeab7c71a54248ff66a414ebd3"
Sep 18 16:49:14.832269 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:14.832133    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e37e2c00ce974c3cfe354909cdc527398bf9bcbeab7c71a54248ff66a414ebd3"} err="failed to get container status \"e37e2c00ce974c3cfe354909cdc527398bf9bcbeab7c71a54248ff66a414ebd3\": rpc error: code = NotFound desc = could not find container \"e37e2c00ce974c3cfe354909cdc527398bf9bcbeab7c71a54248ff66a414ebd3\": container with ID starting with e37e2c00ce974c3cfe354909cdc527398bf9bcbeab7c71a54248ff66a414ebd3 not found: ID does not exist"
Sep 18 16:49:14.832269 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:14.832150    2580 scope.go:117] "RemoveContainer" containerID="40ee9908d146beb866802d98888e82c01976231a81b52a81615d11512eeec93e"
Sep 18 16:49:14.832423 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:49:14.832405    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40ee9908d146beb866802d98888e82c01976231a81b52a81615d11512eeec93e\": container with ID starting with 40ee9908d146beb866802d98888e82c01976231a81b52a81615d11512eeec93e not found: ID does not exist" containerID="40ee9908d146beb866802d98888e82c01976231a81b52a81615d11512eeec93e"
Sep 18 16:49:14.832460 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:14.832429    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ee9908d146beb866802d98888e82c01976231a81b52a81615d11512eeec93e"} err="failed to get container status \"40ee9908d146beb866802d98888e82c01976231a81b52a81615d11512eeec93e\": rpc error: code = NotFound desc = could not find container \"40ee9908d146beb866802d98888e82c01976231a81b52a81615d11512eeec93e\": container with ID starting with 40ee9908d146beb866802d98888e82c01976231a81b52a81615d11512eeec93e not found: ID does not exist"
Sep 18 16:49:14.839088 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:14.839066    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj"]
Sep 18 16:49:14.843920 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:14.843901    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-630d7-predictor-76777f589-z5cbj"]
Sep 18 16:49:16.122652 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:16.122621    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f93bdc0-729b-42b5-9d61-d7eff0a48438" path="/var/lib/kubelet/pods/8f93bdc0-729b-42b5-9d61-d7eff0a48438/volumes"
Sep 18 16:49:16.624345 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:16.624288    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q" podUID="1f9b2d18-1bfc-4ceb-a344-738bd027e59a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 16:49:17.809838 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:17.809809    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26"
Sep 18 16:49:17.810240 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:17.810213    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26" podUID="53c2ad28-054c-4eeb-9b70-1136a0c5cdfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 16:49:26.624781 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:26.624751    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q"
Sep 18 16:49:27.810797 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:27.810759    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26" podUID="53c2ad28-054c-4eeb-9b70-1136a0c5cdfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 16:49:37.810895 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:37.810853    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26" podUID="53c2ad28-054c-4eeb-9b70-1136a0c5cdfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 16:49:39.759329 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:39.759232    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:39.938608 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:39.938573    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:47.810398 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:47.810358    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26" podUID="53c2ad28-054c-4eeb-9b70-1136a0c5cdfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 16:49:50.821169 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:50.821130    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:50.892884 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:50.892852    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q"]
Sep 18 16:49:50.893127 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:50.893100    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q" podUID="1f9b2d18-1bfc-4ceb-a344-738bd027e59a" containerName="kserve-container" containerID="cri-o://71ec1c913bfbfda7ea70467250f12737bd8207975100ab42cbdf4c165c3d463e" gracePeriod=30
Sep 18 16:49:50.893222 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:50.893155    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q" podUID="1f9b2d18-1bfc-4ceb-a344-738bd027e59a" containerName="kube-rbac-proxy" containerID="cri-o://2bf804d80c258812b669df8683e58df16ba5d36edc6ad5e1f80333da5abb7cd5" gracePeriod=30
Sep 18 16:49:50.922283 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:50.922242    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:51.020528 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.020493    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:51.027782 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.027739    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf"]
Sep 18 16:49:51.028116 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.028103    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f93bdc0-729b-42b5-9d61-d7eff0a48438" containerName="kserve-container"
Sep 18 16:49:51.028176 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.028118    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f93bdc0-729b-42b5-9d61-d7eff0a48438" containerName="kserve-container"
Sep 18 16:49:51.028176 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.028141    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f93bdc0-729b-42b5-9d61-d7eff0a48438" containerName="kube-rbac-proxy"
Sep 18 16:49:51.028176 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.028146    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f93bdc0-729b-42b5-9d61-d7eff0a48438" containerName="kube-rbac-proxy"
Sep 18 16:49:51.028284 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.028199    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f93bdc0-729b-42b5-9d61-d7eff0a48438" containerName="kube-rbac-proxy"
Sep 18 16:49:51.028284 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.028208    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f93bdc0-729b-42b5-9d61-d7eff0a48438" containerName="kserve-container"
Sep 18 16:49:51.031094 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.031076    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf"
Sep 18 16:49:51.033609 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.033585    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bb388-predictor-serving-cert\""
Sep 18 16:49:51.033716 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.033588    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bb388-kube-rbac-proxy-sar-config\""
Sep 18 16:49:51.039946 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.039922    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf"]
Sep 18 16:49:51.099827 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.099740    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:51.112856 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.112827    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91e82968-e9e7-472c-9412-0277920c713c-proxy-tls\") pod \"success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf\" (UID: \"91e82968-e9e7-472c-9412-0277920c713c\") " pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf"
Sep 18 16:49:51.113039 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.112875    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91e82968-e9e7-472c-9412-0277920c713c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf\" (UID: \"91e82968-e9e7-472c-9412-0277920c713c\") " pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf"
Sep 18 16:49:51.113039 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.112894    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s49x5\" (UniqueName: \"kubernetes.io/projected/91e82968-e9e7-472c-9412-0277920c713c-kube-api-access-s49x5\") pod \"success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf\" (UID: \"91e82968-e9e7-472c-9412-0277920c713c\") " pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf"
Sep 18 16:49:51.200348 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.200315    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:51.213361 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.213333    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91e82968-e9e7-472c-9412-0277920c713c-proxy-tls\") pod \"success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf\" (UID: \"91e82968-e9e7-472c-9412-0277920c713c\") " pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf"
Sep 18 16:49:51.213518 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.213391    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91e82968-e9e7-472c-9412-0277920c713c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf\" (UID: \"91e82968-e9e7-472c-9412-0277920c713c\") " pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf"
Sep 18 16:49:51.213518 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.213447    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s49x5\" (UniqueName: \"kubernetes.io/projected/91e82968-e9e7-472c-9412-0277920c713c-kube-api-access-s49x5\") pod \"success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf\" (UID: \"91e82968-e9e7-472c-9412-0277920c713c\") " pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf"
Sep 18 16:49:51.214183 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.214155    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91e82968-e9e7-472c-9412-0277920c713c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf\" (UID: \"91e82968-e9e7-472c-9412-0277920c713c\") " pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf"
Sep 18 16:49:51.215866 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.215845    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91e82968-e9e7-472c-9412-0277920c713c-proxy-tls\") pod \"success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf\" (UID: \"91e82968-e9e7-472c-9412-0277920c713c\") " pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf"
Sep 18 16:49:51.221733 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.221712    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s49x5\" (UniqueName: \"kubernetes.io/projected/91e82968-e9e7-472c-9412-0277920c713c-kube-api-access-s49x5\") pod \"success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf\" (UID: \"91e82968-e9e7-472c-9412-0277920c713c\") " pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf"
Sep 18 16:49:51.300516 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.300481    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:51.342664 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.342634    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf"
Sep 18 16:49:51.483892 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.483862    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:51.613048 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.613008    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf"]
Sep 18 16:49:51.615820 ip-10-0-135-110 kubenswrapper[2580]: W0918 16:49:51.615790    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e82968_e9e7_472c_9412_0277920c713c.slice/crio-76fab5423a03a8766d658adf3b8e941cc50d32a1c6d2d37514e0499c88c8076b WatchSource:0}: Error finding container 76fab5423a03a8766d658adf3b8e941cc50d32a1c6d2d37514e0499c88c8076b: Status 404 returned error can't find the container with id 76fab5423a03a8766d658adf3b8e941cc50d32a1c6d2d37514e0499c88c8076b
Sep 18 16:49:51.619779 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.619750    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q" podUID="1f9b2d18-1bfc-4ceb-a344-738bd027e59a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 18 16:49:51.976126 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.976091    2580 generic.go:356] "Generic (PLEG): container finished" podID="1f9b2d18-1bfc-4ceb-a344-738bd027e59a" containerID="2bf804d80c258812b669df8683e58df16ba5d36edc6ad5e1f80333da5abb7cd5" exitCode=2
Sep 18 16:49:51.976563 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.976167    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q" event={"ID":"1f9b2d18-1bfc-4ceb-a344-738bd027e59a","Type":"ContainerDied","Data":"2bf804d80c258812b669df8683e58df16ba5d36edc6ad5e1f80333da5abb7cd5"}
Sep 18 16:49:51.977891 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.977869    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf" event={"ID":"91e82968-e9e7-472c-9412-0277920c713c","Type":"ContainerStarted","Data":"ce71ac328c768d2b11593451effe534b66980cb8f80b30f7ed44270f5035776a"}
Sep 18 16:49:51.977999 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.977914    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf" event={"ID":"91e82968-e9e7-472c-9412-0277920c713c","Type":"ContainerStarted","Data":"4068f4ea4b885b06b3de12d9b0e0d9d9a54e65bafce36280085773d7d089d2c0"}
Sep 18 16:49:51.977999 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.977934    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf" event={"ID":"91e82968-e9e7-472c-9412-0277920c713c","Type":"ContainerStarted","Data":"76fab5423a03a8766d658adf3b8e941cc50d32a1c6d2d37514e0499c88c8076b"}
Sep 18 16:49:51.978084 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.978027    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf"
Sep 18 16:49:51.996073 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:51.996027    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf" podStartSLOduration=0.996013821 podStartE2EDuration="996.013821ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:49:51.994228302 +0000 UTC m=+2038.082543943" watchObservedRunningTime="2026-09-18 16:49:51.996013821 +0000 UTC m=+2038.084329461"
Sep 18 16:49:52.982190 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:52.982161    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf"
Sep 18 16:49:52.983346 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:52.983322    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf" podUID="91e82968-e9e7-472c-9412-0277920c713c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 16:49:53.985797 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:53.985759    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf" podUID="91e82968-e9e7-472c-9412-0277920c713c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 16:49:54.350373 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:54.350351    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q"
Sep 18 16:49:54.443332 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:54.443285    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f9b2d18-1bfc-4ceb-a344-738bd027e59a-proxy-tls\") pod \"1f9b2d18-1bfc-4ceb-a344-738bd027e59a\" (UID: \"1f9b2d18-1bfc-4ceb-a344-738bd027e59a\") "
Sep 18 16:49:54.443472 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:54.443387    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sff4\" (UniqueName: \"kubernetes.io/projected/1f9b2d18-1bfc-4ceb-a344-738bd027e59a-kube-api-access-7sff4\") pod \"1f9b2d18-1bfc-4ceb-a344-738bd027e59a\" (UID: \"1f9b2d18-1bfc-4ceb-a344-738bd027e59a\") "
Sep 18 16:49:54.443472 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:54.443407    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f9b2d18-1bfc-4ceb-a344-738bd027e59a-kube-rbac-proxy-sar-config\") pod \"1f9b2d18-1bfc-4ceb-a344-738bd027e59a\" (UID: \"1f9b2d18-1bfc-4ceb-a344-738bd027e59a\") "
Sep 18 16:49:54.443810 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:54.443787    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f9b2d18-1bfc-4ceb-a344-738bd027e59a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1f9b2d18-1bfc-4ceb-a344-738bd027e59a" (UID: "1f9b2d18-1bfc-4ceb-a344-738bd027e59a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:49:54.445346 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:54.445255    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f9b2d18-1bfc-4ceb-a344-738bd027e59a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1f9b2d18-1bfc-4ceb-a344-738bd027e59a" (UID: "1f9b2d18-1bfc-4ceb-a344-738bd027e59a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:49:54.445452 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:54.445368    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f9b2d18-1bfc-4ceb-a344-738bd027e59a-kube-api-access-7sff4" (OuterVolumeSpecName: "kube-api-access-7sff4") pod "1f9b2d18-1bfc-4ceb-a344-738bd027e59a" (UID: "1f9b2d18-1bfc-4ceb-a344-738bd027e59a"). InnerVolumeSpecName "kube-api-access-7sff4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:49:54.544413 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:54.544326    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f9b2d18-1bfc-4ceb-a344-738bd027e59a-proxy-tls\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:49:54.544413 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:54.544359    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sff4\" (UniqueName: \"kubernetes.io/projected/1f9b2d18-1bfc-4ceb-a344-738bd027e59a-kube-api-access-7sff4\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:49:54.544413 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:54.544369    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f9b2d18-1bfc-4ceb-a344-738bd027e59a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:49:54.996663 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:54.996626    2580 generic.go:356] "Generic (PLEG): container finished" podID="1f9b2d18-1bfc-4ceb-a344-738bd027e59a" containerID="71ec1c913bfbfda7ea70467250f12737bd8207975100ab42cbdf4c165c3d463e" exitCode=0
Sep 18 16:49:54.997086 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:54.996684    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q"
Sep 18 16:49:54.997086 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:54.996704    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q" event={"ID":"1f9b2d18-1bfc-4ceb-a344-738bd027e59a","Type":"ContainerDied","Data":"71ec1c913bfbfda7ea70467250f12737bd8207975100ab42cbdf4c165c3d463e"}
Sep 18 16:49:54.997086 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:54.996780    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q" event={"ID":"1f9b2d18-1bfc-4ceb-a344-738bd027e59a","Type":"ContainerDied","Data":"166984defcc52f300c40b18f6f4800a405aaed58dbfd19ae357bbc8888a199f1"}
Sep 18 16:49:54.997086 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:54.996807    2580 scope.go:117] "RemoveContainer" containerID="2bf804d80c258812b669df8683e58df16ba5d36edc6ad5e1f80333da5abb7cd5"
Sep 18 16:49:55.005802 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:55.005787    2580 scope.go:117] "RemoveContainer" containerID="71ec1c913bfbfda7ea70467250f12737bd8207975100ab42cbdf4c165c3d463e"
Sep 18 16:49:55.015006 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:55.014931    2580 scope.go:117] "RemoveContainer" containerID="2bf804d80c258812b669df8683e58df16ba5d36edc6ad5e1f80333da5abb7cd5"
Sep 18 16:49:55.016757 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:49:55.015729    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bf804d80c258812b669df8683e58df16ba5d36edc6ad5e1f80333da5abb7cd5\": container with ID starting with 2bf804d80c258812b669df8683e58df16ba5d36edc6ad5e1f80333da5abb7cd5 not found: ID does not exist" containerID="2bf804d80c258812b669df8683e58df16ba5d36edc6ad5e1f80333da5abb7cd5"
Sep 18 16:49:55.016757 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:55.015769    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf804d80c258812b669df8683e58df16ba5d36edc6ad5e1f80333da5abb7cd5"} err="failed to get container status \"2bf804d80c258812b669df8683e58df16ba5d36edc6ad5e1f80333da5abb7cd5\": rpc error: code = NotFound desc = could not find container \"2bf804d80c258812b669df8683e58df16ba5d36edc6ad5e1f80333da5abb7cd5\": container with ID starting with 2bf804d80c258812b669df8683e58df16ba5d36edc6ad5e1f80333da5abb7cd5 not found: ID does not exist"
Sep 18 16:49:55.016757 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:55.015792    2580 scope.go:117] "RemoveContainer" containerID="71ec1c913bfbfda7ea70467250f12737bd8207975100ab42cbdf4c165c3d463e"
Sep 18 16:49:55.016757 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:49:55.016506    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71ec1c913bfbfda7ea70467250f12737bd8207975100ab42cbdf4c165c3d463e\": container with ID starting with 71ec1c913bfbfda7ea70467250f12737bd8207975100ab42cbdf4c165c3d463e not found: ID does not exist" containerID="71ec1c913bfbfda7ea70467250f12737bd8207975100ab42cbdf4c165c3d463e"
Sep 18 16:49:55.016757 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:55.016533    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ec1c913bfbfda7ea70467250f12737bd8207975100ab42cbdf4c165c3d463e"} err="failed to get container status \"71ec1c913bfbfda7ea70467250f12737bd8207975100ab42cbdf4c165c3d463e\": rpc error: code = NotFound desc = could not find container \"71ec1c913bfbfda7ea70467250f12737bd8207975100ab42cbdf4c165c3d463e\": container with ID starting with 71ec1c913bfbfda7ea70467250f12737bd8207975100ab42cbdf4c165c3d463e not found: ID does not exist"
Sep 18 16:49:55.018494 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:55.018473    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q"]
Sep 18 16:49:55.021753 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:55.021731    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b3de6-predictor-5f84b875fc-wwp5q"]
Sep 18 16:49:56.123673 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:56.123641    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f9b2d18-1bfc-4ceb-a344-738bd027e59a" path="/var/lib/kubelet/pods/1f9b2d18-1bfc-4ceb-a344-738bd027e59a/volumes"
Sep 18 16:49:57.810419 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:57.810387    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26" podUID="53c2ad28-054c-4eeb-9b70-1136a0c5cdfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 16:49:58.990441 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:58.990414    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf"
Sep 18 16:49:58.990864 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:49:58.990842    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf" podUID="91e82968-e9e7-472c-9412-0277920c713c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 16:50:07.811534 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:50:07.811499    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26"
Sep 18 16:50:08.990993 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:50:08.990953    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf" podUID="91e82968-e9e7-472c-9412-0277920c713c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 16:50:18.991116 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:50:18.991074    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf" podUID="91e82968-e9e7-472c-9412-0277920c713c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 16:50:21.048783 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:50:21.048746    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:50:21.342262 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:50:21.342175    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:50:28.990951 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:50:28.990913    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf" podUID="91e82968-e9e7-472c-9412-0277920c713c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 16:50:38.991098 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:50:38.991053    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf" podUID="91e82968-e9e7-472c-9412-0277920c713c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 16:50:48.991464 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:50:48.991433    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf"
Sep 18 16:51:01.031181 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:51:01.031145    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:51:01.133213 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:51:01.133180    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:52:48.134239 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:52:48.134208    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:52:48.191431 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:52:48.191382    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:52:59.727919 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:52:59.727886    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:56:46.216905 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:56:46.216836    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:59:16.122607 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:16.122522    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:59:16.194449 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:16.194411    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf"]
Sep 18 16:59:16.194692 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:16.194651    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf" podUID="91e82968-e9e7-472c-9412-0277920c713c" containerName="kserve-container" containerID="cri-o://4068f4ea4b885b06b3de12d9b0e0d9d9a54e65bafce36280085773d7d089d2c0" gracePeriod=30
Sep 18 16:59:16.194771 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:16.194682    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf" podUID="91e82968-e9e7-472c-9412-0277920c713c" containerName="kube-rbac-proxy" containerID="cri-o://ce71ac328c768d2b11593451effe534b66980cb8f80b30f7ed44270f5035776a" gracePeriod=30
Sep 18 16:59:16.214234 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:16.214206    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:59:16.310253 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:16.310216    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 16:59:16.341007 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:16.340975    2580 generic.go:356] "Generic (PLEG): container finished" podID="91e82968-e9e7-472c-9412-0277920c713c" containerID="ce71ac328c768d2b11593451effe534b66980cb8f80b30f7ed44270f5035776a" exitCode=2
Sep 18 16:59:16.341160 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:16.341052    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf" event={"ID":"91e82968-e9e7-472c-9412-0277920c713c","Type":"ContainerDied","Data":"ce71ac328c768d2b11593451effe534b66980cb8f80b30f7ed44270f5035776a"}
Sep 18 16:59:18.986898 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:18.986853    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf" podUID="91e82968-e9e7-472c-9412-0277920c713c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 18 16:59:18.991188 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:18.991157    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf" podUID="91e82968-e9e7-472c-9412-0277920c713c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 16:59:19.955372 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:19.955285    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf"
Sep 18 16:59:20.027059 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:20.026954    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s49x5\" (UniqueName: \"kubernetes.io/projected/91e82968-e9e7-472c-9412-0277920c713c-kube-api-access-s49x5\") pod \"91e82968-e9e7-472c-9412-0277920c713c\" (UID: \"91e82968-e9e7-472c-9412-0277920c713c\") "
Sep 18 16:59:20.027059 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:20.027044    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91e82968-e9e7-472c-9412-0277920c713c-proxy-tls\") pod \"91e82968-e9e7-472c-9412-0277920c713c\" (UID: \"91e82968-e9e7-472c-9412-0277920c713c\") "
Sep 18 16:59:20.027616 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:20.027107    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91e82968-e9e7-472c-9412-0277920c713c-kube-rbac-proxy-sar-config\") pod \"91e82968-e9e7-472c-9412-0277920c713c\" (UID: \"91e82968-e9e7-472c-9412-0277920c713c\") "
Sep 18 16:59:20.027616 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:20.027512    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e82968-e9e7-472c-9412-0277920c713c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "91e82968-e9e7-472c-9412-0277920c713c" (UID: "91e82968-e9e7-472c-9412-0277920c713c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:59:20.029197 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:20.029171    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91e82968-e9e7-472c-9412-0277920c713c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "91e82968-e9e7-472c-9412-0277920c713c" (UID: "91e82968-e9e7-472c-9412-0277920c713c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:59:20.029274 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:20.029204    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e82968-e9e7-472c-9412-0277920c713c-kube-api-access-s49x5" (OuterVolumeSpecName: "kube-api-access-s49x5") pod "91e82968-e9e7-472c-9412-0277920c713c" (UID: "91e82968-e9e7-472c-9412-0277920c713c"). InnerVolumeSpecName "kube-api-access-s49x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:59:20.127847 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:20.127809    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s49x5\" (UniqueName: \"kubernetes.io/projected/91e82968-e9e7-472c-9412-0277920c713c-kube-api-access-s49x5\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:59:20.127847 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:20.127845    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91e82968-e9e7-472c-9412-0277920c713c-proxy-tls\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:59:20.128073 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:20.127860    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91e82968-e9e7-472c-9412-0277920c713c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 16:59:20.358708 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:20.358611    2580 generic.go:356] "Generic (PLEG): container finished" podID="91e82968-e9e7-472c-9412-0277920c713c" containerID="4068f4ea4b885b06b3de12d9b0e0d9d9a54e65bafce36280085773d7d089d2c0" exitCode=0
Sep 18 16:59:20.358708 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:20.358687    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf"
Sep 18 16:59:20.358708 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:20.358698    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf" event={"ID":"91e82968-e9e7-472c-9412-0277920c713c","Type":"ContainerDied","Data":"4068f4ea4b885b06b3de12d9b0e0d9d9a54e65bafce36280085773d7d089d2c0"}
Sep 18 16:59:20.358990 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:20.358755    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf" event={"ID":"91e82968-e9e7-472c-9412-0277920c713c","Type":"ContainerDied","Data":"76fab5423a03a8766d658adf3b8e941cc50d32a1c6d2d37514e0499c88c8076b"}
Sep 18 16:59:20.358990 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:20.358781    2580 scope.go:117] "RemoveContainer" containerID="ce71ac328c768d2b11593451effe534b66980cb8f80b30f7ed44270f5035776a"
Sep 18 16:59:20.367736 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:20.367691    2580 scope.go:117] "RemoveContainer" containerID="4068f4ea4b885b06b3de12d9b0e0d9d9a54e65bafce36280085773d7d089d2c0"
Sep 18 16:59:20.376270 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:20.376247    2580 scope.go:117] "RemoveContainer" containerID="ce71ac328c768d2b11593451effe534b66980cb8f80b30f7ed44270f5035776a"
Sep 18 16:59:20.376447 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:20.376424    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf"]
Sep 18 16:59:20.376618 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:59:20.376599    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce71ac328c768d2b11593451effe534b66980cb8f80b30f7ed44270f5035776a\": container with ID starting with ce71ac328c768d2b11593451effe534b66980cb8f80b30f7ed44270f5035776a not found: ID does not exist" containerID="ce71ac328c768d2b11593451effe534b66980cb8f80b30f7ed44270f5035776a"
Sep 18 16:59:20.376661 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:20.376628    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce71ac328c768d2b11593451effe534b66980cb8f80b30f7ed44270f5035776a"} err="failed to get container status \"ce71ac328c768d2b11593451effe534b66980cb8f80b30f7ed44270f5035776a\": rpc error: code = NotFound desc = could not find container \"ce71ac328c768d2b11593451effe534b66980cb8f80b30f7ed44270f5035776a\": container with ID starting with ce71ac328c768d2b11593451effe534b66980cb8f80b30f7ed44270f5035776a not found: ID does not exist"
Sep 18 16:59:20.376661 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:20.376651    2580 scope.go:117] "RemoveContainer" containerID="4068f4ea4b885b06b3de12d9b0e0d9d9a54e65bafce36280085773d7d089d2c0"
Sep 18 16:59:20.376906 ip-10-0-135-110 kubenswrapper[2580]: E0918 16:59:20.376883    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4068f4ea4b885b06b3de12d9b0e0d9d9a54e65bafce36280085773d7d089d2c0\": container with ID starting with 4068f4ea4b885b06b3de12d9b0e0d9d9a54e65bafce36280085773d7d089d2c0 not found: ID does not exist" containerID="4068f4ea4b885b06b3de12d9b0e0d9d9a54e65bafce36280085773d7d089d2c0"
Sep 18 16:59:20.376943 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:20.376912    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4068f4ea4b885b06b3de12d9b0e0d9d9a54e65bafce36280085773d7d089d2c0"} err="failed to get container status \"4068f4ea4b885b06b3de12d9b0e0d9d9a54e65bafce36280085773d7d089d2c0\": rpc error: code = NotFound desc = could not find container \"4068f4ea4b885b06b3de12d9b0e0d9d9a54e65bafce36280085773d7d089d2c0\": container with ID starting with 4068f4ea4b885b06b3de12d9b0e0d9d9a54e65bafce36280085773d7d089d2c0 not found: ID does not exist"
Sep 18 16:59:20.381958 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:20.381933    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bb388-predictor-7fcb976d9f-p9fmf"]
Sep 18 16:59:22.121252 ip-10-0-135-110 kubenswrapper[2580]: I0918 16:59:22.121213    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91e82968-e9e7-472c-9412-0277920c713c" path="/var/lib/kubelet/pods/91e82968-e9e7-472c-9412-0277920c713c/volumes"
Sep 18 17:02:32.338615 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:02:32.338503    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 17:02:47.436473 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:02:47.436438    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 17:06:41.176192 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:41.176094    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 17:06:41.252632 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:41.252589    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26"]
Sep 18 17:06:41.252893 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:41.252839    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26" podUID="53c2ad28-054c-4eeb-9b70-1136a0c5cdfd" containerName="kserve-container" containerID="cri-o://fcf870ad04a8d2d155d2c23c1051a5648c175b45cdbe56a5bb350f152cb75d02" gracePeriod=30
Sep 18 17:06:41.253055 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:41.252966    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26" podUID="53c2ad28-054c-4eeb-9b70-1136a0c5cdfd" containerName="kube-rbac-proxy" containerID="cri-o://262e191480634c7cd11cefba5b17786818bb495ddd51159c88f11d885deda619" gracePeriod=30
Sep 18 17:06:41.277381 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:41.277348    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 17:06:41.378263 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:41.378232    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 17:06:42.040389 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.040342    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qvvt2/must-gather-vcslv"]
Sep 18 17:06:42.040731 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.040717    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91e82968-e9e7-472c-9412-0277920c713c" containerName="kube-rbac-proxy"
Sep 18 17:06:42.040789 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.040733    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="91e82968-e9e7-472c-9412-0277920c713c" containerName="kube-rbac-proxy"
Sep 18 17:06:42.040789 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.040751    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91e82968-e9e7-472c-9412-0277920c713c" containerName="kserve-container"
Sep 18 17:06:42.040789 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.040757    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="91e82968-e9e7-472c-9412-0277920c713c" containerName="kserve-container"
Sep 18 17:06:42.040789 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.040765    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f9b2d18-1bfc-4ceb-a344-738bd027e59a" containerName="kube-rbac-proxy"
Sep 18 17:06:42.040789 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.040771    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f9b2d18-1bfc-4ceb-a344-738bd027e59a" containerName="kube-rbac-proxy"
Sep 18 17:06:42.040789 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.040778    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f9b2d18-1bfc-4ceb-a344-738bd027e59a" containerName="kserve-container"
Sep 18 17:06:42.040789 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.040785    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f9b2d18-1bfc-4ceb-a344-738bd027e59a" containerName="kserve-container"
Sep 18 17:06:42.041004 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.040851    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="91e82968-e9e7-472c-9412-0277920c713c" containerName="kserve-container"
Sep 18 17:06:42.041004 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.040859    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="91e82968-e9e7-472c-9412-0277920c713c" containerName="kube-rbac-proxy"
Sep 18 17:06:42.041004 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.040874    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f9b2d18-1bfc-4ceb-a344-738bd027e59a" containerName="kserve-container"
Sep 18 17:06:42.041004 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.040882    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f9b2d18-1bfc-4ceb-a344-738bd027e59a" containerName="kube-rbac-proxy"
Sep 18 17:06:42.043971 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.043951    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qvvt2/must-gather-vcslv"
Sep 18 17:06:42.046767 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.046746    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qvvt2\"/\"kube-root-ca.crt\""
Sep 18 17:06:42.047022 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.047005    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qvvt2\"/\"default-dockercfg-x7kwp\""
Sep 18 17:06:42.047102 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.047008    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qvvt2\"/\"openshift-service-ca.crt\""
Sep 18 17:06:42.053775 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.053754    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qvvt2/must-gather-vcslv"]
Sep 18 17:06:42.067689 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.067658    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 17:06:42.140732 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.140704    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz27z\" (UniqueName: \"kubernetes.io/projected/0f8351d8-160e-4a16-9969-a021798ec977-kube-api-access-zz27z\") pod \"must-gather-vcslv\" (UID: \"0f8351d8-160e-4a16-9969-a021798ec977\") " pod="openshift-must-gather-qvvt2/must-gather-vcslv"
Sep 18 17:06:42.140867 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.140768    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0f8351d8-160e-4a16-9969-a021798ec977-must-gather-output\") pod \"must-gather-vcslv\" (UID: \"0f8351d8-160e-4a16-9969-a021798ec977\") " pod="openshift-must-gather-qvvt2/must-gather-vcslv"
Sep 18 17:06:42.170151 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.170129    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 17:06:42.177821 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.177785    2580 generic.go:356] "Generic (PLEG): container finished" podID="53c2ad28-054c-4eeb-9b70-1136a0c5cdfd" containerID="262e191480634c7cd11cefba5b17786818bb495ddd51159c88f11d885deda619" exitCode=2
Sep 18 17:06:42.178119 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.177833    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26" event={"ID":"53c2ad28-054c-4eeb-9b70-1136a0c5cdfd","Type":"ContainerDied","Data":"262e191480634c7cd11cefba5b17786818bb495ddd51159c88f11d885deda619"}
Sep 18 17:06:42.241160 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.241129    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz27z\" (UniqueName: \"kubernetes.io/projected/0f8351d8-160e-4a16-9969-a021798ec977-kube-api-access-zz27z\") pod \"must-gather-vcslv\" (UID: \"0f8351d8-160e-4a16-9969-a021798ec977\") " pod="openshift-must-gather-qvvt2/must-gather-vcslv"
Sep 18 17:06:42.241295 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.241219    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0f8351d8-160e-4a16-9969-a021798ec977-must-gather-output\") pod \"must-gather-vcslv\" (UID: \"0f8351d8-160e-4a16-9969-a021798ec977\") " pod="openshift-must-gather-qvvt2/must-gather-vcslv"
Sep 18 17:06:42.241590 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.241574    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0f8351d8-160e-4a16-9969-a021798ec977-must-gather-output\") pod \"must-gather-vcslv\" (UID: \"0f8351d8-160e-4a16-9969-a021798ec977\") " pod="openshift-must-gather-qvvt2/must-gather-vcslv"
Sep 18 17:06:42.251469 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.251449    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz27z\" (UniqueName: \"kubernetes.io/projected/0f8351d8-160e-4a16-9969-a021798ec977-kube-api-access-zz27z\") pod \"must-gather-vcslv\" (UID: \"0f8351d8-160e-4a16-9969-a021798ec977\") " pod="openshift-must-gather-qvvt2/must-gather-vcslv"
Sep 18 17:06:42.362671 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.362595    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qvvt2/must-gather-vcslv"
Sep 18 17:06:42.634602 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.634576    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qvvt2/must-gather-vcslv"]
Sep 18 17:06:42.636622 ip-10-0-135-110 kubenswrapper[2580]: W0918 17:06:42.636588    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f8351d8_160e_4a16_9969_a021798ec977.slice/crio-6b3a8716c5188ccd9b720b0cba5e698e8e3a6fa14e8faf22c994ddb8ecf3f910 WatchSource:0}: Error finding container 6b3a8716c5188ccd9b720b0cba5e698e8e3a6fa14e8faf22c994ddb8ecf3f910: Status 404 returned error can't find the container with id 6b3a8716c5188ccd9b720b0cba5e698e8e3a6fa14e8faf22c994ddb8ecf3f910
Sep 18 17:06:42.638287 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.638271    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 17:06:42.805473 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:42.805441    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26" podUID="53c2ad28-054c-4eeb-9b70-1136a0c5cdfd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 18 17:06:43.187782 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:43.187711    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qvvt2/must-gather-vcslv" event={"ID":"0f8351d8-160e-4a16-9969-a021798ec977","Type":"ContainerStarted","Data":"6b3a8716c5188ccd9b720b0cba5e698e8e3a6fa14e8faf22c994ddb8ecf3f910"}
Sep 18 17:06:47.212765 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:47.212724    2580 generic.go:356] "Generic (PLEG): container finished" podID="53c2ad28-054c-4eeb-9b70-1136a0c5cdfd" containerID="fcf870ad04a8d2d155d2c23c1051a5648c175b45cdbe56a5bb350f152cb75d02" exitCode=0
Sep 18 17:06:47.213208 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:47.212776    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26" event={"ID":"53c2ad28-054c-4eeb-9b70-1136a0c5cdfd","Type":"ContainerDied","Data":"fcf870ad04a8d2d155d2c23c1051a5648c175b45cdbe56a5bb350f152cb75d02"}
Sep 18 17:06:47.751752 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:47.751717    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26"
Sep 18 17:06:47.796291 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:47.796253    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53c2ad28-054c-4eeb-9b70-1136a0c5cdfd-proxy-tls\") pod \"53c2ad28-054c-4eeb-9b70-1136a0c5cdfd\" (UID: \"53c2ad28-054c-4eeb-9b70-1136a0c5cdfd\") "
Sep 18 17:06:47.796491 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:47.796319    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s75dt\" (UniqueName: \"kubernetes.io/projected/53c2ad28-054c-4eeb-9b70-1136a0c5cdfd-kube-api-access-s75dt\") pod \"53c2ad28-054c-4eeb-9b70-1136a0c5cdfd\" (UID: \"53c2ad28-054c-4eeb-9b70-1136a0c5cdfd\") "
Sep 18 17:06:47.796491 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:47.796356    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53c2ad28-054c-4eeb-9b70-1136a0c5cdfd-kube-rbac-proxy-sar-config\") pod \"53c2ad28-054c-4eeb-9b70-1136a0c5cdfd\" (UID: \"53c2ad28-054c-4eeb-9b70-1136a0c5cdfd\") "
Sep 18 17:06:47.796912 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:47.796882    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53c2ad28-054c-4eeb-9b70-1136a0c5cdfd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "53c2ad28-054c-4eeb-9b70-1136a0c5cdfd" (UID: "53c2ad28-054c-4eeb-9b70-1136a0c5cdfd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:06:47.798798 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:47.798752    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53c2ad28-054c-4eeb-9b70-1136a0c5cdfd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "53c2ad28-054c-4eeb-9b70-1136a0c5cdfd" (UID: "53c2ad28-054c-4eeb-9b70-1136a0c5cdfd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:06:47.798798 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:47.798760    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53c2ad28-054c-4eeb-9b70-1136a0c5cdfd-kube-api-access-s75dt" (OuterVolumeSpecName: "kube-api-access-s75dt") pod "53c2ad28-054c-4eeb-9b70-1136a0c5cdfd" (UID: "53c2ad28-054c-4eeb-9b70-1136a0c5cdfd"). InnerVolumeSpecName "kube-api-access-s75dt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:06:47.897248 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:47.897206    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53c2ad28-054c-4eeb-9b70-1136a0c5cdfd-proxy-tls\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 17:06:47.897248 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:47.897239    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s75dt\" (UniqueName: \"kubernetes.io/projected/53c2ad28-054c-4eeb-9b70-1136a0c5cdfd-kube-api-access-s75dt\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 17:06:47.897248 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:47.897253    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53c2ad28-054c-4eeb-9b70-1136a0c5cdfd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 17:06:48.219593 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:48.219559    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26"
Sep 18 17:06:48.220052 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:48.219587    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26" event={"ID":"53c2ad28-054c-4eeb-9b70-1136a0c5cdfd","Type":"ContainerDied","Data":"ffcec94e2b49a86ed6d51a665ae59fa4eda4ab5fe303d0d9106ccbf683db10bc"}
Sep 18 17:06:48.220052 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:48.219661    2580 scope.go:117] "RemoveContainer" containerID="262e191480634c7cd11cefba5b17786818bb495ddd51159c88f11d885deda619"
Sep 18 17:06:48.238361 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:48.238334    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26"]
Sep 18 17:06:48.242315 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:48.242274    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a2696-predictor-6d48d4c789-h4v26"]
Sep 18 17:06:48.670240 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:48.670207    2580 scope.go:117] "RemoveContainer" containerID="fcf870ad04a8d2d155d2c23c1051a5648c175b45cdbe56a5bb350f152cb75d02"
Sep 18 17:06:49.227382 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:49.227281    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qvvt2/must-gather-vcslv" event={"ID":"0f8351d8-160e-4a16-9969-a021798ec977","Type":"ContainerStarted","Data":"7cd2e38f55aef6cacae13e9d687e512805ef4b74e2fd011518f96a0fc8a3ccc6"}
Sep 18 17:06:49.227382 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:49.227344    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qvvt2/must-gather-vcslv" event={"ID":"0f8351d8-160e-4a16-9969-a021798ec977","Type":"ContainerStarted","Data":"9abd8c6744290eba178c1792f37d73f9104d869bc04ed5896cee376e6ea7f1c3"}
Sep 18 17:06:49.244235 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:49.244158    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qvvt2/must-gather-vcslv" podStartSLOduration=1.141610655 podStartE2EDuration="7.244136005s" totalImagesPullingTime="6.10252535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:06:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:06:49.242842056 +0000 UTC m=+3055.331157698" watchObservedRunningTime="2026-09-18 17:06:49.244136005 +0000 UTC m=+3055.332451647"
Sep 18 17:06:50.124014 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:06:50.123984    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53c2ad28-054c-4eeb-9b70-1136a0c5cdfd" path="/var/lib/kubelet/pods/53c2ad28-054c-4eeb-9b70-1136a0c5cdfd/volumes"
Sep 18 17:07:11.331130 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:11.331034    2580 generic.go:356] "Generic (PLEG): container finished" podID="0f8351d8-160e-4a16-9969-a021798ec977" containerID="9abd8c6744290eba178c1792f37d73f9104d869bc04ed5896cee376e6ea7f1c3" exitCode=0
Sep 18 17:07:11.331130 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:11.331106    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qvvt2/must-gather-vcslv" event={"ID":"0f8351d8-160e-4a16-9969-a021798ec977","Type":"ContainerDied","Data":"9abd8c6744290eba178c1792f37d73f9104d869bc04ed5896cee376e6ea7f1c3"}
Sep 18 17:07:11.331721 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:11.331560    2580 scope.go:117] "RemoveContainer" containerID="9abd8c6744290eba178c1792f37d73f9104d869bc04ed5896cee376e6ea7f1c3"
Sep 18 17:07:11.975531 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:11.975497    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-qvvt2_must-gather-vcslv_0f8351d8-160e-4a16-9969-a021798ec977/gather/0.log"
Sep 18 17:07:12.408368 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:12.408223    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 17:07:12.535873 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:12.535833    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zpw2m/must-gather-t7p4m"]
Sep 18 17:07:12.536201 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:12.536187    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53c2ad28-054c-4eeb-9b70-1136a0c5cdfd" containerName="kserve-container"
Sep 18 17:07:12.536241 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:12.536203    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="53c2ad28-054c-4eeb-9b70-1136a0c5cdfd" containerName="kserve-container"
Sep 18 17:07:12.536241 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:12.536222    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53c2ad28-054c-4eeb-9b70-1136a0c5cdfd" containerName="kube-rbac-proxy"
Sep 18 17:07:12.536241 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:12.536228    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="53c2ad28-054c-4eeb-9b70-1136a0c5cdfd" containerName="kube-rbac-proxy"
Sep 18 17:07:12.536373 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:12.536290    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="53c2ad28-054c-4eeb-9b70-1136a0c5cdfd" containerName="kserve-container"
Sep 18 17:07:12.536373 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:12.536312    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="53c2ad28-054c-4eeb-9b70-1136a0c5cdfd" containerName="kube-rbac-proxy"
Sep 18 17:07:12.538640 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:12.538618    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zpw2m/must-gather-t7p4m"
Sep 18 17:07:12.541604 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:12.541523    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zpw2m\"/\"default-dockercfg-2hq6t\""
Sep 18 17:07:12.541722 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:12.541676    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zpw2m\"/\"openshift-service-ca.crt\""
Sep 18 17:07:12.542205 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:12.542187    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zpw2m\"/\"kube-root-ca.crt\""
Sep 18 17:07:12.549780 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:12.549757    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zpw2m/must-gather-t7p4m"]
Sep 18 17:07:12.562362 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:12.562339    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 17:07:12.628389 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:12.628351    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtt2q\" (UniqueName: \"kubernetes.io/projected/be592925-9eff-4561-8bb3-5cb03405f0cd-kube-api-access-qtt2q\") pod \"must-gather-t7p4m\" (UID: \"be592925-9eff-4561-8bb3-5cb03405f0cd\") " pod="openshift-must-gather-zpw2m/must-gather-t7p4m"
Sep 18 17:07:12.628559 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:12.628407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/be592925-9eff-4561-8bb3-5cb03405f0cd-must-gather-output\") pod \"must-gather-t7p4m\" (UID: \"be592925-9eff-4561-8bb3-5cb03405f0cd\") " pod="openshift-must-gather-zpw2m/must-gather-t7p4m"
Sep 18 17:07:12.660975 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:12.660908    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 17:07:12.728874 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:12.728836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/be592925-9eff-4561-8bb3-5cb03405f0cd-must-gather-output\") pod \"must-gather-t7p4m\" (UID: \"be592925-9eff-4561-8bb3-5cb03405f0cd\") " pod="openshift-must-gather-zpw2m/must-gather-t7p4m"
Sep 18 17:07:12.728976 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:12.728915    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtt2q\" (UniqueName: \"kubernetes.io/projected/be592925-9eff-4561-8bb3-5cb03405f0cd-kube-api-access-qtt2q\") pod \"must-gather-t7p4m\" (UID: \"be592925-9eff-4561-8bb3-5cb03405f0cd\") " pod="openshift-must-gather-zpw2m/must-gather-t7p4m"
Sep 18 17:07:12.729238 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:12.729215    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/be592925-9eff-4561-8bb3-5cb03405f0cd-must-gather-output\") pod \"must-gather-t7p4m\" (UID: \"be592925-9eff-4561-8bb3-5cb03405f0cd\") " pod="openshift-must-gather-zpw2m/must-gather-t7p4m"
Sep 18 17:07:12.737553 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:12.737522    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtt2q\" (UniqueName: \"kubernetes.io/projected/be592925-9eff-4561-8bb3-5cb03405f0cd-kube-api-access-qtt2q\") pod \"must-gather-t7p4m\" (UID: \"be592925-9eff-4561-8bb3-5cb03405f0cd\") " pod="openshift-must-gather-zpw2m/must-gather-t7p4m"
Sep 18 17:07:12.847389 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:12.847350    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zpw2m/must-gather-t7p4m"
Sep 18 17:07:13.115870 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:13.115843    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zpw2m/must-gather-t7p4m"]
Sep 18 17:07:13.118701 ip-10-0-135-110 kubenswrapper[2580]: W0918 17:07:13.118667    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe592925_9eff_4561_8bb3_5cb03405f0cd.slice/crio-8daf164bc33a1caa80edbbb8f550d4383f6a3cc614a2f62e09c4c7402ae17ba8 WatchSource:0}: Error finding container 8daf164bc33a1caa80edbbb8f550d4383f6a3cc614a2f62e09c4c7402ae17ba8: Status 404 returned error can't find the container with id 8daf164bc33a1caa80edbbb8f550d4383f6a3cc614a2f62e09c4c7402ae17ba8
Sep 18 17:07:13.340311 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:13.340258    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zpw2m/must-gather-t7p4m" event={"ID":"be592925-9eff-4561-8bb3-5cb03405f0cd","Type":"ContainerStarted","Data":"8daf164bc33a1caa80edbbb8f550d4383f6a3cc614a2f62e09c4c7402ae17ba8"}
Sep 18 17:07:14.348000 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:14.347963    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zpw2m/must-gather-t7p4m" event={"ID":"be592925-9eff-4561-8bb3-5cb03405f0cd","Type":"ContainerStarted","Data":"e88578edd204b925fde85037f00bc037eaba2887323965958b91fe5da4488e12"}
Sep 18 17:07:14.348389 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:14.348013    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zpw2m/must-gather-t7p4m" event={"ID":"be592925-9eff-4561-8bb3-5cb03405f0cd","Type":"ContainerStarted","Data":"36a52ef1ef7a445eebbcc43e3b5dd4f5b10beaf8bf5b1b0c0401835e2fdda4e8"}
Sep 18 17:07:14.367982 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:14.367913    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zpw2m/must-gather-t7p4m" podStartSLOduration=1.6846049619999999 podStartE2EDuration="2.367888268s" totalImagesPullingTime="683.283306ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:07:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:07:14.36493527 +0000 UTC m=+3080.453250912" watchObservedRunningTime="2026-09-18 17:07:14.367888268 +0000 UTC m=+3080.456203913"
Sep 18 17:07:15.223216 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:15.223176    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cmscw_97bef555-d008-4f5b-a974-91fbafd1eeb1/global-pull-secret-syncer/0.log"
Sep 18 17:07:15.428500 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:15.428453    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log"
Sep 18 17:07:15.614044 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:15.613944    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-110.ec2.internal_b8e36c319ff90245c5f11900bd31f47e/haproxy/0.log"
Sep 18 17:07:17.394164 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:17.394122    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-qvvt2/must-gather-vcslv"]
Sep 18 17:07:17.395109 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:17.395030    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-qvvt2/must-gather-vcslv" podUID="0f8351d8-160e-4a16-9969-a021798ec977" containerName="copy" containerID="cri-o://7cd2e38f55aef6cacae13e9d687e512805ef4b74e2fd011518f96a0fc8a3ccc6" gracePeriod=2
Sep 18 17:07:17.397311 ip-10-0-135-110 kubenswrapper[2580]: E0918 17:07:17.397266    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-vcslv\" is forbidden: User \"system:node:ip-10-0-135-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-qvvt2\": no relationship found between node 'ip-10-0-135-110.ec2.internal' and this object" podUID="0f8351d8-160e-4a16-9969-a021798ec977" pod="openshift-must-gather-qvvt2/must-gather-vcslv"
Sep 18 17:07:17.398089 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:17.398064    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-qvvt2/must-gather-vcslv"]
Sep 18 17:07:17.766914 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:17.766884    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-qvvt2_must-gather-vcslv_0f8351d8-160e-4a16-9969-a021798ec977/copy/0.log"
Sep 18 17:07:17.770067 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:17.770035    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qvvt2/must-gather-vcslv"
Sep 18 17:07:17.773816 ip-10-0-135-110 kubenswrapper[2580]: E0918 17:07:17.773749    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-vcslv\" is forbidden: User \"system:node:ip-10-0-135-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-qvvt2\": no relationship found between node 'ip-10-0-135-110.ec2.internal' and this object" podUID="0f8351d8-160e-4a16-9969-a021798ec977" pod="openshift-must-gather-qvvt2/must-gather-vcslv"
Sep 18 17:07:17.890797 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:17.888350    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zz27z\" (UniqueName: \"kubernetes.io/projected/0f8351d8-160e-4a16-9969-a021798ec977-kube-api-access-zz27z\") pod \"0f8351d8-160e-4a16-9969-a021798ec977\" (UID: \"0f8351d8-160e-4a16-9969-a021798ec977\") "
Sep 18 17:07:17.890797 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:17.888431    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0f8351d8-160e-4a16-9969-a021798ec977-must-gather-output\") pod \"0f8351d8-160e-4a16-9969-a021798ec977\" (UID: \"0f8351d8-160e-4a16-9969-a021798ec977\") "
Sep 18 17:07:17.890797 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:17.890148    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8351d8-160e-4a16-9969-a021798ec977-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "0f8351d8-160e-4a16-9969-a021798ec977" (UID: "0f8351d8-160e-4a16-9969-a021798ec977"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:07:17.893008 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:17.892969    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f8351d8-160e-4a16-9969-a021798ec977-kube-api-access-zz27z" (OuterVolumeSpecName: "kube-api-access-zz27z") pod "0f8351d8-160e-4a16-9969-a021798ec977" (UID: "0f8351d8-160e-4a16-9969-a021798ec977"). InnerVolumeSpecName "kube-api-access-zz27z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:07:17.990367 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:17.990221    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz27z\" (UniqueName: \"kubernetes.io/projected/0f8351d8-160e-4a16-9969-a021798ec977-kube-api-access-zz27z\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 17:07:17.990367 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:17.990267    2580 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0f8351d8-160e-4a16-9969-a021798ec977-must-gather-output\") on node \"ip-10-0-135-110.ec2.internal\" DevicePath \"\""
Sep 18 17:07:18.020033 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:18.019866    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 17:07:18.093325 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:18.093170    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k6gdw_8c726ad7-c84f-4629-9797-52cfa3327996/konnectivity-agent/0.log" limit=1024
Sep 18 17:07:18.125716 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:18.125641    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f8351d8-160e-4a16-9969-a021798ec977" path="/var/lib/kubelet/pods/0f8351d8-160e-4a16-9969-a021798ec977/volumes"
Sep 18 17:07:18.371584 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:18.371199    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-qvvt2_must-gather-vcslv_0f8351d8-160e-4a16-9969-a021798ec977/copy/0.log"
Sep 18 17:07:18.371991 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:18.371934    2580 generic.go:356] "Generic (PLEG): container finished" podID="0f8351d8-160e-4a16-9969-a021798ec977" containerID="7cd2e38f55aef6cacae13e9d687e512805ef4b74e2fd011518f96a0fc8a3ccc6" exitCode=143
Sep 18 17:07:18.372240 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:18.372174    2580 scope.go:117] "RemoveContainer" containerID="7cd2e38f55aef6cacae13e9d687e512805ef4b74e2fd011518f96a0fc8a3ccc6"
Sep 18 17:07:18.372547 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:18.372473    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qvvt2/must-gather-vcslv"
Sep 18 17:07:18.392726 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:18.392129    2580 scope.go:117] "RemoveContainer" containerID="9abd8c6744290eba178c1792f37d73f9104d869bc04ed5896cee376e6ea7f1c3"
Sep 18 17:07:18.414476 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:18.414290    2580 scope.go:117] "RemoveContainer" containerID="7cd2e38f55aef6cacae13e9d687e512805ef4b74e2fd011518f96a0fc8a3ccc6"
Sep 18 17:07:18.414937 ip-10-0-135-110 kubenswrapper[2580]: E0918 17:07:18.414784    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cd2e38f55aef6cacae13e9d687e512805ef4b74e2fd011518f96a0fc8a3ccc6\": container with ID starting with 7cd2e38f55aef6cacae13e9d687e512805ef4b74e2fd011518f96a0fc8a3ccc6 not found: ID does not exist" containerID="7cd2e38f55aef6cacae13e9d687e512805ef4b74e2fd011518f96a0fc8a3ccc6"
Sep 18 17:07:18.414937 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:18.414826    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cd2e38f55aef6cacae13e9d687e512805ef4b74e2fd011518f96a0fc8a3ccc6"} err="failed to get container status \"7cd2e38f55aef6cacae13e9d687e512805ef4b74e2fd011518f96a0fc8a3ccc6\": rpc error: code = NotFound desc = could not find container \"7cd2e38f55aef6cacae13e9d687e512805ef4b74e2fd011518f96a0fc8a3ccc6\": container with ID starting with 7cd2e38f55aef6cacae13e9d687e512805ef4b74e2fd011518f96a0fc8a3ccc6 not found: ID does not exist"
Sep 18 17:07:18.414937 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:18.414853    2580 scope.go:117] "RemoveContainer" containerID="9abd8c6744290eba178c1792f37d73f9104d869bc04ed5896cee376e6ea7f1c3"
Sep 18 17:07:18.415284 ip-10-0-135-110 kubenswrapper[2580]: E0918 17:07:18.415199    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9abd8c6744290eba178c1792f37d73f9104d869bc04ed5896cee376e6ea7f1c3\": container with ID starting with 9abd8c6744290eba178c1792f37d73f9104d869bc04ed5896cee376e6ea7f1c3 not found: ID does not exist" containerID="9abd8c6744290eba178c1792f37d73f9104d869bc04ed5896cee376e6ea7f1c3"
Sep 18 17:07:18.415284 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:18.415234    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9abd8c6744290eba178c1792f37d73f9104d869bc04ed5896cee376e6ea7f1c3"} err="failed to get container status \"9abd8c6744290eba178c1792f37d73f9104d869bc04ed5896cee376e6ea7f1c3\": rpc error: code = NotFound desc = could not find container \"9abd8c6744290eba178c1792f37d73f9104d869bc04ed5896cee376e6ea7f1c3\": container with ID starting with 9abd8c6744290eba178c1792f37d73f9104d869bc04ed5896cee376e6ea7f1c3 not found: ID does not exist"
Sep 18 17:07:19.108769 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:19.108744    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a50e7b55-b1f2-49ae-9065-3123aacf551a/alertmanager/0.log"
Sep 18 17:07:19.152984 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:19.152931    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a50e7b55-b1f2-49ae-9065-3123aacf551a/config-reloader/0.log"
Sep 18 17:07:19.178384 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:19.178357    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a50e7b55-b1f2-49ae-9065-3123aacf551a/kube-rbac-proxy-web/0.log"
Sep 18 17:07:19.213690 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:19.213662    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a50e7b55-b1f2-49ae-9065-3123aacf551a/kube-rbac-proxy/0.log"
Sep 18 17:07:19.241759 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:19.241679    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a50e7b55-b1f2-49ae-9065-3123aacf551a/kube-rbac-proxy-metric/0.log"
Sep 18 17:07:19.269785 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:19.269720    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a50e7b55-b1f2-49ae-9065-3123aacf551a/prom-label-proxy/0.log"
Sep 18 17:07:19.302807 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:19.302690    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a50e7b55-b1f2-49ae-9065-3123aacf551a/init-config-reloader/0.log"
Sep 18 17:07:19.385465 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:19.385280    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-4zf52_4eb12922-c4f9-4c98-be06-d213019e4e8f/kube-state-metrics/0.log"
Sep 18 17:07:19.412225 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:19.412183    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-4zf52_4eb12922-c4f9-4c98-be06-d213019e4e8f/kube-rbac-proxy-main/0.log"
Sep 18 17:07:19.452114 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:19.452076    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-4zf52_4eb12922-c4f9-4c98-be06-d213019e4e8f/kube-rbac-proxy-self/0.log"
Sep 18 17:07:19.494268 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:19.494231    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-d7fc747f6-kzpsc_af93db19-c810-405f-be6a-15f98057011c/metrics-server/0.log"
Sep 18 17:07:19.554775 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:19.554737    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-g7fbl_e94551de-4e76-488b-98ea-1d011e6e08d8/monitoring-plugin/0.log"
Sep 18 17:07:19.826013 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:19.825983    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xpc96_94263113-b452-406b-86d5-d01e599fcdc3/node-exporter/0.log"
Sep 18 17:07:19.854160 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:19.854120    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xpc96_94263113-b452-406b-86d5-d01e599fcdc3/kube-rbac-proxy/0.log"
Sep 18 17:07:19.876682 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:19.876656    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xpc96_94263113-b452-406b-86d5-d01e599fcdc3/init-textfile/0.log"
Sep 18 17:07:19.907674 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:19.907622    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-kkt48_71e7cb50-46f2-486f-a2bc-12d7443c8c69/kube-rbac-proxy-main/0.log"
Sep 18 17:07:19.947811 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:19.945225    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-kkt48_71e7cb50-46f2-486f-a2bc-12d7443c8c69/kube-rbac-proxy-self/0.log"
Sep 18 17:07:19.981388 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:19.981254    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-kkt48_71e7cb50-46f2-486f-a2bc-12d7443c8c69/openshift-state-metrics/0.log"
Sep 18 17:07:20.057361 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:20.057292    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4f639657-04f0-4cae-9299-a44af3495a3b/prometheus/0.log"
Sep 18 17:07:20.077615 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:20.077585    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4f639657-04f0-4cae-9299-a44af3495a3b/config-reloader/0.log"
Sep 18 17:07:20.104243 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:20.104210    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4f639657-04f0-4cae-9299-a44af3495a3b/thanos-sidecar/0.log"
Sep 18 17:07:20.138473 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:20.137059    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4f639657-04f0-4cae-9299-a44af3495a3b/kube-rbac-proxy-web/0.log"
Sep 18 17:07:20.161328 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:20.161013    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4f639657-04f0-4cae-9299-a44af3495a3b/kube-rbac-proxy/0.log"
Sep 18 17:07:20.187326 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:20.187178    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4f639657-04f0-4cae-9299-a44af3495a3b/kube-rbac-proxy-thanos/0.log"
Sep 18 17:07:20.228401 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:20.228371    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4f639657-04f0-4cae-9299-a44af3495a3b/init-config-reloader/0.log"
Sep 18 17:07:20.273522 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:20.273496    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-f6s64_451d27be-cf0b-4599-90a2-8a3ff8ab3e28/prometheus-operator/0.log"
Sep 18 17:07:20.297405 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:20.297375    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-f6s64_451d27be-cf0b-4599-90a2-8a3ff8ab3e28/kube-rbac-proxy/0.log"
Sep 18 17:07:20.432410 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:20.432376    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f87cdf499-fp65v_20fa737b-a4d1-4ddc-af3b-f47a75433a78/thanos-query/0.log"
Sep 18 17:07:20.459836 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:20.459803    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f87cdf499-fp65v_20fa737b-a4d1-4ddc-af3b-f47a75433a78/kube-rbac-proxy-web/0.log"
Sep 18 17:07:20.483637 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:20.483603    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f87cdf499-fp65v_20fa737b-a4d1-4ddc-af3b-f47a75433a78/kube-rbac-proxy/0.log"
Sep 18 17:07:20.508921 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:20.508862    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f87cdf499-fp65v_20fa737b-a4d1-4ddc-af3b-f47a75433a78/prom-label-proxy/0.log"
Sep 18 17:07:20.531871 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:20.531798    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f87cdf499-fp65v_20fa737b-a4d1-4ddc-af3b-f47a75433a78/kube-rbac-proxy-rules/0.log"
Sep 18 17:07:20.555889 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:20.555835    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f87cdf499-fp65v_20fa737b-a4d1-4ddc-af3b-f47a75433a78/kube-rbac-proxy-metrics/0.log"
Sep 18 17:07:22.928473 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:22.928439    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7c6c96cb85-rc9lz_04664e65-e692-46ad-8c1a-22e7c399965f/console/0.log"
Sep 18 17:07:23.930558 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:23.930521    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zpw2m/perf-node-gather-daemonset-w72hn"]
Sep 18 17:07:23.931067 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:23.931019    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f8351d8-160e-4a16-9969-a021798ec977" containerName="copy"
Sep 18 17:07:23.931067 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:23.931033    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f8351d8-160e-4a16-9969-a021798ec977" containerName="copy"
Sep 18 17:07:23.931067 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:23.931048    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f8351d8-160e-4a16-9969-a021798ec977" containerName="gather"
Sep 18 17:07:23.931067 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:23.931057    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f8351d8-160e-4a16-9969-a021798ec977" containerName="gather"
Sep 18 17:07:23.931319 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:23.931147    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f8351d8-160e-4a16-9969-a021798ec977" containerName="copy"
Sep 18 17:07:23.931319 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:23.931161    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f8351d8-160e-4a16-9969-a021798ec977" containerName="gather"
Sep 18 17:07:23.937447 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:23.937420    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-w72hn"
Sep 18 17:07:23.949421 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:23.949390    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zpw2m/perf-node-gather-daemonset-w72hn"]
Sep 18 17:07:24.053058 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:24.053028    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxm8j\" (UniqueName: \"kubernetes.io/projected/c4c5ae52-a7b2-49a4-9f42-0791221b0bf5-kube-api-access-xxm8j\") pod \"perf-node-gather-daemonset-w72hn\" (UID: \"c4c5ae52-a7b2-49a4-9f42-0791221b0bf5\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-w72hn"
Sep 18 17:07:24.053225 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:24.053072    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c4c5ae52-a7b2-49a4-9f42-0791221b0bf5-podres\") pod \"perf-node-gather-daemonset-w72hn\" (UID: \"c4c5ae52-a7b2-49a4-9f42-0791221b0bf5\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-w72hn"
Sep 18 17:07:24.053225 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:24.053109    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c4c5ae52-a7b2-49a4-9f42-0791221b0bf5-proc\") pod \"perf-node-gather-daemonset-w72hn\" (UID: \"c4c5ae52-a7b2-49a4-9f42-0791221b0bf5\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-w72hn"
Sep 18 17:07:24.053225 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:24.053136    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4c5ae52-a7b2-49a4-9f42-0791221b0bf5-sys\") pod \"perf-node-gather-daemonset-w72hn\" (UID: \"c4c5ae52-a7b2-49a4-9f42-0791221b0bf5\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-w72hn"
Sep 18 17:07:24.053225 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:24.053152    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4c5ae52-a7b2-49a4-9f42-0791221b0bf5-lib-modules\") pod \"perf-node-gather-daemonset-w72hn\" (UID: \"c4c5ae52-a7b2-49a4-9f42-0791221b0bf5\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-w72hn"
Sep 18 17:07:24.153951 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:24.153908    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c4c5ae52-a7b2-49a4-9f42-0791221b0bf5-podres\") pod \"perf-node-gather-daemonset-w72hn\" (UID: \"c4c5ae52-a7b2-49a4-9f42-0791221b0bf5\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-w72hn"
Sep 18 17:07:24.154137 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:24.153968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c4c5ae52-a7b2-49a4-9f42-0791221b0bf5-proc\") pod \"perf-node-gather-daemonset-w72hn\" (UID: \"c4c5ae52-a7b2-49a4-9f42-0791221b0bf5\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-w72hn"
Sep 18 17:07:24.154137 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:24.154026    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4c5ae52-a7b2-49a4-9f42-0791221b0bf5-sys\") pod \"perf-node-gather-daemonset-w72hn\" (UID: \"c4c5ae52-a7b2-49a4-9f42-0791221b0bf5\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-w72hn"
Sep 18 17:07:24.154137 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:24.154051    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4c5ae52-a7b2-49a4-9f42-0791221b0bf5-lib-modules\") pod \"perf-node-gather-daemonset-w72hn\" (UID: \"c4c5ae52-a7b2-49a4-9f42-0791221b0bf5\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-w72hn"
Sep 18 17:07:24.154137 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:24.154082    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c4c5ae52-a7b2-49a4-9f42-0791221b0bf5-podres\") pod \"perf-node-gather-daemonset-w72hn\" (UID: \"c4c5ae52-a7b2-49a4-9f42-0791221b0bf5\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-w72hn"
Sep 18 17:07:24.154137 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:24.154076    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c4c5ae52-a7b2-49a4-9f42-0791221b0bf5-proc\") pod \"perf-node-gather-daemonset-w72hn\" (UID: \"c4c5ae52-a7b2-49a4-9f42-0791221b0bf5\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-w72hn"
Sep 18 17:07:24.154362 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:24.154134    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4c5ae52-a7b2-49a4-9f42-0791221b0bf5-sys\") pod \"perf-node-gather-daemonset-w72hn\" (UID: \"c4c5ae52-a7b2-49a4-9f42-0791221b0bf5\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-w72hn"
Sep 18 17:07:24.154362 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:24.154176    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxm8j\" (UniqueName: \"kubernetes.io/projected/c4c5ae52-a7b2-49a4-9f42-0791221b0bf5-kube-api-access-xxm8j\") pod \"perf-node-gather-daemonset-w72hn\" (UID: \"c4c5ae52-a7b2-49a4-9f42-0791221b0bf5\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-w72hn"
Sep 18 17:07:24.154362 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:24.154215    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4c5ae52-a7b2-49a4-9f42-0791221b0bf5-lib-modules\") pod \"perf-node-gather-daemonset-w72hn\" (UID: \"c4c5ae52-a7b2-49a4-9f42-0791221b0bf5\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-w72hn"
Sep 18 17:07:24.162678 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:24.162648    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxm8j\" (UniqueName: \"kubernetes.io/projected/c4c5ae52-a7b2-49a4-9f42-0791221b0bf5-kube-api-access-xxm8j\") pod \"perf-node-gather-daemonset-w72hn\" (UID: \"c4c5ae52-a7b2-49a4-9f42-0791221b0bf5\") " pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-w72hn"
Sep 18 17:07:24.250318 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:24.250210    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-w72hn"
Sep 18 17:07:24.428320 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:24.428288    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-86jqq_f4ee6c64-ba20-49bc-9d59-c86ce91a51bf/dns/0.log"
Sep 18 17:07:24.450609 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:24.450577    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-86jqq_f4ee6c64-ba20-49bc-9d59-c86ce91a51bf/kube-rbac-proxy/0.log"
Sep 18 17:07:24.538519 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:24.538491    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zpw2m/perf-node-gather-daemonset-w72hn"]
Sep 18 17:07:24.540701 ip-10-0-135-110 kubenswrapper[2580]: W0918 17:07:24.540667    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc4c5ae52_a7b2_49a4_9f42_0791221b0bf5.slice/crio-736b12ce8a874ab2dae2d20fcc78762dc30069073c78851438c368e11bbd1734 WatchSource:0}: Error finding container 736b12ce8a874ab2dae2d20fcc78762dc30069073c78851438c368e11bbd1734: Status 404 returned error can't find the container with id 736b12ce8a874ab2dae2d20fcc78762dc30069073c78851438c368e11bbd1734
Sep 18 17:07:24.619528 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:24.619497    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-krchj_e945222a-603b-4489-a528-2f607d157b14/dns-node-resolver/0.log"
Sep 18 17:07:25.150932 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:25.150894    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4vmq6_cfa7efc4-7409-41f5-b891-7ce9e5b6982f/node-ca/0.log"
Sep 18 17:07:25.418038 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:25.417943    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-w72hn" event={"ID":"c4c5ae52-a7b2-49a4-9f42-0791221b0bf5","Type":"ContainerStarted","Data":"0649ed65ffb99f3f4fdb9f6f2b25e76b9c892b06ec913d857fed7472e20d3016"}
Sep 18 17:07:25.418038 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:25.418010    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-w72hn" event={"ID":"c4c5ae52-a7b2-49a4-9f42-0791221b0bf5","Type":"ContainerStarted","Data":"736b12ce8a874ab2dae2d20fcc78762dc30069073c78851438c368e11bbd1734"}
Sep 18 17:07:25.418276 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:25.418240    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-w72hn"
Sep 18 17:07:25.437218 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:25.437158    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-w72hn" podStartSLOduration=2.437141498 podStartE2EDuration="2.437141498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:07:25.435296314 +0000 UTC m=+3091.523611955" watchObservedRunningTime="2026-09-18 17:07:25.437141498 +0000 UTC m=+3091.525457137"
Sep 18 17:07:26.386557 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:26.386531    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-thwr5_1cea6a79-7d5a-4093-a1b8-0d07198052e3/serve-healthcheck-canary/0.log"
Sep 18 17:07:26.851756 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:26.851727    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f5szk_be090468-5a51-41dc-97fe-8a8569135cef/kube-rbac-proxy/0.log"
Sep 18 17:07:26.871944 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:26.871920    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f5szk_be090468-5a51-41dc-97fe-8a8569135cef/exporter/0.log"
Sep 18 17:07:26.894778 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:26.894744    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f5szk_be090468-5a51-41dc-97fe-8a8569135cef/extractor/0.log"
Sep 18 17:07:29.349773 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:29.349743    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-n4vd5_1fae0365-7eda-4270-a2a0-0dbbbbf9dc49/manager/0.log"
Sep 18 17:07:29.381793 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:29.381768    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-ft4kd_a25c8493-10c1-4f75-bee2-fbfde5d9179b/s3-init/0.log"
Sep 18 17:07:29.396685 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:29.396660    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-ft4kd_a25c8493-10c1-4f75-bee2-fbfde5d9179b/download-hf-model/0.log"
Sep 18 17:07:31.434461 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:31.434430    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zpw2m/perf-node-gather-daemonset-w72hn"
Sep 18 17:07:34.840572 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:34.840535    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8tkmt_e80a48a0-bdd3-4128-999a-cc2574ced499/kube-multus/0.log"
Sep 18 17:07:35.176575 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:35.176537    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ls6gx_68f204b6-4215-48a9-973e-d1e84ae4fed5/kube-multus-additional-cni-plugins/0.log"
Sep 18 17:07:35.198730 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:35.198705    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ls6gx_68f204b6-4215-48a9-973e-d1e84ae4fed5/egress-router-binary-copy/0.log"
Sep 18 17:07:35.218610 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:35.218576    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ls6gx_68f204b6-4215-48a9-973e-d1e84ae4fed5/cni-plugins/0.log"
Sep 18 17:07:35.240155 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:35.240131    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ls6gx_68f204b6-4215-48a9-973e-d1e84ae4fed5/bond-cni-plugin/0.log"
Sep 18 17:07:35.264100 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:35.264079    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ls6gx_68f204b6-4215-48a9-973e-d1e84ae4fed5/routeoverride-cni/0.log"
Sep 18 17:07:35.291525 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:35.291486    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ls6gx_68f204b6-4215-48a9-973e-d1e84ae4fed5/whereabouts-cni-bincopy/0.log"
Sep 18 17:07:35.312892 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:35.312863    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ls6gx_68f204b6-4215-48a9-973e-d1e84ae4fed5/whereabouts-cni/0.log"
Sep 18 17:07:35.474920 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:35.474893    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tnb7c_c861f27f-910e-48aa-b4bc-3c20bcd911ef/network-metrics-daemon/0.log"
Sep 18 17:07:35.496005 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:35.495980    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tnb7c_c861f27f-910e-48aa-b4bc-3c20bcd911ef/kube-rbac-proxy/0.log"
Sep 18 17:07:36.223334 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:36.223220    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gw4fg_96f857f8-22b6-4d0b-b6fd-826b9b507ef8/ovn-controller/0.log"
Sep 18 17:07:36.240598 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:36.240572    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gw4fg_96f857f8-22b6-4d0b-b6fd-826b9b507ef8/ovn-acl-logging/0.log"
Sep 18 17:07:36.268121 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:36.268093    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gw4fg_96f857f8-22b6-4d0b-b6fd-826b9b507ef8/kube-rbac-proxy-node/0.log"
Sep 18 17:07:36.291114 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:36.291089    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gw4fg_96f857f8-22b6-4d0b-b6fd-826b9b507ef8/kube-rbac-proxy-ovn-metrics/0.log"
Sep 18 17:07:36.314251 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:36.314214    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gw4fg_96f857f8-22b6-4d0b-b6fd-826b9b507ef8/northd/0.log"
Sep 18 17:07:36.334663 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:36.334637    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gw4fg_96f857f8-22b6-4d0b-b6fd-826b9b507ef8/nbdb/0.log"
Sep 18 17:07:36.359043 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:36.359012    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gw4fg_96f857f8-22b6-4d0b-b6fd-826b9b507ef8/sbdb/0.log"
Sep 18 17:07:36.405186 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:36.405157    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gw4fg_96f857f8-22b6-4d0b-b6fd-826b9b507ef8/ovnkube-controller/0.log"
Sep 18 17:07:38.097022 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:38.096990    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-fs4cj_8b313705-71ee-4c64-a650-b308f048dc2a/network-check-target-container/0.log"
Sep 18 17:07:39.041859 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:39.041830    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-h6tt8_11366fd3-c5b8-4bf4-8ba6-217fd3a1b658/iptables-alerter/0.log"
Sep 18 17:07:39.750650 ip-10-0-135-110 kubenswrapper[2580]: I0918 17:07:39.750623    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-96w8b_752ec166-f680-4caf-b569-5ca2d59307dc/tuned/0.log"