Jul 07 16:46:45.577746 ip-10-0-141-189 systemd[1]: Starting Kubernetes Kubelet...
Jul 07 16:46:46.007621 ip-10-0-141-189 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 16:46:46.007621 ip-10-0-141-189 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 07 16:46:46.007621 ip-10-0-141-189 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 16:46:46.007621 ip-10-0-141-189 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 07 16:46:46.007621 ip-10-0-141-189 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 16:46:46.009405 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.009315    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 07 16:46:46.013670 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013653    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:46:46.013670 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013670    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:46:46.013738 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013674    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:46:46.013738 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013677    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:46:46.013738 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013681    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:46:46.013738 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013683    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:46:46.013738 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013687    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:46:46.013738 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013689    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:46:46.013738 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013692    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:46:46.013738 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013695    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:46:46.013738 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013698    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:46:46.013738 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013700    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:46:46.013738 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013703    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:46:46.013738 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013707    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:46:46.013738 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013711    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:46:46.013738 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013715    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:46:46.013738 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013719    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:46:46.013738 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013722    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:46:46.013738 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013724    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:46:46.013738 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013727    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:46:46.013738 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013729    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:46:46.014227 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013732    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:46:46.014227 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013734    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:46:46.014227 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013737    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:46:46.014227 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013739    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:46:46.014227 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013742    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:46:46.014227 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013745    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:46:46.014227 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013748    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:46:46.014227 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013750    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:46:46.014227 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013753    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:46:46.014227 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013755    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:46:46.014227 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013758    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:46:46.014227 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013761    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:46:46.014227 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013763    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:46:46.014227 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013766    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:46:46.014227 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013768    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:46:46.014227 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013771    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:46:46.014227 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013773    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:46:46.014227 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013776    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:46:46.014227 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013778    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:46:46.014227 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013780    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:46:46.014748 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013783    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:46:46.014748 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013785    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:46:46.014748 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013788    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:46:46.014748 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013790    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:46:46.014748 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013793    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:46:46.014748 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013796    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:46:46.014748 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013798    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:46:46.014748 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013800    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:46:46.014748 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013803    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:46:46.014748 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013805    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:46:46.014748 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013807    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:46:46.014748 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013810    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:46:46.014748 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013813    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:46:46.014748 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013816    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:46:46.014748 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013818    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:46:46.014748 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013821    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:46:46.014748 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013824    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:46:46.014748 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013826    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:46:46.014748 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013828    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:46:46.014748 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013831    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:46:46.015229 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013833    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:46:46.015229 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013835    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:46:46.015229 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013838    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:46:46.015229 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013840    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:46:46.015229 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013843    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:46:46.015229 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013845    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:46:46.015229 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013847    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:46:46.015229 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013852    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:46:46.015229 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013856    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:46:46.015229 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013859    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:46:46.015229 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013867    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:46:46.015229 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013870    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:46:46.015229 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013872    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:46:46.015229 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013875    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:46:46.015229 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013878    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:46:46.015229 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013880    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:46:46.015229 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013882    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:46:46.015229 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013885    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:46:46.015229 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013887    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:46:46.015229 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013890    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:46:46.015720 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013893    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:46:46.015720 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013896    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:46:46.015720 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013898    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:46:46.015720 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013901    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:46:46.015720 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013904    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:46:46.015720 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013907    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:46:46.015720 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013910    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:46:46.015720 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013912    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:46:46.015720 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013916    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:46:46.015720 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013919    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:46:46.015720 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013921    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:46:46.015720 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013923    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:46:46.015720 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013926    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:46:46.015720 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013928    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:46:46.015720 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013931    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:46:46.015720 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013933    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:46:46.015720 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013936    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:46:46.015720 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.013938    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:46:46.015720 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014745    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:46:46.015720 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014753    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:46:46.016239 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014755    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:46:46.016239 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014759    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:46:46.016239 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014762    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:46:46.016239 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014767    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:46:46.016239 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014772    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:46:46.016239 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014775    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:46:46.016239 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014779    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:46:46.016239 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014782    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:46:46.016239 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014786    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:46:46.016239 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014789    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:46:46.016239 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014791    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:46:46.016239 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014794    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:46:46.016239 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014796    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:46:46.016239 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014799    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:46:46.016239 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014801    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:46:46.016239 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014804    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:46:46.016239 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014806    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:46:46.016239 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014809    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:46:46.016239 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014812    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:46:46.016753 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014815    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:46:46.016753 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014817    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:46:46.016753 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014820    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:46:46.016753 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014822    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:46:46.016753 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014825    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:46:46.016753 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014828    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:46:46.016753 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014830    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:46:46.016753 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014832    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:46:46.016753 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014835    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:46:46.016753 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014837    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:46:46.016753 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014840    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:46:46.016753 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014842    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:46:46.016753 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014844    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:46:46.016753 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014848    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:46:46.016753 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014851    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:46:46.016753 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014854    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:46:46.016753 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014857    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:46:46.016753 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014860    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:46:46.016753 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014863    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:46:46.017235 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014865    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:46:46.017235 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014868    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:46:46.017235 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014871    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:46:46.017235 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014873    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:46:46.017235 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014875    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:46:46.017235 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014878    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:46:46.017235 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014880    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:46:46.017235 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014882    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:46:46.017235 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014885    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:46:46.017235 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014887    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:46:46.017235 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014890    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:46:46.017235 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014892    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:46:46.017235 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014895    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:46:46.017235 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014898    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:46:46.017235 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014900    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:46:46.017235 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014902    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:46:46.017235 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014905    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:46:46.017235 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014907    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:46:46.017235 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014910    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:46:46.017235 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014912    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:46:46.017235 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014914    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:46:46.017768 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014918    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:46:46.017768 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014920    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:46:46.017768 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014922    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:46:46.017768 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014925    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:46:46.017768 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014927    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:46:46.017768 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014929    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:46:46.017768 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014932    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:46:46.017768 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014934    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:46:46.017768 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014936    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:46:46.017768 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014939    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:46:46.017768 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014942    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:46:46.017768 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014944    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:46:46.017768 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014947    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:46:46.017768 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014950    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:46:46.017768 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014952    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:46:46.017768 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014954    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:46:46.017768 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014957    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:46:46.017768 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014960    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:46:46.017768 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014962    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:46:46.017768 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014964    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:46:46.018257 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014967    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:46:46.018257 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014970    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:46:46.018257 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014973    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:46:46.018257 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014975    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:46:46.018257 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014977    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:46:46.018257 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014980    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:46:46.018257 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014982    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:46:46.018257 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014984    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:46:46.018257 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014987    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:46:46.018257 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014989    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:46:46.018257 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014991    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:46:46.018257 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014994    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:46:46.018257 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014996    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:46:46.018257 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.014998    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:46:46.018257 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015001    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:46:46.018257 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015003    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:46:46.018257 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015005    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:46:46.018257 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015008    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:46:46.018257 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015505    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:46:46.018740 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015512    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:46:46.018740 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015516    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:46:46.018740 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015519    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:46:46.018740 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015522    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:46:46.018740 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015525    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:46:46.018740 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015528    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:46:46.018740 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015533    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:46:46.018740 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015536    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:46:46.018740 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015539    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:46:46.018740 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015541    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:46:46.018740 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015544    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:46:46.018740 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015546    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:46:46.018740 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015549    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:46:46.018740 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015551    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:46:46.018740 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015555    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:46:46.018740 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015558    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:46:46.018740 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015560    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:46:46.018740 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015563    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:46:46.018740 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015565    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:46:46.018740 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015567    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:46:46.019226 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015570    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:46:46.019226 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015573    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:46:46.019226 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015575    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:46:46.019226 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015578    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:46:46.019226 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015580    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:46:46.019226 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015582    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:46:46.019226 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015585    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:46:46.019226 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015588    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:46:46.019226 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015590    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:46:46.019226 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015593    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:46:46.019226 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015596    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:46:46.019226 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015598    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:46:46.019226 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015601    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:46:46.019226 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015603    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:46:46.019226 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015606    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:46:46.019226 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015608    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:46:46.019226 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015611    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:46:46.019226 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015613    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:46:46.019226 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015618    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:46:46.019226 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015620    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:46:46.019762 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015622    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:46:46.019762 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015625    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:46:46.019762 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015628    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:46:46.019762 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015630    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:46:46.019762 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015633    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:46:46.019762 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015635    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:46:46.019762 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015639    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:46:46.019762 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015642    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:46:46.019762 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015645    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:46:46.019762 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015647    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:46:46.019762 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015650    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:46:46.019762 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015653    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:46:46.019762 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015655    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:46:46.019762 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015658    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:46:46.019762 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015660    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:46:46.019762 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015663    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:46:46.019762 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015665    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:46:46.019762 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015667    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:46:46.019762 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015670    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:46:46.020306 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015674    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:46:46.020306 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015677    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:46:46.020306 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015680    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:46:46.020306 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015683    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:46:46.020306 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015685    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:46:46.020306 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015688    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:46:46.020306 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015690    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:46:46.020306 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015693    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:46:46.020306 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015695    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:46:46.020306 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015698    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:46:46.020306 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015701    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:46:46.020306 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015705    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:46:46.020306 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015707    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:46:46.020306 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015709    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:46:46.020306 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015712    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:46:46.020306 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015714    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:46:46.020306 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015717    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:46:46.020306 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015719    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:46:46.020306 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015722    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:46:46.020837 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015725    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:46:46.020837 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015728    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:46:46.020837 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015731    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:46:46.020837 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015733    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:46:46.020837 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015736    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:46:46.020837 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015738    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:46:46.020837 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015741    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:46:46.020837 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015743    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:46:46.020837 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015748    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:46:46.020837 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015751    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:46:46.020837 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015754    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:46:46.020837 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015757    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:46:46.020837 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015759    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:46:46.020837 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015762    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:46:46.020837 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015764    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:46:46.020837 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015766    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:46:46.020837 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015769    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:46:46.020837 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015772    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:46:46.020837 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015774    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:46:46.020837 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.015777    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:46:46.021400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015861    2583 flags.go:64] FLAG: --address="0.0.0.0"
Jul 07 16:46:46.021400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015868    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 07 16:46:46.021400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015874    2583 flags.go:64] FLAG: --anonymous-auth="true"
Jul 07 16:46:46.021400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015879    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 07 16:46:46.021400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015886    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 07 16:46:46.021400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015890    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 07 16:46:46.021400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015894    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 07 16:46:46.021400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015898    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 07 16:46:46.021400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015901    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 07 16:46:46.021400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015904    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 07 16:46:46.021400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015907    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 07 16:46:46.021400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015910    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 07 16:46:46.021400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015914    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 07 16:46:46.021400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015917    2583 flags.go:64] FLAG: --cgroup-root=""
Jul 07 16:46:46.021400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015920    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 07 16:46:46.021400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015923    2583 flags.go:64] FLAG: --client-ca-file=""
Jul 07 16:46:46.021400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015926    2583 flags.go:64] FLAG: --cloud-provider="external"
Jul 07 16:46:46.021400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015929    2583 flags.go:64] FLAG: --cluster-dns="[]"
Jul 07 16:46:46.021400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015933    2583 flags.go:64] FLAG: --cluster-domain=""
Jul 07 16:46:46.021400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015936    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 07 16:46:46.021400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015939    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 07 16:46:46.021400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015942    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 07 16:46:46.021400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015946    2583 flags.go:64] FLAG: --container-log-max-files="5"
Jul 07 16:46:46.021400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015950    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 07 16:46:46.022193 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015952    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 07 16:46:46.022193 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015955    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 07 16:46:46.022193 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015958    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 07 16:46:46.022193 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015961    2583 flags.go:64] FLAG: --contention-profiling="false"
Jul 07 16:46:46.022193 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015964    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 07 16:46:46.022193 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015967    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 07 16:46:46.022193 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015970    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 07 16:46:46.022193 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015973    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 07 16:46:46.022193 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015976    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 07 16:46:46.022193 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015979    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 07 16:46:46.022193 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015982    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 07 16:46:46.022193 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015985    2583 flags.go:64] FLAG: --enable-load-reader="false"
Jul 07 16:46:46.022193 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015989    2583 flags.go:64] FLAG: --enable-server="true"
Jul 07 16:46:46.022193 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015993    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 07 16:46:46.022193 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.015997    2583 flags.go:64] FLAG: --event-burst="100"
Jul 07 16:46:46.022193 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016000    2583 flags.go:64] FLAG: --event-qps="50"
Jul 07 16:46:46.022193 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016003    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 07 16:46:46.022193 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016005    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 07 16:46:46.022193 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016008    2583 flags.go:64] FLAG: --eviction-hard=""
Jul 07 16:46:46.022193 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016012    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 07 16:46:46.022193 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016016    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 07 16:46:46.022193 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016018    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 07 16:46:46.022193 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016021    2583 flags.go:64] FLAG: --eviction-soft=""
Jul 07 16:46:46.022193 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016024    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 07 16:46:46.022193 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016027    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 07 16:46:46.022946 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016030    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 07 16:46:46.022946 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016033    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 07 16:46:46.022946 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016036    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 07 16:46:46.022946 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016039    2583 flags.go:64] FLAG: --fail-swap-on="true"
Jul 07 16:46:46.022946 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016042    2583 flags.go:64] FLAG: --feature-gates=""
Jul 07 16:46:46.022946 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016046    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 07 16:46:46.022946 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016049    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 07 16:46:46.022946 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016052    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 07 16:46:46.022946 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016054    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 07 16:46:46.022946 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016057    2583 flags.go:64] FLAG: --healthz-port="10248"
Jul 07 16:46:46.022946 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016060    2583 flags.go:64] FLAG: --help="false"
Jul 07 16:46:46.022946 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016063    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-141-189.ec2.internal"
Jul 07 16:46:46.022946 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016066    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 07 16:46:46.022946 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016069    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 07 16:46:46.022946 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016071    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 07 16:46:46.022946 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016075    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 07 16:46:46.022946 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016078    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 07 16:46:46.022946 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016080    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 07 16:46:46.022946 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016083    2583 flags.go:64] FLAG: --image-service-endpoint=""
Jul 07 16:46:46.022946 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016087    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 07 16:46:46.022946 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016090    2583 flags.go:64] FLAG: --kube-api-burst="100"
Jul 07 16:46:46.022946 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016093    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 07 16:46:46.022946 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016096    2583 flags.go:64] FLAG: --kube-api-qps="50"
Jul 07 16:46:46.022946 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016099    2583 flags.go:64] FLAG: --kube-reserved=""
Jul 07 16:46:46.023609 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016102    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 07 16:46:46.023609 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016105    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 07 16:46:46.023609 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016107    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 07 16:46:46.023609 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016112    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 07 16:46:46.023609 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016114    2583 flags.go:64] FLAG: --lock-file=""
Jul 07 16:46:46.023609 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016117    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 07 16:46:46.023609 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016120    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 07 16:46:46.023609 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016123    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 07 16:46:46.023609 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016128    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 07 16:46:46.023609 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016131    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 07 16:46:46.023609 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016134    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 07 16:46:46.023609 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016136    2583 flags.go:64] FLAG: --logging-format="text"
Jul 07 16:46:46.023609 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016139    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 07 16:46:46.023609 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016142    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 07 16:46:46.023609 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016145    2583 flags.go:64] FLAG: --manifest-url=""
Jul 07 16:46:46.023609 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016148    2583 flags.go:64] FLAG: --manifest-url-header=""
Jul 07 16:46:46.023609 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016152    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 07 16:46:46.023609 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016155    2583 flags.go:64] FLAG: --max-open-files="1000000"
Jul 07 16:46:46.023609 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016159    2583 flags.go:64] FLAG: --max-pods="110"
Jul 07 16:46:46.023609 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016162    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 07 16:46:46.023609 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016164    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 07 16:46:46.023609 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016167    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 07 16:46:46.023609 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016170    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 07 16:46:46.023609 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016173    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 07 16:46:46.023609 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016176    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 07 16:46:46.024238 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016179    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 07 16:46:46.024238 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016186    2583 flags.go:64] FLAG: --node-status-max-images="50"
Jul 07 16:46:46.024238 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016191    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 07 16:46:46.024238 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016194    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 07 16:46:46.024238 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016196    2583 flags.go:64] FLAG: --pod-cidr=""
Jul 07 16:46:46.024238 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016200    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 07 16:46:46.024238 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016204    2583 flags.go:64] FLAG: --pod-manifest-path=""
Jul 07 16:46:46.024238 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016207    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 07 16:46:46.024238 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016210    2583 flags.go:64] FLAG: --pods-per-core="0"
Jul 07 16:46:46.024238 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016213    2583 flags.go:64] FLAG: --port="10250"
Jul 07 16:46:46.024238 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016215    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 07 16:46:46.024238 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016220    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e11cb36466c6e5ef"
Jul 07 16:46:46.024238 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016223    2583 flags.go:64] FLAG: --qos-reserved=""
Jul 07 16:46:46.024238 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016225    2583 flags.go:64] FLAG: --read-only-port="10255"
Jul 07 16:46:46.024238 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016228    2583 flags.go:64] FLAG: --register-node="true"
Jul 07 16:46:46.024238 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016231    2583 flags.go:64] FLAG: --register-with-taints=""
Jul 07 16:46:46.024238 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016235    2583 flags.go:64] FLAG: --registry-burst="10"
Jul 07 16:46:46.024238 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016238    2583 flags.go:64] FLAG: --registry-qps="5"
Jul 07 16:46:46.024238 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016241    2583 flags.go:64] FLAG: --reserved-cpus=""
Jul 07 16:46:46.024238 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016244    2583 flags.go:64] FLAG: --reserved-memory=""
Jul 07 16:46:46.024238 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016247    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 07 16:46:46.024238 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016250    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 07 16:46:46.024238 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016254    2583 flags.go:64] FLAG: --rotate-certificates="false"
Jul 07 16:46:46.024238 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016256    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 07 16:46:46.024238 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016259    2583 flags.go:64] FLAG: --runonce="false"
Jul 07 16:46:46.024869 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016262    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 07 16:46:46.024869 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016264    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 07 16:46:46.024869 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016267    2583 flags.go:64] FLAG: --seccomp-default="false"
Jul 07 16:46:46.024869 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016270    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 07 16:46:46.024869 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016273    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 07 16:46:46.024869 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016276    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 07 16:46:46.024869 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016278    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 07 16:46:46.024869 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016281    2583 flags.go:64] FLAG: --storage-driver-password="root"
Jul 07 16:46:46.024869 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016284    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 07 16:46:46.024869 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016288    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 07 16:46:46.024869 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016291    2583 flags.go:64] FLAG: --storage-driver-user="root"
Jul 07 16:46:46.024869 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016299    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 07 16:46:46.024869 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016302    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 07 16:46:46.024869 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016306    2583 flags.go:64] FLAG: --system-cgroups=""
Jul 07 16:46:46.024869 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016308    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 07 16:46:46.024869 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016314    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 07 16:46:46.024869 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016317    2583 flags.go:64] FLAG: --tls-cert-file=""
Jul 07 16:46:46.024869 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016320    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 07 16:46:46.024869 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016325    2583 flags.go:64] FLAG: --tls-min-version=""
Jul 07 16:46:46.024869 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016327    2583 flags.go:64] FLAG: --tls-private-key-file=""
Jul 07 16:46:46.024869 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016330    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 07 16:46:46.024869 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016333    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 07 16:46:46.024869 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016335    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 07 16:46:46.024869 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016338    2583 flags.go:64] FLAG: --v="2"
Jul 07 16:46:46.024869 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016342    2583 flags.go:64] FLAG: --version="false"
Jul 07 16:46:46.025587 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016346    2583 flags.go:64] FLAG: --vmodule=""
Jul 07 16:46:46.025587 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016349    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 07 16:46:46.025587 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.016352    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 07 16:46:46.025587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016496    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:46:46.025587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016501    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:46:46.025587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016504    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:46:46.025587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016506    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:46:46.025587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016509    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:46:46.025587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016512    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:46:46.025587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016515    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:46:46.025587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016517    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:46:46.025587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016520    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:46:46.025587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016522    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:46:46.025587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016524    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:46:46.025587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016527    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:46:46.025587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016530    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:46:46.025587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016532    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:46:46.025587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016535    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:46:46.025587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016538    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:46:46.025587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016540    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:46:46.026080 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016542    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:46:46.026080 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016545    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:46:46.026080 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016548    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:46:46.026080 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016550    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:46:46.026080 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016552    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:46:46.026080 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016555    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:46:46.026080 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016558    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:46:46.026080 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016560    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:46:46.026080 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016563    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:46:46.026080 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016565    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:46:46.026080 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016567    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:46:46.026080 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016570    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:46:46.026080 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016572    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:46:46.026080 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016575    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:46:46.026080 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016578    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:46:46.026080 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016580    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:46:46.026080 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016583    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:46:46.026080 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016585    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:46:46.026080 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016588    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:46:46.026080 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016590    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:46:46.026614 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016593    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:46:46.026614 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016597    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:46:46.026614 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016600    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:46:46.026614 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016603    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:46:46.026614 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016606    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:46:46.026614 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016608    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:46:46.026614 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016611    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:46:46.026614 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016614    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:46:46.026614 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016617    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:46:46.026614 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016620    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:46:46.026614 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016622    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:46:46.026614 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016625    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:46:46.026614 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016628    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:46:46.026614 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016630    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:46:46.026614 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016633    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:46:46.026614 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016636    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:46:46.026614 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016639    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:46:46.026614 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016642    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:46:46.026614 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016644    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:46:46.027083 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016648    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:46:46.027083 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016651    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:46:46.027083 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016654    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:46:46.027083 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016657    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:46:46.027083 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016660    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:46:46.027083 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016662    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:46:46.027083 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016665    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:46:46.027083 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016667    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:46:46.027083 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016669    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:46:46.027083 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016672    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:46:46.027083 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016674    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:46:46.027083 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016676    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:46:46.027083 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016679    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:46:46.027083 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016681    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:46:46.027083 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016683    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:46:46.027083 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016686    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:46:46.027083 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016688    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:46:46.027083 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016690    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:46:46.027083 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016693    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:46:46.027083 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016695    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:46:46.027587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016697    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:46:46.027587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016700    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:46:46.027587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016702    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:46:46.027587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016705    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:46:46.027587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016707    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:46:46.027587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016710    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:46:46.027587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016712    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:46:46.027587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016715    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:46:46.027587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016718    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:46:46.027587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016720    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:46:46.027587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016723    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:46:46.027587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016725    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:46:46.027587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016728    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:46:46.027587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016730    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:46:46.027587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016733    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:46:46.027587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016735    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:46:46.027587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016739    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:46:46.027587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016741    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:46:46.027587 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016744    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:46:46.028043 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016746    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:46:46.028043 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016748    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:46:46.028043 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016751    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:46:46.028043 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.016753    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:46:46.028043 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.017419    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 16:46:46.028043 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.025213    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 07 16:46:46.028043 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.025230    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 07 16:46:46.028043 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.025251    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 07 16:46:46.028043 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.025260    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 07 16:46:46.028043 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025340    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:46:46.028043 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025345    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:46:46.028043 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025349    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:46:46.028043 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025352    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:46:46.028043 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025355    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:46:46.028394 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025358    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:46:46.028394 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025360    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:46:46.028394 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025363    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:46:46.028394 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025366    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:46:46.028394 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025368    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:46:46.028394 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025371    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:46:46.028394 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025373    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:46:46.028394 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025376    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:46:46.028394 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025379    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:46:46.028394 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025381    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:46:46.028394 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025384    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:46:46.028394 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025386    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:46:46.028394 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025389    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:46:46.028394 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025391    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:46:46.028394 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025394    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:46:46.028394 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025396    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:46:46.028394 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025399    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:46:46.028394 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025401    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:46:46.028394 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025404    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:46:46.028394 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025406    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:46:46.028912 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025409    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:46:46.028912 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025412    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:46:46.028912 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025414    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:46:46.028912 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025417    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:46:46.028912 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025444    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:46:46.028912 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025447    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:46:46.028912 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025450    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:46:46.028912 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025452    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:46:46.028912 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025455    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:46:46.028912 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025457    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:46:46.028912 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025460    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:46:46.028912 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025462    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:46:46.028912 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025464    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:46:46.028912 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025467    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:46:46.028912 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025469    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:46:46.028912 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025471    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:46:46.028912 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025474    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:46:46.028912 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025476    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:46:46.028912 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025479    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:46:46.028912 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025481    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:46:46.029406 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025486    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:46:46.029406 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025489    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:46:46.029406 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025492    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:46:46.029406 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025495    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:46:46.029406 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025498    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:46:46.029406 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025501    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:46:46.029406 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025504    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:46:46.029406 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025507    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:46:46.029406 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025510    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:46:46.029406 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025513    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:46:46.029406 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025515    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:46:46.029406 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025518    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:46:46.029406 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025520    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:46:46.029406 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025522    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:46:46.029406 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025525    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:46:46.029406 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025527    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:46:46.029406 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025530    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:46:46.029406 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025538    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:46:46.029406 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025540    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:46:46.030000 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025543    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:46:46.030000 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025545    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:46:46.030000 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025548    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:46:46.030000 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025550    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:46:46.030000 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025552    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:46:46.030000 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025555    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:46:46.030000 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025557    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:46:46.030000 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025559    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:46:46.030000 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025562    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:46:46.030000 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025564    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:46:46.030000 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025566    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:46:46.030000 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025569    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:46:46.030000 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025571    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:46:46.030000 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025574    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:46:46.030000 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025576    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:46:46.030000 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025578    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:46:46.030000 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025581    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:46:46.030000 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025583    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:46:46.030000 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025585    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:46:46.030655 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025589    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:46:46.030655 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025593    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:46:46.030655 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025596    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:46:46.030655 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025599    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:46:46.030655 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025602    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:46:46.030655 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025605    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:46:46.030655 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025607    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:46:46.030655 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025611    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:46:46.030655 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025613    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:46:46.030655 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025616    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:46:46.030655 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025618    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:46:46.030655 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025621    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:46:46.030655 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025629    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:46:46.030655 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025631    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:46:46.030655 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025634    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:46:46.030655 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025636    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:46:46.031218 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.025641    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 16:46:46.031218 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025796    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:46:46.031218 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025801    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:46:46.031218 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025804    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:46:46.031218 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025807    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:46:46.031218 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025810    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:46:46.031218 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025814    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:46:46.031218 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025823    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:46:46.031218 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025826    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:46:46.031218 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025828    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:46:46.031218 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025831    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:46:46.031218 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025833    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:46:46.031218 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025836    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:46:46.031218 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025839    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:46:46.031218 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025841    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:46:46.031676 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025843    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:46:46.031676 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025846    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:46:46.031676 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025848    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:46:46.031676 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025851    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:46:46.031676 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025853    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:46:46.031676 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025855    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:46:46.031676 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025858    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:46:46.031676 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025860    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:46:46.031676 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025862    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:46:46.031676 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025865    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:46:46.031676 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025868    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:46:46.031676 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025870    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:46:46.031676 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025873    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:46:46.031676 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025875    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:46:46.031676 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025883    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:46:46.031676 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025885    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:46:46.031676 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025889    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:46:46.031676 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025892    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:46:46.031676 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025895    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:46:46.031676 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025897    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:46:46.032295 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025900    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:46:46.032295 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025902    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:46:46.032295 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025905    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:46:46.032295 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025907    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:46:46.032295 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025909    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:46:46.032295 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025912    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:46:46.032295 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025914    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:46:46.032295 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025916    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:46:46.032295 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025919    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:46:46.032295 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025921    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:46:46.032295 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025923    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:46:46.032295 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025926    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:46:46.032295 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025928    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:46:46.032295 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025930    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:46:46.032295 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025933    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:46:46.032295 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025935    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:46:46.032295 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025937    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:46:46.032295 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025940    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:46:46.032295 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025942    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:46:46.032295 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025944    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:46:46.032799 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025947    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:46:46.032799 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025949    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:46:46.032799 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025952    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:46:46.032799 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025954    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:46:46.032799 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025957    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:46:46.032799 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025959    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:46:46.032799 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025962    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:46:46.032799 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025971    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:46:46.032799 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025974    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:46:46.032799 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025976    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:46:46.032799 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025979    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:46:46.032799 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025981    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:46:46.032799 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025983    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:46:46.032799 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025986    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:46:46.032799 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025988    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:46:46.032799 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025991    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:46:46.032799 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025993    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:46:46.032799 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025995    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:46:46.032799 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.025998    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:46:46.033266 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.026000    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:46:46.033266 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.026003    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:46:46.033266 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.026005    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:46:46.033266 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.026007    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:46:46.033266 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.026009    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:46:46.033266 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.026012    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:46:46.033266 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.026014    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:46:46.033266 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.026016    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:46:46.033266 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.026019    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:46:46.033266 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.026021    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:46:46.033266 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.026023    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:46:46.033266 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.026026    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:46:46.033266 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.026028    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:46:46.033266 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.026033    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:46:46.033266 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.026036    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:46:46.033266 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.026039    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:46:46.033266 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.026042    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:46:46.033266 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.026045    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:46:46.033266 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.026047    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:46:46.033734 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.026050    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:46:46.033734 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.026052    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:46:46.033734 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.026060    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:46:46.033734 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.026062    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:46:46.033734 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.026065    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:46:46.033734 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.026067    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:46:46.033734 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:46.026070    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:46:46.033734 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.026074    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 16:46:46.033734 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.026194    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 07 16:46:46.033734 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.028212    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 07 16:46:46.033734 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.031485    2583 server.go:1019] "Starting client certificate rotation"
Jul 07 16:46:46.033734 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.031536    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 16:46:46.033734 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.031564    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 16:46:46.034491 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.034469    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 16:46:46.035696 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.035680    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 16:46:46.047991 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.047973    2583 log.go:25] "Validated CRI v1 runtime API"
Jul 07 16:46:46.050118 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.050103    2583 log.go:25] "Validated CRI v1 image API"
Jul 07 16:46:46.051772 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.051751    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 07 16:46:46.056167 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.056148    2583 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a1b173b6-444a-4376-83fd-28b171cf564e:/dev/nvme0n1p3 c22a8d5c-b3f1-4031-8d98-aebbdfeb0812:/dev/nvme0n1p4]
Jul 07 16:46:46.056234 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.056167    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 07 16:46:46.062780 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.062647    2583 manager.go:216] Machine: {Timestamp:2026-07-07 16:46:46.06134455 +0000 UTC m=+0.361060217 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097869 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25c2aa00f3634253104c31b6b3aee2 SystemUUID:ec25c2aa-00f3-6342-5310-4c31b6b3aee2 BootID:3510ffb2-eb8e-475c-b830-323e8aa309cd Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:89:10:d1:c7:2f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:89:10:d1:c7:2f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d2:62:75:8c:32:93 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 07 16:46:46.062780 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.062775    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 07 16:46:46.062884 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.062865    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 07 16:46:46.066111 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.066081    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 07 16:46:46.066261 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.066116    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-189.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 07 16:46:46.066918 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.066905    2583 topology_manager.go:138] "Creating topology manager with none policy"
Jul 07 16:46:46.066918 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.066918    2583 container_manager_linux.go:309] "Creating device plugin manager"
Jul 07 16:46:46.067006 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.066926    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 16:46:46.067559 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.067547    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 16:46:46.068860 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.068849    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 07 16:46:46.070578 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.070563    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 16:46:46.071849 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.071837    2583 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 16:46:46.071972 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.071962    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 07 16:46:46.072009 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.071992    2583 kubelet.go:486] "Attempting to sync node with API server"
Jul 07 16:46:46.072009 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.072002    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 07 16:46:46.072076 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.072014    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 07 16:46:46.072076 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.072023    2583 kubelet.go:398] "Adding apiserver pod source"
Jul 07 16:46:46.072076 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.072032    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 07 16:46:46.072415 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.072403    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 16:46:46.072469 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.072447    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 16:46:46.075857 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.075819    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 07 16:46:46.077816 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.077802    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 07 16:46:46.077867 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.077834    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 07 16:46:46.079214 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.079186    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 07 16:46:46.079214 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.079209    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 07 16:46:46.079214 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.079217    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 07 16:46:46.079398 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.079225    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 07 16:46:46.079398 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.079238    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 07 16:46:46.079398 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.079246    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 07 16:46:46.079398 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.079257    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 07 16:46:46.079398 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.079264    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 07 16:46:46.079398 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.079274    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 07 16:46:46.079398 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.079281    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 07 16:46:46.079398 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.079293    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 07 16:46:46.079398 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.079300    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 07 16:46:46.080153 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.080141    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 07 16:46:46.080202 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.080156    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 07 16:46:46.083421 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.083405    2583 server.go:1268] "Started kubelet"
Jul 07 16:46:46.083573 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.083505    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 07 16:46:46.083644 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.083518    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 07 16:46:46.083644 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.083599    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 07 16:46:46.084296 ip-10-0-141-189 systemd[1]: Started Kubernetes Kubelet.
Jul 07 16:46:46.085770 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.085750    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 07 16:46:46.086831 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.086817    2583 server.go:310] "Adding debug handlers to kubelet server"
Jul 07 16:46:46.088096 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.088077    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-189.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 07 16:46:46.089344 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:46.089321    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-189.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 07 16:46:46.089458 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:46.089403    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 07 16:46:46.091058 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:46.091036    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 07 16:46:46.092007 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:46.090936    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-189.ec2.internal.18c00fc875d34c93  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-189.ec2.internal,UID:ip-10-0-141-189.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-189.ec2.internal,},FirstTimestamp:2026-07-07 16:46:46.083374227 +0000 UTC m=+0.383089893,LastTimestamp:2026-07-07 16:46:46.083374227 +0000 UTC m=+0.383089893,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-189.ec2.internal,}"
Jul 07 16:46:46.092702 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.092683    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 07 16:46:46.092702 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.092692    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 07 16:46:46.092834 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:46.092766    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-189.ec2.internal\" not found"
Jul 07 16:46:46.093464 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.093294    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 07 16:46:46.093551 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.093291    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 07 16:46:46.093551 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.093482    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 07 16:46:46.093682 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.093577    2583 factory.go:55] Registering systemd factory
Jul 07 16:46:46.093682 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.093593    2583 factory.go:223] Registration of the systemd container factory successfully
Jul 07 16:46:46.093682 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.093614    2583 reconstruct.go:97] "Volume reconstruction finished"
Jul 07 16:46:46.093682 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.093624    2583 reconciler.go:29] "Reconciler: start to sync state"
Jul 07 16:46:46.093835 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.093821    2583 factory.go:153] Registering CRI-O factory
Jul 07 16:46:46.093835 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.093833    2583 factory.go:223] Registration of the crio container factory successfully
Jul 07 16:46:46.093903 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.093880    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 07 16:46:46.093903 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.093897    2583 factory.go:103] Registering Raw factory
Jul 07 16:46:46.093963 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.093906    2583 manager.go:1196] Started watching for new ooms in manager
Jul 07 16:46:46.094956 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.094940    2583 manager.go:349] Starting recovery of all containers
Jul 07 16:46:46.104618 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.104536    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 07 16:46:46.104831 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:46.104787    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 07 16:46:46.105155 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:46.104990    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-189.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 07 16:46:46.107157 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.107106    2583 manager.go:354] Recovery completed
Jul 07 16:46:46.108571 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:46.108550    2583 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 07 16:46:46.111666 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.111654    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:46:46.113543 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.113527    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-189.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:46:46.113606 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.113558    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-189.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:46:46.113606 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.113572    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-189.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:46:46.113914 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.113895    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9fb2f"
Jul 07 16:46:46.114109 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.114095    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:46:46.114159 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.114096    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 07 16:46:46.114159 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.114123    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 07 16:46:46.114159 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.114138    2583 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 16:46:46.114758 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.114744    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-189.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:46:46.114818 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.114774    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-189.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:46:46.114818 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.114789    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-189.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:46:46.115657 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.115646    2583 policy_none.go:49] "None policy: Start"
Jul 07 16:46:46.115707 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.115661    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 07 16:46:46.115707 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.115671    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 07 16:46:46.115886 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:46.115824    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-189.ec2.internal.18c00fc8779fa4d8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-189.ec2.internal,UID:ip-10-0-141-189.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-189.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-189.ec2.internal,},FirstTimestamp:2026-07-07 16:46:46.113543384 +0000 UTC m=+0.413259051,LastTimestamp:2026-07-07 16:46:46.113543384 +0000 UTC m=+0.413259051,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-189.ec2.internal,}"
Jul 07 16:46:46.117047 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.117035    2583 policy_none.go:47] "Start"
Jul 07 16:46:46.124003 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.123986    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9fb2f"
Jul 07 16:46:46.126531 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:46.126465    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-189.ec2.internal.18c00fc8779ff9c1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-189.ec2.internal,UID:ip-10-0-141-189.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-189.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-189.ec2.internal,},FirstTimestamp:2026-07-07 16:46:46.113565121 +0000 UTC m=+0.413280788,LastTimestamp:2026-07-07 16:46:46.113565121 +0000 UTC m=+0.413280788,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-189.ec2.internal,}"
Jul 07 16:46:46.153066 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.153050    2583 manager.go:337] "Starting Device Plugin manager"
Jul 07 16:46:46.153194 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:46.153092    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 07 16:46:46.153194 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.153106    2583 server.go:87] "Starting device plugin registration server"
Jul 07 16:46:46.153416 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.153404    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 07 16:46:46.153507 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.153417    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 07 16:46:46.153549 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.153501    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 07 16:46:46.153594 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.153582    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 07 16:46:46.153594 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.153589    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 07 16:46:46.154141 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:46.154120    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 07 16:46:46.154210 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:46.154167    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-189.ec2.internal\" not found"
Jul 07 16:46:46.253885 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.253853    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:46:46.255079 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.254833    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-189.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:46:46.255079 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.254861    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-189.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:46:46.255079 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.254870    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-189.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:46:46.255079 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.254902    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-189.ec2.internal"
Jul 07 16:46:46.265538 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.265475    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-189.ec2.internal"
Jul 07 16:46:46.265538 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:46.265497    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-189.ec2.internal\": node \"ip-10-0-141-189.ec2.internal\" not found"
Jul 07 16:46:46.293823 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.293795    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 07 16:46:46.293929 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.293844    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 07 16:46:46.293929 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.293865    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 07 16:46:46.293929 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:46.293901    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 07 16:46:46.296892 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:46.296867    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-189.ec2.internal\" not found"
Jul 07 16:46:46.296988 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.296951    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 16:46:46.394421 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.394386    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-189.ec2.internal"]
Jul 07 16:46:46.394569 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.394485    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:46:46.395493 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.395467    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-189.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:46:46.395553 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.395501    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-189.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:46:46.395553 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.395515    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-189.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:46:46.397079 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.397066    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:46:46.397137 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:46.397121    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-189.ec2.internal\" not found"
Jul 07 16:46:46.397191 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.397176    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.ec2.internal"
Jul 07 16:46:46.397226 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.397207    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:46:46.397790 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.397776    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-189.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:46:46.397852 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.397781    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-189.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:46:46.397852 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.397834    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-189.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:46:46.397852 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.397847    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-189.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:46:46.397952 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.397809    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-189.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:46:46.397952 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.397883    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-189.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:46:46.399144 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.399131    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-189.ec2.internal"
Jul 07 16:46:46.399195 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.399155    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:46:46.399856 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.399841    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-189.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:46:46.399944 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.399866    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-189.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:46:46.399944 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.399876    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-189.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:46:46.428164 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:46.428141    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-189.ec2.internal\" not found" node="ip-10-0-141-189.ec2.internal"
Jul 07 16:46:46.432592 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:46.432575    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-189.ec2.internal\" not found" node="ip-10-0-141-189.ec2.internal"
Jul 07 16:46:46.495030 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.494995    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/89c5295fbef6e1eabe85a83f67a07bbd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-189.ec2.internal\" (UID: \"89c5295fbef6e1eabe85a83f67a07bbd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.ec2.internal"
Jul 07 16:46:46.495030 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.495026    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89c5295fbef6e1eabe85a83f67a07bbd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-189.ec2.internal\" (UID: \"89c5295fbef6e1eabe85a83f67a07bbd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.ec2.internal"
Jul 07 16:46:46.495229 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:46.495044    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e3a5ffb0c2c44b15a7860c5703342945-config\") pod \"kube-apiserver-proxy-ip-10-0-141-189.ec2.internal\" (UID: \"e3a5ffb0c2c44b15a7860c5703342945\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-189.ec2.internal"
Jul 07 16:46:46.498112 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:46.498095    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-189.ec2.internal\" not found"
Jul 07 16:46:47.882684 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:47.882650    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/89c5295fbef6e1eabe85a83f67a07bbd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-189.ec2.internal\" (UID: \"89c5295fbef6e1eabe85a83f67a07bbd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.ec2.internal"
Jul 07 16:46:47.882684 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:47.882617    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/89c5295fbef6e1eabe85a83f67a07bbd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-189.ec2.internal\" (UID: \"89c5295fbef6e1eabe85a83f67a07bbd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.ec2.internal"
Jul 07 16:46:47.882879 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:47.882739    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89c5295fbef6e1eabe85a83f67a07bbd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-189.ec2.internal\" (UID: \"89c5295fbef6e1eabe85a83f67a07bbd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.ec2.internal"
Jul 07 16:46:47.882879 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:47.882757    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e3a5ffb0c2c44b15a7860c5703342945-config\") pod \"kube-apiserver-proxy-ip-10-0-141-189.ec2.internal\" (UID: \"e3a5ffb0c2c44b15a7860c5703342945\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-189.ec2.internal"
Jul 07 16:46:47.882879 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:47.882786    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e3a5ffb0c2c44b15a7860c5703342945-config\") pod \"kube-apiserver-proxy-ip-10-0-141-189.ec2.internal\" (UID: \"e3a5ffb0c2c44b15a7860c5703342945\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-189.ec2.internal"
Jul 07 16:46:47.882879 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:47.882828    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89c5295fbef6e1eabe85a83f67a07bbd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-189.ec2.internal\" (UID: \"89c5295fbef6e1eabe85a83f67a07bbd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.ec2.internal"
Jul 07 16:46:47.885730 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:47.885708    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-189.ec2.internal\" not found"
Jul 07 16:46:47.986345 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:47.986304    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-189.ec2.internal\" not found"
Jul 07 16:46:48.018504 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:48.018466    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.ec2.internal"
Jul 07 16:46:48.024298 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:48.024148    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-189.ec2.internal"
Jul 07 16:46:48.086711 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:48.086637    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-189.ec2.internal\" not found"
Jul 07 16:46:48.187244 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:48.187208    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-189.ec2.internal\" not found"
Jul 07 16:46:48.287720 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:48.287679    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-189.ec2.internal\" not found"
Jul 07 16:46:48.316853 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:48.316824    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 07 16:46:48.317356 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:48.316960    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 16:46:48.380724 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:48.380655    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 16:46:48.387931 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:48.387908    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-189.ec2.internal\" not found"
Jul 07 16:46:48.412899 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:48.412861    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 16:41:47 +0000 UTC" deadline="2028-02-11 12:32:23.054917589 +0000 UTC"
Jul 07 16:46:48.412980 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:48.412898    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14011h45m34.642024352s"
Jul 07 16:46:48.484405 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:48.484377    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:46:48.485518 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:48.485497    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-189.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:46:48.485637 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:48.485528    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-189.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:46:48.485637 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:48.485538    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-189.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:46:48.488632 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:48.488613    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-189.ec2.internal\" not found"
Jul 07 16:46:48.522677 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:48.522657    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 16:46:48.589296 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:48.589267    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-189.ec2.internal\" not found"
Jul 07 16:46:48.613800 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:48.613773    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a5ffb0c2c44b15a7860c5703342945.slice/crio-5f73c38669104fe8ef61827920c3eefed716f20cb45339161a06e7dbd35bdd0a WatchSource:0}: Error finding container 5f73c38669104fe8ef61827920c3eefed716f20cb45339161a06e7dbd35bdd0a: Status 404 returned error can't find the container with id 5f73c38669104fe8ef61827920c3eefed716f20cb45339161a06e7dbd35bdd0a
Jul 07 16:46:48.614194 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:48.614173    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c5295fbef6e1eabe85a83f67a07bbd.slice/crio-c35296755f2cc61f7046880cd4b7fc437aa55b4e7631d1a30087d60373d66caf WatchSource:0}: Error finding container c35296755f2cc61f7046880cd4b7fc437aa55b4e7631d1a30087d60373d66caf: Status 404 returned error can't find the container with id c35296755f2cc61f7046880cd4b7fc437aa55b4e7631d1a30087d60373d66caf
Jul 07 16:46:48.621832 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:48.621812    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 16:46:48.690322 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:48.690297    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-189.ec2.internal\" not found"
Jul 07 16:46:48.737278 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:48.737251    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 16:46:48.780354 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:48.780318    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-189.ec2.internal"
Jul 07 16:46:48.792653 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:48.792628    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 16:46:48.793584 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:48.793571    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.ec2.internal"
Jul 07 16:46:48.805850 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:48.805832    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 16:46:49.127517 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.127427    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 16:46:49.360118 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.360086    2583 apiserver.go:52] "Watching apiserver"
Jul 07 16:46:49.368970 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.368944    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 07 16:46:49.371010 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.370972    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-kprb9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9","openshift-cluster-node-tuning-operator/tuned-xs8l4","openshift-image-registry/node-ca-mj2p8","openshift-multus/network-metrics-daemon-vv52w","openshift-ovn-kubernetes/ovnkube-node-mgf45","kube-system/global-pull-secret-syncer-x5chr","kube-system/kube-apiserver-proxy-ip-10-0-141-189.ec2.internal","openshift-dns/node-resolver-p6nfc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.ec2.internal","openshift-multus/multus-additional-cni-plugins-gbzdh","openshift-multus/multus-gsc2d","openshift-network-diagnostics/network-check-target-l7frr","openshift-network-operator/iptables-alerter-vdwvw"]
Jul 07 16:46:49.372630 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.372610    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:46:49.372754 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:49.372711    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:46:49.374030 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.374012    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:46:49.374171 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:49.374074    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:46:49.374922 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.374902    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.376125 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.376100    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mj2p8"
Jul 07 16:46:49.377402 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.377382    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:46:49.377574 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:49.377454    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:46:49.377949 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.377927    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 07 16:46:49.378966 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.378616    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kn8sv\""
Jul 07 16:46:49.378966 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.378839    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 07 16:46:49.379068 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.379027    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 07 16:46:49.379228 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.379210    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 07 16:46:49.379427 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.379410    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 07 16:46:49.379993 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.379948    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.380080 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.380024    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4fx66\""
Jul 07 16:46:49.380516 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.380497    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p6nfc"
Jul 07 16:46:49.380611 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.380519    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 07 16:46:49.381739 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.381722    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kprb9"
Jul 07 16:46:49.383612 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.383589    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9"
Jul 07 16:46:49.384639 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.384553    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vdwvw"
Jul 07 16:46:49.385540 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.385124    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 07 16:46:49.385540 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.385359    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-st6q8\""
Jul 07 16:46:49.385540 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.385431    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 07 16:46:49.386595 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.386578    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 07 16:46:49.386685 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.386620    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8b7nl\""
Jul 07 16:46:49.386685 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.386657    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 07 16:46:49.386806 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.386789    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 07 16:46:49.386872 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.386855    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 07 16:46:49.387684 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.387560    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gbzdh"
Jul 07 16:46:49.387797 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.387776    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lt2cz\""
Jul 07 16:46:49.387797 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.387792    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 07 16:46:49.387900 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.387797    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 07 16:46:49.389589 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.389568    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mhnn9\""
Jul 07 16:46:49.389683 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.389584    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 07 16:46:49.390122 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.389942    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 07 16:46:49.390348 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.390332    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 07 16:46:49.390547 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.390529    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 07 16:46:49.390645 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.390630    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-98ncg\""
Jul 07 16:46:49.391099 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.391076    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-host-cni-bin\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.391190 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.391115    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4541624c-a787-41a9-9d15-7b26c2aff7ec-ovnkube-config\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.391190 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.391156    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4541624c-a787-41a9-9d15-7b26c2aff7ec-ovnkube-script-lib\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.391190 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.391176    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 07 16:46:49.391377 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.391186    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-etc-sysconfig\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.391377 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.391302    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-run\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.391377 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.391326    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/89989a39-fdfd-431e-b96d-ae32d7ee482c-etc-tuned\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.391377 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.391359    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-run-systemd\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.391558 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.391393    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4541624c-a787-41a9-9d15-7b26c2aff7ec-ovn-node-metrics-cert\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.391558 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.391490    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7530471-b4ef-492d-b1bd-6393da0da34b-hosts-file\") pod \"node-resolver-p6nfc\" (UID: \"f7530471-b4ef-492d-b1bd-6393da0da34b\") " pod="openshift-dns/node-resolver-p6nfc"
Jul 07 16:46:49.391558 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.391532    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-lib-modules\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.391690 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.391573    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89989a39-fdfd-431e-b96d-ae32d7ee482c-tmp\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.391690 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.391642    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v465x\" (UniqueName: \"kubernetes.io/projected/20811d55-4cd6-4c95-9e50-e53a2b3bf2c7-kube-api-access-v465x\") pod \"node-ca-mj2p8\" (UID: \"20811d55-4cd6-4c95-9e50-e53a2b3bf2c7\") " pod="openshift-image-registry/node-ca-mj2p8"
Jul 07 16:46:49.391792 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.391698    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-systemd-units\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.391792 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.391736    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-var-lib-openvswitch\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.391889 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.391790    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 07 16:46:49.391889 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.391799    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-etc-openvswitch\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.391889 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.391834    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-etc-kubernetes\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.391889 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.391865    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-host\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.391889 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.391877    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 07 16:46:49.392164 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.391892    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-dbus\") pod \"global-pull-secret-syncer-x5chr\" (UID: \"41c32952-21a9-4e19-bc8a-ceee6a08ceeb\") " pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:46:49.392164 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.391939    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-original-pull-secret\") pod \"global-pull-secret-syncer-x5chr\" (UID: \"41c32952-21a9-4e19-bc8a-ceee6a08ceeb\") " pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:46:49.392164 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.391970    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwsmx\" (UniqueName: \"kubernetes.io/projected/b399a96b-729b-484d-8edf-e97618163cfa-kube-api-access-lwsmx\") pod \"network-metrics-daemon-vv52w\" (UID: \"b399a96b-729b-484d-8edf-e97618163cfa\") " pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:46:49.392164 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.392001    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-host-kubelet\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.392164 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.392026    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-host-cni-netd\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.392164 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.392059    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrhm9\" (UniqueName: \"kubernetes.io/projected/f7530471-b4ef-492d-b1bd-6393da0da34b-kube-api-access-wrhm9\") pod \"node-resolver-p6nfc\" (UID: \"f7530471-b4ef-492d-b1bd-6393da0da34b\") " pod="openshift-dns/node-resolver-p6nfc"
Jul 07 16:46:49.392164 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.392129    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-etc-sysctl-conf\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.392526 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.392191    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/20811d55-4cd6-4c95-9e50-e53a2b3bf2c7-serviceca\") pod \"node-ca-mj2p8\" (UID: \"20811d55-4cd6-4c95-9e50-e53a2b3bf2c7\") " pod="openshift-image-registry/node-ca-mj2p8"
Jul 07 16:46:49.392526 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.392243    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ae6f29a9-b432-4a71-a94a-27710a455ea6-agent-certs\") pod \"konnectivity-agent-kprb9\" (UID: \"ae6f29a9-b432-4a71-a94a-27710a455ea6\") " pod="kube-system/konnectivity-agent-kprb9"
Jul 07 16:46:49.392526 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.392286    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ae6f29a9-b432-4a71-a94a-27710a455ea6-konnectivity-ca\") pod \"konnectivity-agent-kprb9\" (UID: \"ae6f29a9-b432-4a71-a94a-27710a455ea6\") " pod="kube-system/konnectivity-agent-kprb9"
Jul 07 16:46:49.392526 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.392451    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-host-run-ovn-kubernetes\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.392526 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.392479    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 07 16:46:49.392526 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.392482    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4541624c-a787-41a9-9d15-7b26c2aff7ec-env-overrides\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.392526 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.392526    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-etc-modprobe-d\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.392872 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.392573    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-sys\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.392872 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.392622    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmz6p\" (UniqueName: \"kubernetes.io/projected/89989a39-fdfd-431e-b96d-ae32d7ee482c-kube-api-access-lmz6p\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.392872 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.392658    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7530471-b4ef-492d-b1bd-6393da0da34b-tmp-dir\") pod \"node-resolver-p6nfc\" (UID: \"f7530471-b4ef-492d-b1bd-6393da0da34b\") " pod="openshift-dns/node-resolver-p6nfc"
Jul 07 16:46:49.392872 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.392706    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-etc-sysctl-d\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.392872 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.392745    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-var-lib-kubelet\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.392872 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.392777    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20811d55-4cd6-4c95-9e50-e53a2b3bf2c7-host\") pod \"node-ca-mj2p8\" (UID: \"20811d55-4cd6-4c95-9e50-e53a2b3bf2c7\") " pod="openshift-image-registry/node-ca-mj2p8"
Jul 07 16:46:49.392872 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.392810    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-kubelet-config\") pod \"global-pull-secret-syncer-x5chr\" (UID: \"41c32952-21a9-4e19-bc8a-ceee6a08ceeb\") " pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:46:49.392872 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.392840    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b399a96b-729b-484d-8edf-e97618163cfa-metrics-certs\") pod \"network-metrics-daemon-vv52w\" (UID: \"b399a96b-729b-484d-8edf-e97618163cfa\") " pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:46:49.392872 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.392866    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26r9t\" (UniqueName: \"kubernetes.io/projected/9fc991d0-7b29-41d9-bf1a-7ba28833be28-kube-api-access-26r9t\") pod \"network-check-target-l7frr\" (UID: \"9fc991d0-7b29-41d9-bf1a-7ba28833be28\") " pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:46:49.393376 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.392933    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-host-run-netns\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.393376 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.393000    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-run-openvswitch\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.393376 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.393047    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-run-ovn\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.393376 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.393077    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-node-log\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.393376 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.393177    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-etc-systemd\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.393376 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.393210    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-host-slash\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.393376 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.393252    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-log-socket\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.393376 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.393284    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.393847 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.393420    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h9sm\" (UniqueName: \"kubernetes.io/projected/4541624c-a787-41a9-9d15-7b26c2aff7ec-kube-api-access-7h9sm\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.395030 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.395010    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-snskv\""
Jul 07 16:46:49.395163 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.395112    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 07 16:46:49.395235 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.395172    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.396421 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.396398    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 07 16:46:49.396511 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.396484    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 07 16:46:49.396511 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.396402    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 07 16:46:49.396751 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.396723    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 07 16:46:49.398924 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.398023    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 07 16:46:49.398924 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.398673    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bppwg\""
Jul 07 16:46:49.401538 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.401518    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 16:46:49.425057 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.425035    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-g2n2b"
Jul 07 16:46:49.434444 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.434416    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-g2n2b"
Jul 07 16:46:49.481596 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.481561    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 07 16:46:49.494062 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494029    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-var-lib-kubelet\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.494265 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494071    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20811d55-4cd6-4c95-9e50-e53a2b3bf2c7-host\") pod \"node-ca-mj2p8\" (UID: \"20811d55-4cd6-4c95-9e50-e53a2b3bf2c7\") " pod="openshift-image-registry/node-ca-mj2p8"
Jul 07 16:46:49.494265 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494151    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20811d55-4cd6-4c95-9e50-e53a2b3bf2c7-host\") pod \"node-ca-mj2p8\" (UID: \"20811d55-4cd6-4c95-9e50-e53a2b3bf2c7\") " pod="openshift-image-registry/node-ca-mj2p8"
Jul 07 16:46:49.494265 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494178    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-var-lib-kubelet\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.494265 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494196    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b399a96b-729b-484d-8edf-e97618163cfa-metrics-certs\") pod \"network-metrics-daemon-vv52w\" (UID: \"b399a96b-729b-484d-8edf-e97618163cfa\") " pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:46:49.494265 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494232    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f403e1aa-f12e-4df4-9626-79a45ab81549-cni-binary-copy\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.494595 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494273    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26r9t\" (UniqueName: \"kubernetes.io/projected/9fc991d0-7b29-41d9-bf1a-7ba28833be28-kube-api-access-26r9t\") pod \"network-check-target-l7frr\" (UID: \"9fc991d0-7b29-41d9-bf1a-7ba28833be28\") " pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:46:49.494595 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494298    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-run-openvswitch\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.494595 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494339    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-run-ovn\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.494595 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494365    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-run-openvswitch\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.494595 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494368    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-etc-systemd\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.494595 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494416    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4xtc\" (UniqueName: \"kubernetes.io/projected/f403e1aa-f12e-4df4-9626-79a45ab81549-kube-api-access-j4xtc\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.494595 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494425    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-etc-systemd\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.494595 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:49.494341    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:46:49.494595 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494443    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-host-slash\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.494595 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494475    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-log-socket\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.494595 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494496    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.494595 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494517    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-system-cni-dir\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.494595 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494537    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-multus-conf-dir\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.494595 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494555    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f403e1aa-f12e-4df4-9626-79a45ab81549-multus-daemon-config\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.494595 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494546    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-host-slash\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.494595 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494578    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/779e8dac-267c-4848-ac3f-4e8e0c9a0a0a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zdmv9\" (UID: \"779e8dac-267c-4848-ac3f-4e8e0c9a0a0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9"
Jul 07 16:46:49.494595 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494454    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-run-ovn\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.494595 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494583    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-log-socket\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.495454 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494604    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/99e5a676-4f78-4532-a5a4-87f1e55d2cba-cnibin\") pod \"multus-additional-cni-plugins-gbzdh\" (UID: \"99e5a676-4f78-4532-a5a4-87f1e55d2cba\") " pod="openshift-multus/multus-additional-cni-plugins-gbzdh"
Jul 07 16:46:49.495454 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494620    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4541624c-a787-41a9-9d15-7b26c2aff7ec-ovnkube-config\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.495454 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494633    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.495454 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:49.494644    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b399a96b-729b-484d-8edf-e97618163cfa-metrics-certs podName:b399a96b-729b-484d-8edf-e97618163cfa nodeName:}" failed. No retries permitted until 2026-07-07 16:46:49.994620033 +0000 UTC m=+3.007629847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b399a96b-729b-484d-8edf-e97618163cfa-metrics-certs") pod "network-metrics-daemon-vv52w" (UID: "b399a96b-729b-484d-8edf-e97618163cfa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:46:49.495454 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494680    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4541624c-a787-41a9-9d15-7b26c2aff7ec-ovnkube-script-lib\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.495454 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494701    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/89989a39-fdfd-431e-b96d-ae32d7ee482c-etc-tuned\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.495454 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494718    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-etc-kubernetes\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.495454 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494735    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/779e8dac-267c-4848-ac3f-4e8e0c9a0a0a-sys-fs\") pod \"aws-ebs-csi-driver-node-zdmv9\" (UID: \"779e8dac-267c-4848-ac3f-4e8e0c9a0a0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9"
Jul 07 16:46:49.495454 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494759    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/99e5a676-4f78-4532-a5a4-87f1e55d2cba-os-release\") pod \"multus-additional-cni-plugins-gbzdh\" (UID: \"99e5a676-4f78-4532-a5a4-87f1e55d2cba\") " pod="openshift-multus/multus-additional-cni-plugins-gbzdh"
Jul 07 16:46:49.495454 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494804    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpmb4\" (UniqueName: \"kubernetes.io/projected/e7674168-32c3-4c84-a753-2af1455d5c30-kube-api-access-vpmb4\") pod \"iptables-alerter-vdwvw\" (UID: \"e7674168-32c3-4c84-a753-2af1455d5c30\") " pod="openshift-network-operator/iptables-alerter-vdwvw"
Jul 07 16:46:49.495454 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494830    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-run-systemd\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.495454 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494851    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7530471-b4ef-492d-b1bd-6393da0da34b-hosts-file\") pod \"node-resolver-p6nfc\" (UID: \"f7530471-b4ef-492d-b1bd-6393da0da34b\") " pod="openshift-dns/node-resolver-p6nfc"
Jul 07 16:46:49.495454 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494885    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89989a39-fdfd-431e-b96d-ae32d7ee482c-tmp\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.495454 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494915    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-host-var-lib-cni-multus\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.495454 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494936    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-hostroot\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.495454 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.494974    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-systemd-units\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.496300 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495002    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-etc-openvswitch\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.496300 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495046    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-etc-kubernetes\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.496300 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495074    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-dbus\") pod \"global-pull-secret-syncer-x5chr\" (UID: \"41c32952-21a9-4e19-bc8a-ceee6a08ceeb\") " pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:46:49.496300 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495066    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 07 16:46:49.496300 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495095    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7530471-b4ef-492d-b1bd-6393da0da34b-hosts-file\") pod \"node-resolver-p6nfc\" (UID: \"f7530471-b4ef-492d-b1bd-6393da0da34b\") " pod="openshift-dns/node-resolver-p6nfc"
Jul 07 16:46:49.496300 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495100    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-original-pull-secret\") pod \"global-pull-secret-syncer-x5chr\" (UID: \"41c32952-21a9-4e19-bc8a-ceee6a08ceeb\") " pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:46:49.496300 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495168    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-host-run-netns\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.496300 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:49.495186    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:46:49.496300 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495198    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4s2s\" (UniqueName: \"kubernetes.io/projected/779e8dac-267c-4848-ac3f-4e8e0c9a0a0a-kube-api-access-s4s2s\") pod \"aws-ebs-csi-driver-node-zdmv9\" (UID: \"779e8dac-267c-4848-ac3f-4e8e0c9a0a0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9"
Jul 07 16:46:49.496300 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495212    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4541624c-a787-41a9-9d15-7b26c2aff7ec-ovnkube-config\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.496300 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495229    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e7674168-32c3-4c84-a753-2af1455d5c30-iptables-alerter-script\") pod \"iptables-alerter-vdwvw\" (UID: \"e7674168-32c3-4c84-a753-2af1455d5c30\") " pod="openshift-network-operator/iptables-alerter-vdwvw"
Jul 07 16:46:49.496300 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:49.495241    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-original-pull-secret podName:41c32952-21a9-4e19-bc8a-ceee6a08ceeb nodeName:}" failed. No retries permitted until 2026-07-07 16:46:49.995222797 +0000 UTC m=+3.008232605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-original-pull-secret") pod "global-pull-secret-syncer-x5chr" (UID: "41c32952-21a9-4e19-bc8a-ceee6a08ceeb") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:46:49.496300 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495271    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-host-kubelet\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.496300 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495327    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-run-systemd\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.496300 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495336    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-etc-openvswitch\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.496300 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495370    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-host-cni-netd\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.496300 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495419    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrhm9\" (UniqueName: \"kubernetes.io/projected/f7530471-b4ef-492d-b1bd-6393da0da34b-kube-api-access-wrhm9\") pod \"node-resolver-p6nfc\" (UID: \"f7530471-b4ef-492d-b1bd-6393da0da34b\") " pod="openshift-dns/node-resolver-p6nfc"
Jul 07 16:46:49.497091 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495418    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4541624c-a787-41a9-9d15-7b26c2aff7ec-ovnkube-script-lib\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.497091 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495453    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-etc-kubernetes\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.497091 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495460    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-dbus\") pod \"global-pull-secret-syncer-x5chr\" (UID: \"41c32952-21a9-4e19-bc8a-ceee6a08ceeb\") " pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:46:49.497091 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495466    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-systemd-units\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.497091 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495485    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ae6f29a9-b432-4a71-a94a-27710a455ea6-agent-certs\") pod \"konnectivity-agent-kprb9\" (UID: \"ae6f29a9-b432-4a71-a94a-27710a455ea6\") " pod="kube-system/konnectivity-agent-kprb9"
Jul 07 16:46:49.497091 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495541    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-host-kubelet\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.497091 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495516    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ae6f29a9-b432-4a71-a94a-27710a455ea6-konnectivity-ca\") pod \"konnectivity-agent-kprb9\" (UID: \"ae6f29a9-b432-4a71-a94a-27710a455ea6\") " pod="kube-system/konnectivity-agent-kprb9"
Jul 07 16:46:49.497091 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495588    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-host-cni-netd\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.497091 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495611    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-host-run-k8s-cni-cncf-io\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.497091 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495644    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-host-var-lib-cni-bin\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.497091 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495676    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-host-run-multus-certs\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.497091 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495703    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-host-run-ovn-kubernetes\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.497091 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495774    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-host-run-ovn-kubernetes\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.497091 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495817    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-etc-modprobe-d\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.497091 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495865    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-sys\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.497091 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495914    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-etc-modprobe-d\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.497091 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495944    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7530471-b4ef-492d-b1bd-6393da0da34b-tmp-dir\") pod \"node-resolver-p6nfc\" (UID: \"f7530471-b4ef-492d-b1bd-6393da0da34b\") " pod="openshift-dns/node-resolver-p6nfc"
Jul 07 16:46:49.497091 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495944    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-sys\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.497928 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.495985    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-kubelet-config\") pod \"global-pull-secret-syncer-x5chr\" (UID: \"41c32952-21a9-4e19-bc8a-ceee6a08ceeb\") " pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:46:49.497928 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496013    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/99e5a676-4f78-4532-a5a4-87f1e55d2cba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gbzdh\" (UID: \"99e5a676-4f78-4532-a5a4-87f1e55d2cba\") " pod="openshift-multus/multus-additional-cni-plugins-gbzdh"
Jul 07 16:46:49.497928 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496044    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-host-run-netns\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.497928 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496074    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-node-log\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.497928 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496113    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/99e5a676-4f78-4532-a5a4-87f1e55d2cba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gbzdh\" (UID: \"99e5a676-4f78-4532-a5a4-87f1e55d2cba\") " pod="openshift-multus/multus-additional-cni-plugins-gbzdh"
Jul 07 16:46:49.497928 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496160    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7674168-32c3-4c84-a753-2af1455d5c30-host-slash\") pod \"iptables-alerter-vdwvw\" (UID: \"e7674168-32c3-4c84-a753-2af1455d5c30\") " pod="openshift-network-operator/iptables-alerter-vdwvw"
Jul 07 16:46:49.497928 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496191    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h9sm\" (UniqueName: \"kubernetes.io/projected/4541624c-a787-41a9-9d15-7b26c2aff7ec-kube-api-access-7h9sm\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.497928 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496215    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ae6f29a9-b432-4a71-a94a-27710a455ea6-konnectivity-ca\") pod \"konnectivity-agent-kprb9\" (UID: \"ae6f29a9-b432-4a71-a94a-27710a455ea6\") " pod="kube-system/konnectivity-agent-kprb9"
Jul 07 16:46:49.497928 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496221    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-host-cni-bin\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.497928 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496255    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-host-cni-bin\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.497928 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496272    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-etc-sysconfig\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.497928 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496297    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-run\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.497928 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496305    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-host-run-netns\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.497928 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496323    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-multus-cni-dir\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.497928 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496352    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-node-log\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.497928 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496365    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-host-var-lib-kubelet\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.497928 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496391    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/99e5a676-4f78-4532-a5a4-87f1e55d2cba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gbzdh\" (UID: \"99e5a676-4f78-4532-a5a4-87f1e55d2cba\") " pod="openshift-multus/multus-additional-cni-plugins-gbzdh"
Jul 07 16:46:49.498985 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496433    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tvfm\" (UniqueName: \"kubernetes.io/projected/99e5a676-4f78-4532-a5a4-87f1e55d2cba-kube-api-access-7tvfm\") pod \"multus-additional-cni-plugins-gbzdh\" (UID: \"99e5a676-4f78-4532-a5a4-87f1e55d2cba\") " pod="openshift-multus/multus-additional-cni-plugins-gbzdh"
Jul 07 16:46:49.498985 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496481    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4541624c-a787-41a9-9d15-7b26c2aff7ec-ovn-node-metrics-cert\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.498985 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496540    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-lib-modules\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.498985 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496552    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-kubelet-config\") pod \"global-pull-secret-syncer-x5chr\" (UID: \"41c32952-21a9-4e19-bc8a-ceee6a08ceeb\") " pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:46:49.498985 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496571    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7530471-b4ef-492d-b1bd-6393da0da34b-tmp-dir\") pod \"node-resolver-p6nfc\" (UID: \"f7530471-b4ef-492d-b1bd-6393da0da34b\") " pod="openshift-dns/node-resolver-p6nfc"
Jul 07 16:46:49.498985 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496583    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v465x\" (UniqueName: \"kubernetes.io/projected/20811d55-4cd6-4c95-9e50-e53a2b3bf2c7-kube-api-access-v465x\") pod \"node-ca-mj2p8\" (UID: \"20811d55-4cd6-4c95-9e50-e53a2b3bf2c7\") " pod="openshift-image-registry/node-ca-mj2p8"
Jul 07 16:46:49.498985 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496620    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-etc-sysconfig\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.498985 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496636    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-run\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.498985 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496654    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-os-release\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.498985 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496673    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/779e8dac-267c-4848-ac3f-4e8e0c9a0a0a-device-dir\") pod \"aws-ebs-csi-driver-node-zdmv9\" (UID: \"779e8dac-267c-4848-ac3f-4e8e0c9a0a0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9"
Jul 07 16:46:49.498985 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496692    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-var-lib-openvswitch\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.498985 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496714    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-host\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.498985 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496737    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwsmx\" (UniqueName: \"kubernetes.io/projected/b399a96b-729b-484d-8edf-e97618163cfa-kube-api-access-lwsmx\") pod \"network-metrics-daemon-vv52w\" (UID: \"b399a96b-729b-484d-8edf-e97618163cfa\") " pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:46:49.498985 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496754    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/779e8dac-267c-4848-ac3f-4e8e0c9a0a0a-registration-dir\") pod \"aws-ebs-csi-driver-node-zdmv9\" (UID: \"779e8dac-267c-4848-ac3f-4e8e0c9a0a0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9"
Jul 07 16:46:49.498985 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496768    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/779e8dac-267c-4848-ac3f-4e8e0c9a0a0a-etc-selinux\") pod \"aws-ebs-csi-driver-node-zdmv9\" (UID: \"779e8dac-267c-4848-ac3f-4e8e0c9a0a0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9"
Jul 07 16:46:49.498985 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496783    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99e5a676-4f78-4532-a5a4-87f1e55d2cba-system-cni-dir\") pod \"multus-additional-cni-plugins-gbzdh\" (UID: \"99e5a676-4f78-4532-a5a4-87f1e55d2cba\") " pod="openshift-multus/multus-additional-cni-plugins-gbzdh"
Jul 07 16:46:49.498985 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496801    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4541624c-a787-41a9-9d15-7b26c2aff7ec-var-lib-openvswitch\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.499824 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496809    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-etc-sysctl-conf\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.499824 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496838    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/20811d55-4cd6-4c95-9e50-e53a2b3bf2c7-serviceca\") pod \"node-ca-mj2p8\" (UID: \"20811d55-4cd6-4c95-9e50-e53a2b3bf2c7\") " pod="openshift-image-registry/node-ca-mj2p8"
Jul 07 16:46:49.499824 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496864    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-multus-socket-dir-parent\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.499824 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496892    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/779e8dac-267c-4848-ac3f-4e8e0c9a0a0a-socket-dir\") pod \"aws-ebs-csi-driver-node-zdmv9\" (UID: \"779e8dac-267c-4848-ac3f-4e8e0c9a0a0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9"
Jul 07 16:46:49.499824 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496884    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-host\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.499824 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496915    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-etc-sysctl-conf\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.499824 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496921    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4541624c-a787-41a9-9d15-7b26c2aff7ec-env-overrides\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.499824 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496766    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-lib-modules\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.499824 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.496955    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmz6p\" (UniqueName: \"kubernetes.io/projected/89989a39-fdfd-431e-b96d-ae32d7ee482c-kube-api-access-lmz6p\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.499824 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.497103    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-cnibin\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.499824 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.497169    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/99e5a676-4f78-4532-a5a4-87f1e55d2cba-cni-binary-copy\") pod \"multus-additional-cni-plugins-gbzdh\" (UID: \"99e5a676-4f78-4532-a5a4-87f1e55d2cba\") " pod="openshift-multus/multus-additional-cni-plugins-gbzdh"
Jul 07 16:46:49.499824 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.497308    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-etc-sysctl-d\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.499824 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.497479    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/89989a39-fdfd-431e-b96d-ae32d7ee482c-etc-sysctl-d\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.499824 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.497483    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/20811d55-4cd6-4c95-9e50-e53a2b3bf2c7-serviceca\") pod \"node-ca-mj2p8\" (UID: \"20811d55-4cd6-4c95-9e50-e53a2b3bf2c7\") " pod="openshift-image-registry/node-ca-mj2p8"
Jul 07 16:46:49.499824 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.497714    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4541624c-a787-41a9-9d15-7b26c2aff7ec-env-overrides\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.499824 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.499158    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89989a39-fdfd-431e-b96d-ae32d7ee482c-tmp\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.499824 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.499205    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/89989a39-fdfd-431e-b96d-ae32d7ee482c-etc-tuned\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.500500 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.499884    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4541624c-a787-41a9-9d15-7b26c2aff7ec-ovn-node-metrics-cert\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.500532 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.500498    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ae6f29a9-b432-4a71-a94a-27710a455ea6-agent-certs\") pod \"konnectivity-agent-kprb9\" (UID: \"ae6f29a9-b432-4a71-a94a-27710a455ea6\") " pod="kube-system/konnectivity-agent-kprb9"
Jul 07 16:46:49.502957 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:49.502860    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:46:49.502957 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:49.502882    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:46:49.502957 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:49.502894    2583 projected.go:196] Error preparing data for projected volume kube-api-access-26r9t for pod openshift-network-diagnostics/network-check-target-l7frr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:46:49.502957 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:49.502937    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9fc991d0-7b29-41d9-bf1a-7ba28833be28-kube-api-access-26r9t podName:9fc991d0-7b29-41d9-bf1a-7ba28833be28 nodeName:}" failed. No retries permitted until 2026-07-07 16:46:50.002925269 +0000 UTC m=+3.015935087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-26r9t" (UniqueName: "kubernetes.io/projected/9fc991d0-7b29-41d9-bf1a-7ba28833be28-kube-api-access-26r9t") pod "network-check-target-l7frr" (UID: "9fc991d0-7b29-41d9-bf1a-7ba28833be28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:46:49.505195 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.505174    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrhm9\" (UniqueName: \"kubernetes.io/projected/f7530471-b4ef-492d-b1bd-6393da0da34b-kube-api-access-wrhm9\") pod \"node-resolver-p6nfc\" (UID: \"f7530471-b4ef-492d-b1bd-6393da0da34b\") " pod="openshift-dns/node-resolver-p6nfc"
Jul 07 16:46:49.509551 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.509531    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h9sm\" (UniqueName: \"kubernetes.io/projected/4541624c-a787-41a9-9d15-7b26c2aff7ec-kube-api-access-7h9sm\") pod \"ovnkube-node-mgf45\" (UID: \"4541624c-a787-41a9-9d15-7b26c2aff7ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.509637 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.509577    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmz6p\" (UniqueName: \"kubernetes.io/projected/89989a39-fdfd-431e-b96d-ae32d7ee482c-kube-api-access-lmz6p\") pod \"tuned-xs8l4\" (UID: \"89989a39-fdfd-431e-b96d-ae32d7ee482c\") " pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.509637 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.509599    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwsmx\" (UniqueName: \"kubernetes.io/projected/b399a96b-729b-484d-8edf-e97618163cfa-kube-api-access-lwsmx\") pod \"network-metrics-daemon-vv52w\" (UID: \"b399a96b-729b-484d-8edf-e97618163cfa\") " pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:46:49.509898 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.509880    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v465x\" (UniqueName: \"kubernetes.io/projected/20811d55-4cd6-4c95-9e50-e53a2b3bf2c7-kube-api-access-v465x\") pod \"node-ca-mj2p8\" (UID: \"20811d55-4cd6-4c95-9e50-e53a2b3bf2c7\") " pod="openshift-image-registry/node-ca-mj2p8"
Jul 07 16:46:49.586455 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.586395    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.ec2.internal" event={"ID":"89c5295fbef6e1eabe85a83f67a07bbd","Type":"ContainerStarted","Data":"c35296755f2cc61f7046880cd4b7fc437aa55b4e7631d1a30087d60373d66caf"}
Jul 07 16:46:49.587720 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.587692    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-189.ec2.internal" event={"ID":"e3a5ffb0c2c44b15a7860c5703342945","Type":"ContainerStarted","Data":"5f73c38669104fe8ef61827920c3eefed716f20cb45339161a06e7dbd35bdd0a"}
Jul 07 16:46:49.597680 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.597639    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/99e5a676-4f78-4532-a5a4-87f1e55d2cba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gbzdh\" (UID: \"99e5a676-4f78-4532-a5a4-87f1e55d2cba\") " pod="openshift-multus/multus-additional-cni-plugins-gbzdh"
Jul 07 16:46:49.597893 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.597696    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/99e5a676-4f78-4532-a5a4-87f1e55d2cba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gbzdh\" (UID: \"99e5a676-4f78-4532-a5a4-87f1e55d2cba\") " pod="openshift-multus/multus-additional-cni-plugins-gbzdh"
Jul 07 16:46:49.597893 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.597725    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7674168-32c3-4c84-a753-2af1455d5c30-host-slash\") pod \"iptables-alerter-vdwvw\" (UID: \"e7674168-32c3-4c84-a753-2af1455d5c30\") " pod="openshift-network-operator/iptables-alerter-vdwvw"
Jul 07 16:46:49.597893 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.597758    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-multus-cni-dir\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.597893 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.597782    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-host-var-lib-kubelet\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.597893 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.597812    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/99e5a676-4f78-4532-a5a4-87f1e55d2cba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gbzdh\" (UID: \"99e5a676-4f78-4532-a5a4-87f1e55d2cba\") " pod="openshift-multus/multus-additional-cni-plugins-gbzdh"
Jul 07 16:46:49.597893 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.597838    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tvfm\" (UniqueName: \"kubernetes.io/projected/99e5a676-4f78-4532-a5a4-87f1e55d2cba-kube-api-access-7tvfm\") pod \"multus-additional-cni-plugins-gbzdh\" (UID: \"99e5a676-4f78-4532-a5a4-87f1e55d2cba\") " pod="openshift-multus/multus-additional-cni-plugins-gbzdh"
Jul 07 16:46:49.597893 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.597880    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-multus-cni-dir\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.598269 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.597878    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-host-var-lib-kubelet\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.598269 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.597877    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7674168-32c3-4c84-a753-2af1455d5c30-host-slash\") pod \"iptables-alerter-vdwvw\" (UID: \"e7674168-32c3-4c84-a753-2af1455d5c30\") " pod="openshift-network-operator/iptables-alerter-vdwvw"
Jul 07 16:46:49.598269 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.597946    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-os-release\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.598269 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.597951    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/99e5a676-4f78-4532-a5a4-87f1e55d2cba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gbzdh\" (UID: \"99e5a676-4f78-4532-a5a4-87f1e55d2cba\") " pod="openshift-multus/multus-additional-cni-plugins-gbzdh"
Jul 07 16:46:49.598269 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.597971    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/779e8dac-267c-4848-ac3f-4e8e0c9a0a0a-device-dir\") pod \"aws-ebs-csi-driver-node-zdmv9\" (UID: \"779e8dac-267c-4848-ac3f-4e8e0c9a0a0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9"
Jul 07 16:46:49.598269 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.597999    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/779e8dac-267c-4848-ac3f-4e8e0c9a0a0a-registration-dir\") pod \"aws-ebs-csi-driver-node-zdmv9\" (UID: \"779e8dac-267c-4848-ac3f-4e8e0c9a0a0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9"
Jul 07 16:46:49.598269 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598022    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/779e8dac-267c-4848-ac3f-4e8e0c9a0a0a-etc-selinux\") pod \"aws-ebs-csi-driver-node-zdmv9\" (UID: \"779e8dac-267c-4848-ac3f-4e8e0c9a0a0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9"
Jul 07 16:46:49.598269 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598042    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-os-release\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.598269 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598049    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99e5a676-4f78-4532-a5a4-87f1e55d2cba-system-cni-dir\") pod \"multus-additional-cni-plugins-gbzdh\" (UID: \"99e5a676-4f78-4532-a5a4-87f1e55d2cba\") " pod="openshift-multus/multus-additional-cni-plugins-gbzdh"
Jul 07 16:46:49.598269 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598065    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/779e8dac-267c-4848-ac3f-4e8e0c9a0a0a-registration-dir\") pod \"aws-ebs-csi-driver-node-zdmv9\" (UID: \"779e8dac-267c-4848-ac3f-4e8e0c9a0a0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9"
Jul 07 16:46:49.598269 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598082    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-multus-socket-dir-parent\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.598269 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598065    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/779e8dac-267c-4848-ac3f-4e8e0c9a0a0a-device-dir\") pod \"aws-ebs-csi-driver-node-zdmv9\" (UID: \"779e8dac-267c-4848-ac3f-4e8e0c9a0a0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9"
Jul 07 16:46:49.598269 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598101    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99e5a676-4f78-4532-a5a4-87f1e55d2cba-system-cni-dir\") pod \"multus-additional-cni-plugins-gbzdh\" (UID: \"99e5a676-4f78-4532-a5a4-87f1e55d2cba\") " pod="openshift-multus/multus-additional-cni-plugins-gbzdh"
Jul 07 16:46:49.598269 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598108    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/779e8dac-267c-4848-ac3f-4e8e0c9a0a0a-socket-dir\") pod \"aws-ebs-csi-driver-node-zdmv9\" (UID: \"779e8dac-267c-4848-ac3f-4e8e0c9a0a0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9"
Jul 07 16:46:49.598269 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598162    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-multus-socket-dir-parent\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.598269 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598170    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-cnibin\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.598269 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598186    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/779e8dac-267c-4848-ac3f-4e8e0c9a0a0a-etc-selinux\") pod \"aws-ebs-csi-driver-node-zdmv9\" (UID: \"779e8dac-267c-4848-ac3f-4e8e0c9a0a0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9"
Jul 07 16:46:49.599045 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598196    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/99e5a676-4f78-4532-a5a4-87f1e55d2cba-cni-binary-copy\") pod \"multus-additional-cni-plugins-gbzdh\" (UID: \"99e5a676-4f78-4532-a5a4-87f1e55d2cba\") " pod="openshift-multus/multus-additional-cni-plugins-gbzdh"
Jul 07 16:46:49.599045 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598236    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/779e8dac-267c-4848-ac3f-4e8e0c9a0a0a-socket-dir\") pod \"aws-ebs-csi-driver-node-zdmv9\" (UID: \"779e8dac-267c-4848-ac3f-4e8e0c9a0a0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9"
Jul 07 16:46:49.599045 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598242    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f403e1aa-f12e-4df4-9626-79a45ab81549-cni-binary-copy\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.599045 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598259    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-cnibin\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.599045 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598300    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4xtc\" (UniqueName: \"kubernetes.io/projected/f403e1aa-f12e-4df4-9626-79a45ab81549-kube-api-access-j4xtc\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.599045 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598327    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-system-cni-dir\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.599045 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598350    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-multus-conf-dir\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.599045 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598372    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f403e1aa-f12e-4df4-9626-79a45ab81549-multus-daemon-config\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.599045 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598393    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/779e8dac-267c-4848-ac3f-4e8e0c9a0a0a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zdmv9\" (UID: \"779e8dac-267c-4848-ac3f-4e8e0c9a0a0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9"
Jul 07 16:46:49.599045 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598389    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/99e5a676-4f78-4532-a5a4-87f1e55d2cba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gbzdh\" (UID: \"99e5a676-4f78-4532-a5a4-87f1e55d2cba\") " pod="openshift-multus/multus-additional-cni-plugins-gbzdh"
Jul 07 16:46:49.599045 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598413    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/99e5a676-4f78-4532-a5a4-87f1e55d2cba-cnibin\") pod \"multus-additional-cni-plugins-gbzdh\" (UID: \"99e5a676-4f78-4532-a5a4-87f1e55d2cba\") " pod="openshift-multus/multus-additional-cni-plugins-gbzdh"
Jul 07 16:46:49.599045 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598429    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-system-cni-dir\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.599045 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598438    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-etc-kubernetes\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.599045 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598466    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/99e5a676-4f78-4532-a5a4-87f1e55d2cba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gbzdh\" (UID: \"99e5a676-4f78-4532-a5a4-87f1e55d2cba\") " pod="openshift-multus/multus-additional-cni-plugins-gbzdh"
Jul 07 16:46:49.599045 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598514    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/779e8dac-267c-4848-ac3f-4e8e0c9a0a0a-sys-fs\") pod \"aws-ebs-csi-driver-node-zdmv9\" (UID: \"779e8dac-267c-4848-ac3f-4e8e0c9a0a0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9"
Jul 07 16:46:49.599045 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598521    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/99e5a676-4f78-4532-a5a4-87f1e55d2cba-cnibin\") pod \"multus-additional-cni-plugins-gbzdh\" (UID: \"99e5a676-4f78-4532-a5a4-87f1e55d2cba\") " pod="openshift-multus/multus-additional-cni-plugins-gbzdh"
Jul 07 16:46:49.599045 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598477    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-etc-kubernetes\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.599935 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598476    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/779e8dac-267c-4848-ac3f-4e8e0c9a0a0a-sys-fs\") pod \"aws-ebs-csi-driver-node-zdmv9\" (UID: \"779e8dac-267c-4848-ac3f-4e8e0c9a0a0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9"
Jul 07 16:46:49.599935 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598543    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/779e8dac-267c-4848-ac3f-4e8e0c9a0a0a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zdmv9\" (UID: \"779e8dac-267c-4848-ac3f-4e8e0c9a0a0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9"
Jul 07 16:46:49.599935 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598567    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/99e5a676-4f78-4532-a5a4-87f1e55d2cba-os-release\") pod \"multus-additional-cni-plugins-gbzdh\" (UID: \"99e5a676-4f78-4532-a5a4-87f1e55d2cba\") " pod="openshift-multus/multus-additional-cni-plugins-gbzdh"
Jul 07 16:46:49.599935 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598595    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpmb4\" (UniqueName: \"kubernetes.io/projected/e7674168-32c3-4c84-a753-2af1455d5c30-kube-api-access-vpmb4\") pod \"iptables-alerter-vdwvw\" (UID: \"e7674168-32c3-4c84-a753-2af1455d5c30\") " pod="openshift-network-operator/iptables-alerter-vdwvw"
Jul 07 16:46:49.599935 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598615    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/99e5a676-4f78-4532-a5a4-87f1e55d2cba-os-release\") pod \"multus-additional-cni-plugins-gbzdh\" (UID: \"99e5a676-4f78-4532-a5a4-87f1e55d2cba\") " pod="openshift-multus/multus-additional-cni-plugins-gbzdh"
Jul 07 16:46:49.599935 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598624    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-host-var-lib-cni-multus\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.599935 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598568    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-multus-conf-dir\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.599935 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598664    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-hostroot\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.599935 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598674    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-host-var-lib-cni-multus\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.599935 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598706    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-host-run-netns\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.599935 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598732    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4s2s\" (UniqueName: \"kubernetes.io/projected/779e8dac-267c-4848-ac3f-4e8e0c9a0a0a-kube-api-access-s4s2s\") pod \"aws-ebs-csi-driver-node-zdmv9\" (UID: \"779e8dac-267c-4848-ac3f-4e8e0c9a0a0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9"
Jul 07 16:46:49.599935 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598756    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-hostroot\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.599935 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598758    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e7674168-32c3-4c84-a753-2af1455d5c30-iptables-alerter-script\") pod \"iptables-alerter-vdwvw\" (UID: \"e7674168-32c3-4c84-a753-2af1455d5c30\") " pod="openshift-network-operator/iptables-alerter-vdwvw"
Jul 07 16:46:49.599935 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598795    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-host-run-netns\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.599935 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598816    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-host-run-k8s-cni-cncf-io\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.599935 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598839    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/99e5a676-4f78-4532-a5a4-87f1e55d2cba-cni-binary-copy\") pod \"multus-additional-cni-plugins-gbzdh\" (UID: \"99e5a676-4f78-4532-a5a4-87f1e55d2cba\") " pod="openshift-multus/multus-additional-cni-plugins-gbzdh"
Jul 07 16:46:49.599935 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598860    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-host-var-lib-cni-bin\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.600734 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598865    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f403e1aa-f12e-4df4-9626-79a45ab81549-multus-daemon-config\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.600734 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598867    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f403e1aa-f12e-4df4-9626-79a45ab81549-cni-binary-copy\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.600734 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598905    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-host-var-lib-cni-bin\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.600734 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598912    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-host-run-k8s-cni-cncf-io\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.600734 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598899    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-host-run-multus-certs\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.600734 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.598954    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f403e1aa-f12e-4df4-9626-79a45ab81549-host-run-multus-certs\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.600734 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.599279    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e7674168-32c3-4c84-a753-2af1455d5c30-iptables-alerter-script\") pod \"iptables-alerter-vdwvw\" (UID: \"e7674168-32c3-4c84-a753-2af1455d5c30\") " pod="openshift-network-operator/iptables-alerter-vdwvw"
Jul 07 16:46:49.609890 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.609859    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpmb4\" (UniqueName: \"kubernetes.io/projected/e7674168-32c3-4c84-a753-2af1455d5c30-kube-api-access-vpmb4\") pod \"iptables-alerter-vdwvw\" (UID: \"e7674168-32c3-4c84-a753-2af1455d5c30\") " pod="openshift-network-operator/iptables-alerter-vdwvw"
Jul 07 16:46:49.610672 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.610641    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4s2s\" (UniqueName: \"kubernetes.io/projected/779e8dac-267c-4848-ac3f-4e8e0c9a0a0a-kube-api-access-s4s2s\") pod \"aws-ebs-csi-driver-node-zdmv9\" (UID: \"779e8dac-267c-4848-ac3f-4e8e0c9a0a0a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9"
Jul 07 16:46:49.610766 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.610734    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4xtc\" (UniqueName: \"kubernetes.io/projected/f403e1aa-f12e-4df4-9626-79a45ab81549-kube-api-access-j4xtc\") pod \"multus-gsc2d\" (UID: \"f403e1aa-f12e-4df4-9626-79a45ab81549\") " pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:49.612092 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.612067    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tvfm\" (UniqueName: \"kubernetes.io/projected/99e5a676-4f78-4532-a5a4-87f1e55d2cba-kube-api-access-7tvfm\") pod \"multus-additional-cni-plugins-gbzdh\" (UID: \"99e5a676-4f78-4532-a5a4-87f1e55d2cba\") " pod="openshift-multus/multus-additional-cni-plugins-gbzdh"
Jul 07 16:46:49.689851 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.689817    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xs8l4"
Jul 07 16:46:49.697001 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.696972    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mj2p8"
Jul 07 16:46:49.707571 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.707540    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:46:49.712473 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.712448    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p6nfc"
Jul 07 16:46:49.720242 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.720206    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kprb9"
Jul 07 16:46:49.728152 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.728115    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9"
Jul 07 16:46:49.737091 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.736923    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vdwvw"
Jul 07 16:46:49.743983 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.743961    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gbzdh"
Jul 07 16:46:49.749866 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:49.749844    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gsc2d"
Jul 07 16:46:50.003028 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:50.002993    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-original-pull-secret\") pod \"global-pull-secret-syncer-x5chr\" (UID: \"41c32952-21a9-4e19-bc8a-ceee6a08ceeb\") " pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:46:50.003195 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:50.003045    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b399a96b-729b-484d-8edf-e97618163cfa-metrics-certs\") pod \"network-metrics-daemon-vv52w\" (UID: \"b399a96b-729b-484d-8edf-e97618163cfa\") " pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:46:50.003195 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:50.003117    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26r9t\" (UniqueName: \"kubernetes.io/projected/9fc991d0-7b29-41d9-bf1a-7ba28833be28-kube-api-access-26r9t\") pod \"network-check-target-l7frr\" (UID: \"9fc991d0-7b29-41d9-bf1a-7ba28833be28\") " pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:46:50.003195 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:50.003129    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:46:50.003327 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:50.003204    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b399a96b-729b-484d-8edf-e97618163cfa-metrics-certs podName:b399a96b-729b-484d-8edf-e97618163cfa nodeName:}" failed. No retries permitted until 2026-07-07 16:46:51.003185767 +0000 UTC m=+4.016195581 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b399a96b-729b-484d-8edf-e97618163cfa-metrics-certs") pod "network-metrics-daemon-vv52w" (UID: "b399a96b-729b-484d-8edf-e97618163cfa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:46:50.003327 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:50.003256    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:46:50.003327 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:50.003259    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:46:50.003327 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:50.003289    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:46:50.003327 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:50.003303    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-original-pull-secret podName:41c32952-21a9-4e19-bc8a-ceee6a08ceeb nodeName:}" failed. No retries permitted until 2026-07-07 16:46:51.003291554 +0000 UTC m=+4.016301355 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-original-pull-secret") pod "global-pull-secret-syncer-x5chr" (UID: "41c32952-21a9-4e19-bc8a-ceee6a08ceeb") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:46:50.003327 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:50.003304    2583 projected.go:196] Error preparing data for projected volume kube-api-access-26r9t for pod openshift-network-diagnostics/network-check-target-l7frr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:46:50.003543 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:50.003374    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9fc991d0-7b29-41d9-bf1a-7ba28833be28-kube-api-access-26r9t podName:9fc991d0-7b29-41d9-bf1a-7ba28833be28 nodeName:}" failed. No retries permitted until 2026-07-07 16:46:51.003355136 +0000 UTC m=+4.016364954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-26r9t" (UniqueName: "kubernetes.io/projected/9fc991d0-7b29-41d9-bf1a-7ba28833be28-kube-api-access-26r9t") pod "network-check-target-l7frr" (UID: "9fc991d0-7b29-41d9-bf1a-7ba28833be28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:46:50.437465 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:50.437422    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 16:41:49 +0000 UTC" deadline="2027-12-25 12:40:41.345265659 +0000 UTC"
Jul 07 16:46:50.437465 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:50.437463    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12859h53m50.907807224s"
Jul 07 16:46:50.618905 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:50.618874    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89989a39_fdfd_431e_b96d_ae32d7ee482c.slice/crio-320c8efb046694026c954dff4ce6f123b8b74c3e3044307394352be3e21bf294 WatchSource:0}: Error finding container 320c8efb046694026c954dff4ce6f123b8b74c3e3044307394352be3e21bf294: Status 404 returned error can't find the container with id 320c8efb046694026c954dff4ce6f123b8b74c3e3044307394352be3e21bf294
Jul 07 16:46:50.619730 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:50.619698    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e5a676_4f78_4532_a5a4_87f1e55d2cba.slice/crio-c8f7ca10369c41267145cafd914dc800ed4ffccab37bafd7b83a2c0d0395f90f WatchSource:0}: Error finding container c8f7ca10369c41267145cafd914dc800ed4ffccab37bafd7b83a2c0d0395f90f: Status 404 returned error can't find the container with id c8f7ca10369c41267145cafd914dc800ed4ffccab37bafd7b83a2c0d0395f90f
Jul 07 16:46:50.621310 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:50.621283    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7674168_32c3_4c84_a753_2af1455d5c30.slice/crio-ac135b47de2685cc58ab4d558e35b47c63aad4eed74d77ad443ec1168c91457b WatchSource:0}: Error finding container ac135b47de2685cc58ab4d558e35b47c63aad4eed74d77ad443ec1168c91457b: Status 404 returned error can't find the container with id ac135b47de2685cc58ab4d558e35b47c63aad4eed74d77ad443ec1168c91457b
Jul 07 16:46:50.627691 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:50.627664    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf403e1aa_f12e_4df4_9626_79a45ab81549.slice/crio-5661654027ad2f56bf43893a3ddc45fdfc36bfd616efea0892f2dec9dd486683 WatchSource:0}: Error finding container 5661654027ad2f56bf43893a3ddc45fdfc36bfd616efea0892f2dec9dd486683: Status 404 returned error can't find the container with id 5661654027ad2f56bf43893a3ddc45fdfc36bfd616efea0892f2dec9dd486683
Jul 07 16:46:50.631231 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:50.631206    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20811d55_4cd6_4c95_9e50_e53a2b3bf2c7.slice/crio-052acbf5cc3708d53cf055f3aff88debb536244b04b97408cc2308ca27731885 WatchSource:0}: Error finding container 052acbf5cc3708d53cf055f3aff88debb536244b04b97408cc2308ca27731885: Status 404 returned error can't find the container with id 052acbf5cc3708d53cf055f3aff88debb536244b04b97408cc2308ca27731885
Jul 07 16:46:50.633265 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:50.633164    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6f29a9_b432_4a71_a94a_27710a455ea6.slice/crio-8a98633976998b63994d94b59df7c7677e20cd980d8fba1883b740cfac8dece8 WatchSource:0}: Error finding container 8a98633976998b63994d94b59df7c7677e20cd980d8fba1883b740cfac8dece8: Status 404 returned error can't find the container with id 8a98633976998b63994d94b59df7c7677e20cd980d8fba1883b740cfac8dece8
Jul 07 16:46:50.634156 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:50.634116    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7530471_b4ef_492d_b1bd_6393da0da34b.slice/crio-04aae6368008211354c10d2e5c799a83ea32f6e8521db0389d93565f0ab4bdff WatchSource:0}: Error finding container 04aae6368008211354c10d2e5c799a83ea32f6e8521db0389d93565f0ab4bdff: Status 404 returned error can't find the container with id 04aae6368008211354c10d2e5c799a83ea32f6e8521db0389d93565f0ab4bdff
Jul 07 16:46:50.636047 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:50.636001    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod779e8dac_267c_4848_ac3f_4e8e0c9a0a0a.slice/crio-2c48bdbe34c3dca793cb32a9c858784fd04325307920d42cc903328fce0bf405 WatchSource:0}: Error finding container 2c48bdbe34c3dca793cb32a9c858784fd04325307920d42cc903328fce0bf405: Status 404 returned error can't find the container with id 2c48bdbe34c3dca793cb32a9c858784fd04325307920d42cc903328fce0bf405
Jul 07 16:46:50.636995 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:46:50.636936    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4541624c_a787_41a9_9d15_7b26c2aff7ec.slice/crio-5ab2298d0551666bc8ec90525f78536908682da2262066ccb7bdc73277f1e388 WatchSource:0}: Error finding container 5ab2298d0551666bc8ec90525f78536908682da2262066ccb7bdc73277f1e388: Status 404 returned error can't find the container with id 5ab2298d0551666bc8ec90525f78536908682da2262066ccb7bdc73277f1e388
Jul 07 16:46:51.010588 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:51.010281    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-original-pull-secret\") pod \"global-pull-secret-syncer-x5chr\" (UID: \"41c32952-21a9-4e19-bc8a-ceee6a08ceeb\") " pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:46:51.010747 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:51.010437    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:46:51.010747 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:51.010621    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b399a96b-729b-484d-8edf-e97618163cfa-metrics-certs\") pod \"network-metrics-daemon-vv52w\" (UID: \"b399a96b-729b-484d-8edf-e97618163cfa\") " pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:46:51.010747 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:51.010663    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26r9t\" (UniqueName: \"kubernetes.io/projected/9fc991d0-7b29-41d9-bf1a-7ba28833be28-kube-api-access-26r9t\") pod \"network-check-target-l7frr\" (UID: \"9fc991d0-7b29-41d9-bf1a-7ba28833be28\") " pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:46:51.010747 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:51.010682    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-original-pull-secret podName:41c32952-21a9-4e19-bc8a-ceee6a08ceeb nodeName:}" failed. No retries permitted until 2026-07-07 16:46:53.010658935 +0000 UTC m=+6.023668751 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-original-pull-secret") pod "global-pull-secret-syncer-x5chr" (UID: "41c32952-21a9-4e19-bc8a-ceee6a08ceeb") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:46:51.010958 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:51.010762    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:46:51.010958 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:51.010820    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b399a96b-729b-484d-8edf-e97618163cfa-metrics-certs podName:b399a96b-729b-484d-8edf-e97618163cfa nodeName:}" failed. No retries permitted until 2026-07-07 16:46:53.010803805 +0000 UTC m=+6.023813616 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b399a96b-729b-484d-8edf-e97618163cfa-metrics-certs") pod "network-metrics-daemon-vv52w" (UID: "b399a96b-729b-484d-8edf-e97618163cfa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:46:51.010958 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:51.010764    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:46:51.010958 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:51.010854    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:46:51.010958 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:51.010868    2583 projected.go:196] Error preparing data for projected volume kube-api-access-26r9t for pod openshift-network-diagnostics/network-check-target-l7frr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:46:51.010958 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:51.010908    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9fc991d0-7b29-41d9-bf1a-7ba28833be28-kube-api-access-26r9t podName:9fc991d0-7b29-41d9-bf1a-7ba28833be28 nodeName:}" failed. No retries permitted until 2026-07-07 16:46:53.010896228 +0000 UTC m=+6.023906044 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-26r9t" (UniqueName: "kubernetes.io/projected/9fc991d0-7b29-41d9-bf1a-7ba28833be28-kube-api-access-26r9t") pod "network-check-target-l7frr" (UID: "9fc991d0-7b29-41d9-bf1a-7ba28833be28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:46:51.437803 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:51.437731    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 16:41:49 +0000 UTC" deadline="2028-02-13 14:31:08.224419305 +0000 UTC"
Jul 07 16:46:51.437803 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:51.437795    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14061h44m16.786628s"
Jul 07 16:46:51.582520 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:51.581677    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:46:51.582520 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:51.581806    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:46:51.582520 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:51.582232    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:46:51.582520 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:51.582330    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:46:51.582520 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:51.582403    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:46:51.582520 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:51.582479    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:46:51.603830 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:51.603790    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9" event={"ID":"779e8dac-267c-4848-ac3f-4e8e0c9a0a0a","Type":"ContainerStarted","Data":"2c48bdbe34c3dca793cb32a9c858784fd04325307920d42cc903328fce0bf405"}
Jul 07 16:46:51.625043 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:51.625003    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kprb9" event={"ID":"ae6f29a9-b432-4a71-a94a-27710a455ea6","Type":"ContainerStarted","Data":"8a98633976998b63994d94b59df7c7677e20cd980d8fba1883b740cfac8dece8"}
Jul 07 16:46:51.627916 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:51.627803    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gsc2d" event={"ID":"f403e1aa-f12e-4df4-9626-79a45ab81549","Type":"ContainerStarted","Data":"5661654027ad2f56bf43893a3ddc45fdfc36bfd616efea0892f2dec9dd486683"}
Jul 07 16:46:51.654613 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:51.654447    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gbzdh" event={"ID":"99e5a676-4f78-4532-a5a4-87f1e55d2cba","Type":"ContainerStarted","Data":"c8f7ca10369c41267145cafd914dc800ed4ffccab37bafd7b83a2c0d0395f90f"}
Jul 07 16:46:51.677177 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:51.674348    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xs8l4" event={"ID":"89989a39-fdfd-431e-b96d-ae32d7ee482c","Type":"ContainerStarted","Data":"320c8efb046694026c954dff4ce6f123b8b74c3e3044307394352be3e21bf294"}
Jul 07 16:46:51.681670 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:51.680111    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vdwvw" event={"ID":"e7674168-32c3-4c84-a753-2af1455d5c30","Type":"ContainerStarted","Data":"ac135b47de2685cc58ab4d558e35b47c63aad4eed74d77ad443ec1168c91457b"}
Jul 07 16:46:51.690230 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:51.689702    2583 generic.go:356] "Generic (PLEG): container finished" podID="89c5295fbef6e1eabe85a83f67a07bbd" containerID="1bd9ac521228b0af3929aa3fa995c8af3f37641a70f83fed87fe4684856dfad2" exitCode=0
Jul 07 16:46:51.690230 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:51.689782    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.ec2.internal" event={"ID":"89c5295fbef6e1eabe85a83f67a07bbd","Type":"ContainerDied","Data":"1bd9ac521228b0af3929aa3fa995c8af3f37641a70f83fed87fe4684856dfad2"}
Jul 07 16:46:51.702610 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:51.702554    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-189.ec2.internal" event={"ID":"e3a5ffb0c2c44b15a7860c5703342945","Type":"ContainerStarted","Data":"a14db345df896e53507a3511157694b717c686dbb5943d6bcc222727f09a2559"}
Jul 07 16:46:51.715809 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:51.715416    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgf45" event={"ID":"4541624c-a787-41a9-9d15-7b26c2aff7ec","Type":"ContainerStarted","Data":"5ab2298d0551666bc8ec90525f78536908682da2262066ccb7bdc73277f1e388"}
Jul 07 16:46:51.732477 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:51.732397    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mj2p8" event={"ID":"20811d55-4cd6-4c95-9e50-e53a2b3bf2c7","Type":"ContainerStarted","Data":"052acbf5cc3708d53cf055f3aff88debb536244b04b97408cc2308ca27731885"}
Jul 07 16:46:51.748700 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:51.748656    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p6nfc" event={"ID":"f7530471-b4ef-492d-b1bd-6393da0da34b","Type":"ContainerStarted","Data":"04aae6368008211354c10d2e5c799a83ea32f6e8521db0389d93565f0ab4bdff"}
Jul 07 16:46:52.765568 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:52.765368    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.ec2.internal" event={"ID":"89c5295fbef6e1eabe85a83f67a07bbd","Type":"ContainerStarted","Data":"c02d33ec3d15ac0311424338ea8be118d0e935c8b155d13979c60ecdd7aebdf9"}
Jul 07 16:46:52.795175 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:52.795093    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-189.ec2.internal" podStartSLOduration=4.795072231 podStartE2EDuration="4.795072231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:46:51.732439445 +0000 UTC m=+4.745449266" watchObservedRunningTime="2026-07-07 16:46:52.795072231 +0000 UTC m=+5.808082051"
Jul 07 16:46:53.028638 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:53.028537    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-original-pull-secret\") pod \"global-pull-secret-syncer-x5chr\" (UID: \"41c32952-21a9-4e19-bc8a-ceee6a08ceeb\") " pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:46:53.028638 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:53.028618    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b399a96b-729b-484d-8edf-e97618163cfa-metrics-certs\") pod \"network-metrics-daemon-vv52w\" (UID: \"b399a96b-729b-484d-8edf-e97618163cfa\") " pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:46:53.028907 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:53.028645    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26r9t\" (UniqueName: \"kubernetes.io/projected/9fc991d0-7b29-41d9-bf1a-7ba28833be28-kube-api-access-26r9t\") pod \"network-check-target-l7frr\" (UID: \"9fc991d0-7b29-41d9-bf1a-7ba28833be28\") " pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:46:53.028907 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:53.028789    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:46:53.028907 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:53.028812    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:46:53.028907 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:53.028825    2583 projected.go:196] Error preparing data for projected volume kube-api-access-26r9t for pod openshift-network-diagnostics/network-check-target-l7frr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:46:53.028907 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:53.028882    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9fc991d0-7b29-41d9-bf1a-7ba28833be28-kube-api-access-26r9t podName:9fc991d0-7b29-41d9-bf1a-7ba28833be28 nodeName:}" failed. No retries permitted until 2026-07-07 16:46:57.028864966 +0000 UTC m=+10.041874777 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-26r9t" (UniqueName: "kubernetes.io/projected/9fc991d0-7b29-41d9-bf1a-7ba28833be28-kube-api-access-26r9t") pod "network-check-target-l7frr" (UID: "9fc991d0-7b29-41d9-bf1a-7ba28833be28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:46:53.029314 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:53.028955    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:46:53.029314 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:53.028994    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-original-pull-secret podName:41c32952-21a9-4e19-bc8a-ceee6a08ceeb nodeName:}" failed. No retries permitted until 2026-07-07 16:46:57.028981756 +0000 UTC m=+10.041991611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-original-pull-secret") pod "global-pull-secret-syncer-x5chr" (UID: "41c32952-21a9-4e19-bc8a-ceee6a08ceeb") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:46:53.029314 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:53.029050    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:46:53.029314 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:53.029084    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b399a96b-729b-484d-8edf-e97618163cfa-metrics-certs podName:b399a96b-729b-484d-8edf-e97618163cfa nodeName:}" failed. No retries permitted until 2026-07-07 16:46:57.029072227 +0000 UTC m=+10.042082027 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b399a96b-729b-484d-8edf-e97618163cfa-metrics-certs") pod "network-metrics-daemon-vv52w" (UID: "b399a96b-729b-484d-8edf-e97618163cfa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:46:53.583113 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:53.583080    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:46:53.583290 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:53.583263    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:46:53.583672 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:53.583652    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:46:53.583776 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:53.583747    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:46:53.583829 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:53.583655    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:46:53.583922 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:53.583897    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:46:55.582896 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:55.582240    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:46:55.582896 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:55.582372    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:46:55.582896 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:55.582750    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:46:55.582896 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:55.582842    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:46:55.585780 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:55.583613    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:46:55.585780 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:55.583731    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:46:57.060609 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:57.060456    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-original-pull-secret\") pod \"global-pull-secret-syncer-x5chr\" (UID: \"41c32952-21a9-4e19-bc8a-ceee6a08ceeb\") " pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:46:57.061037 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:57.060782    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b399a96b-729b-484d-8edf-e97618163cfa-metrics-certs\") pod \"network-metrics-daemon-vv52w\" (UID: \"b399a96b-729b-484d-8edf-e97618163cfa\") " pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:46:57.061037 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:57.060828    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26r9t\" (UniqueName: \"kubernetes.io/projected/9fc991d0-7b29-41d9-bf1a-7ba28833be28-kube-api-access-26r9t\") pod \"network-check-target-l7frr\" (UID: \"9fc991d0-7b29-41d9-bf1a-7ba28833be28\") " pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:46:57.061121 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:57.061049    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:46:57.061121 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:57.061071    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:46:57.061121 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:57.061084    2583 projected.go:196] Error preparing data for projected volume kube-api-access-26r9t for pod openshift-network-diagnostics/network-check-target-l7frr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:46:57.061243 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:57.061180    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9fc991d0-7b29-41d9-bf1a-7ba28833be28-kube-api-access-26r9t podName:9fc991d0-7b29-41d9-bf1a-7ba28833be28 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:05.061159661 +0000 UTC m=+18.074169467 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-26r9t" (UniqueName: "kubernetes.io/projected/9fc991d0-7b29-41d9-bf1a-7ba28833be28-kube-api-access-26r9t") pod "network-check-target-l7frr" (UID: "9fc991d0-7b29-41d9-bf1a-7ba28833be28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:46:57.061677 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:57.061659    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:46:57.061738 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:57.061722    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-original-pull-secret podName:41c32952-21a9-4e19-bc8a-ceee6a08ceeb nodeName:}" failed. No retries permitted until 2026-07-07 16:47:05.061700937 +0000 UTC m=+18.074710738 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-original-pull-secret") pod "global-pull-secret-syncer-x5chr" (UID: "41c32952-21a9-4e19-bc8a-ceee6a08ceeb") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:46:57.061816 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:57.061797    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:46:57.061869 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:57.061840    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b399a96b-729b-484d-8edf-e97618163cfa-metrics-certs podName:b399a96b-729b-484d-8edf-e97618163cfa nodeName:}" failed. No retries permitted until 2026-07-07 16:47:05.061828794 +0000 UTC m=+18.074838593 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b399a96b-729b-484d-8edf-e97618163cfa-metrics-certs") pod "network-metrics-daemon-vv52w" (UID: "b399a96b-729b-484d-8edf-e97618163cfa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:46:57.582659 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:57.582185    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:46:57.582659 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:57.582299    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:46:57.582659 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:57.582307    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:46:57.582659 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:57.582338    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:46:57.582659 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:57.582477    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:46:57.582659 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:57.582578    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:46:59.583218 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:59.583188    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:46:59.583672 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:59.583320    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:46:59.583754 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:59.583715    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:46:59.583995 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:59.583825    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:46:59.583995 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:46:59.583884    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:46:59.583995 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:46:59.583948    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:47:01.582037 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:01.581563    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:01.582037 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:01.581563    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:01.582037 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:01.581696    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:47:01.582037 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:01.581578    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:01.582037 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:01.581796    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:47:01.582037 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:01.581908    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:47:03.581569 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:03.581537    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:03.582054 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:03.581650    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:03.582054 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:03.581665    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:47:03.582054 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:03.581535    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:03.582054 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:03.581732    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:47:03.582054 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:03.581801    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:47:05.127076 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:05.125963    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b399a96b-729b-484d-8edf-e97618163cfa-metrics-certs\") pod \"network-metrics-daemon-vv52w\" (UID: \"b399a96b-729b-484d-8edf-e97618163cfa\") " pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:05.127076 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:05.126241    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26r9t\" (UniqueName: \"kubernetes.io/projected/9fc991d0-7b29-41d9-bf1a-7ba28833be28-kube-api-access-26r9t\") pod \"network-check-target-l7frr\" (UID: \"9fc991d0-7b29-41d9-bf1a-7ba28833be28\") " pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:05.127076 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:05.126292    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-original-pull-secret\") pod \"global-pull-secret-syncer-x5chr\" (UID: \"41c32952-21a9-4e19-bc8a-ceee6a08ceeb\") " pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:05.127076 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:05.126425    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:47:05.127076 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:05.126481    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-original-pull-secret podName:41c32952-21a9-4e19-bc8a-ceee6a08ceeb nodeName:}" failed. No retries permitted until 2026-07-07 16:47:21.126462643 +0000 UTC m=+34.139472460 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-original-pull-secret") pod "global-pull-secret-syncer-x5chr" (UID: "41c32952-21a9-4e19-bc8a-ceee6a08ceeb") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:47:05.127076 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:05.126546    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:47:05.127076 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:05.126580    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b399a96b-729b-484d-8edf-e97618163cfa-metrics-certs podName:b399a96b-729b-484d-8edf-e97618163cfa nodeName:}" failed. No retries permitted until 2026-07-07 16:47:21.126567563 +0000 UTC m=+34.139577363 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b399a96b-729b-484d-8edf-e97618163cfa-metrics-certs") pod "network-metrics-daemon-vv52w" (UID: "b399a96b-729b-484d-8edf-e97618163cfa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:47:05.127076 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:05.126655    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:47:05.127076 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:05.126670    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:47:05.127076 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:05.126695    2583 projected.go:196] Error preparing data for projected volume kube-api-access-26r9t for pod openshift-network-diagnostics/network-check-target-l7frr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:47:05.127076 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:05.126730    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9fc991d0-7b29-41d9-bf1a-7ba28833be28-kube-api-access-26r9t podName:9fc991d0-7b29-41d9-bf1a-7ba28833be28 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:21.126718617 +0000 UTC m=+34.139728428 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-26r9t" (UniqueName: "kubernetes.io/projected/9fc991d0-7b29-41d9-bf1a-7ba28833be28-kube-api-access-26r9t") pod "network-check-target-l7frr" (UID: "9fc991d0-7b29-41d9-bf1a-7ba28833be28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:47:05.581656 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:05.581621    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:05.581820 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:05.581621    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:05.581820 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:05.581730    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:05.581820 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:05.581749    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:47:05.581960 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:05.581842    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:47:05.581960 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:05.581921    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:47:05.791744 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:05.791649    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xs8l4" event={"ID":"89989a39-fdfd-431e-b96d-ae32d7ee482c","Type":"ContainerStarted","Data":"33df92889a673684a62f3a8ad1440e2b22af92c65337458a9a37e647cd3023b2"}
Jul 07 16:47:05.793225 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:05.793195    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mj2p8" event={"ID":"20811d55-4cd6-4c95-9e50-e53a2b3bf2c7","Type":"ContainerStarted","Data":"46ffddaef562fc11c2f6e65c66b8f5b480e3fcadc9c715a31765c4d1812fa411"}
Jul 07 16:47:05.794858 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:05.794832    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p6nfc" event={"ID":"f7530471-b4ef-492d-b1bd-6393da0da34b","Type":"ContainerStarted","Data":"63e1adcac1d19bba0f39efc9884196cb1099da242f021366ce4cb4c6f5db4e10"}
Jul 07 16:47:05.796454 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:05.796417    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9" event={"ID":"779e8dac-267c-4848-ac3f-4e8e0c9a0a0a","Type":"ContainerStarted","Data":"444d25f9204d3e98c74a84e1e48e6861871e78c32000741d62d7eba85e35d3e8"}
Jul 07 16:47:05.797887 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:05.797850    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kprb9" event={"ID":"ae6f29a9-b432-4a71-a94a-27710a455ea6","Type":"ContainerStarted","Data":"7966a1aae29e536701b28a4012f967f8415b6b7ea831f160111e00732bfb9e31"}
Jul 07 16:47:05.799469 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:05.799447    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gsc2d" event={"ID":"f403e1aa-f12e-4df4-9626-79a45ab81549","Type":"ContainerStarted","Data":"dd7c988de9ab6bf7566a1fb913663ca9d9d654d1462599700a21c245f2924b2e"}
Jul 07 16:47:05.801642 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:05.801617    2583 generic.go:356] "Generic (PLEG): container finished" podID="99e5a676-4f78-4532-a5a4-87f1e55d2cba" containerID="8b5f9347661eee0aeb9f423101922f149013a0b44d08686240cf515537e1cf0f" exitCode=0
Jul 07 16:47:05.801748 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:05.801650    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gbzdh" event={"ID":"99e5a676-4f78-4532-a5a4-87f1e55d2cba","Type":"ContainerDied","Data":"8b5f9347661eee0aeb9f423101922f149013a0b44d08686240cf515537e1cf0f"}
Jul 07 16:47:05.818447 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:05.818392    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-189.ec2.internal" podStartSLOduration=17.818373425 podStartE2EDuration="17.818373425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:46:52.796379645 +0000 UTC m=+5.809389464" watchObservedRunningTime="2026-07-07 16:47:05.818373425 +0000 UTC m=+18.831383245"
Jul 07 16:47:05.818954 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:05.818926    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xs8l4" podStartSLOduration=4.531252556 podStartE2EDuration="18.818914353s" totalImagesPullingTime="14.287661797s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:05.817963295 +0000 UTC m=+18.830973115" watchObservedRunningTime="2026-07-07 16:47:05.818914353 +0000 UTC m=+18.831924177"
Jul 07 16:47:05.876215 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:05.876157    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kprb9" podStartSLOduration=4.661820559 podStartE2EDuration="18.876125105s" totalImagesPullingTime="14.214304546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:05.845500216 +0000 UTC m=+18.858510037" watchObservedRunningTime="2026-07-07 16:47:05.876125105 +0000 UTC m=+18.889134920"
Jul 07 16:47:05.900806 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:05.900752    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gsc2d" podStartSLOduration=4.602987105 podStartE2EDuration="18.900729889s" totalImagesPullingTime="14.297742784s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:05.900185484 +0000 UTC m=+18.913195305" watchObservedRunningTime="2026-07-07 16:47:05.900729889 +0000 UTC m=+18.913739708"
Jul 07 16:47:05.918459 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:05.918405    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mj2p8" podStartSLOduration=4.662738285 podStartE2EDuration="18.918389383s" totalImagesPullingTime="14.255651098s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:05.918372289 +0000 UTC m=+18.931382110" watchObservedRunningTime="2026-07-07 16:47:05.918389383 +0000 UTC m=+18.931399201"
Jul 07 16:47:06.805919 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:06.805876    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vdwvw" event={"ID":"e7674168-32c3-4c84-a753-2af1455d5c30","Type":"ContainerStarted","Data":"ddfd778682eed02fec88e9f67a3e5c670d260a5a081128676fd9fb685fe11b0c"}
Jul 07 16:47:06.830291 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:06.830177    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-p6nfc" podStartSLOduration=5.557355642 podStartE2EDuration="19.83015507s" totalImagesPullingTime="14.272799428s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:05.936902456 +0000 UTC m=+18.949912276" watchObservedRunningTime="2026-07-07 16:47:06.83015507 +0000 UTC m=+19.843164884"
Jul 07 16:47:06.830533 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:06.830507    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vdwvw" podStartSLOduration=5.543693847 podStartE2EDuration="19.830499881s" totalImagesPullingTime="14.286806034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:06.8299997 +0000 UTC m=+19.843009530" watchObservedRunningTime="2026-07-07 16:47:06.830499881 +0000 UTC m=+19.843509700"
Jul 07 16:47:07.036126 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:07.036092    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kprb9"
Jul 07 16:47:07.036799 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:07.036774    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kprb9"
Jul 07 16:47:07.582536 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:07.582497    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:07.582709 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:07.582583    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:07.582709 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:07.582600    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:47:07.582804 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:07.582701    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:47:07.582804 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:07.582767    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:07.582892 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:07.582846    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:47:07.808787 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:07.808753    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kprb9"
Jul 07 16:47:07.809207 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:07.808893    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kprb9"
Jul 07 16:47:09.581154 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:09.581097    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:09.581886 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:09.581101    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:09.581886 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:09.581253    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:47:09.581886 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:09.581265    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:09.581886 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:09.581394    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:47:09.581886 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:09.581467    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:47:10.653634 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:10.653341    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 07 16:47:10.816721 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:10.816681    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9" event={"ID":"779e8dac-267c-4848-ac3f-4e8e0c9a0a0a","Type":"ContainerStarted","Data":"f5e3307687163d8d2ed39aaa132c9914d80463d873f846ee4f037b08fdf6534f"}
Jul 07 16:47:10.820167 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:10.820123    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgf45" event={"ID":"4541624c-a787-41a9-9d15-7b26c2aff7ec","Type":"ContainerStarted","Data":"599088a9d88035da4f99f4c3bdcf1faaee7445bd33722a8c862373729e0c752c"}
Jul 07 16:47:10.820306 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:10.820203    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgf45" event={"ID":"4541624c-a787-41a9-9d15-7b26c2aff7ec","Type":"ContainerStarted","Data":"3b3224480205cc96d4ff861d490cc035529e1cb3117d8383fbaddf3725331d81"}
Jul 07 16:47:10.820306 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:10.820242    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgf45" event={"ID":"4541624c-a787-41a9-9d15-7b26c2aff7ec","Type":"ContainerStarted","Data":"b98677a23b8929ca285675e54b1b2018a1d6ec6c7aec74deb8e284c94a8baa38"}
Jul 07 16:47:10.820306 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:10.820276    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgf45" event={"ID":"4541624c-a787-41a9-9d15-7b26c2aff7ec","Type":"ContainerStarted","Data":"06f9e3526ac5580c15af06845515a8ac20e135dd1ad7a5dca6806dcd10d8d2c8"}
Jul 07 16:47:10.820465 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:10.820325    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgf45" event={"ID":"4541624c-a787-41a9-9d15-7b26c2aff7ec","Type":"ContainerStarted","Data":"01c967be47e4e2cbe9289b45523131ece10e65f84d493e7f80cfbf51d377c557"}
Jul 07 16:47:10.820465 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:10.820381    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgf45" event={"ID":"4541624c-a787-41a9-9d15-7b26c2aff7ec","Type":"ContainerStarted","Data":"f7f8a7472dac1645e719694333fb070bd06284e0b41df82283441570afd43637"}
Jul 07 16:47:11.457895 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:11.457633    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T16:47:10.653612924Z","UUID":"5aa77cc9-6950-4828-bd2f-00344493b94a","Handler":null,"Name":"","Endpoint":""}
Jul 07 16:47:11.459677 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:11.459647    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 07 16:47:11.459875 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:11.459695    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 07 16:47:11.581349 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:11.581320    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:11.581510 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:11.581322    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:11.581588 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:11.581445    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:47:11.581588 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:11.581574    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:47:11.581696 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:11.581601    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:11.581757 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:11.581721    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:47:12.828087 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:12.827998    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgf45" event={"ID":"4541624c-a787-41a9-9d15-7b26c2aff7ec","Type":"ContainerStarted","Data":"4c3bcc6a535c38b67411fb60e0fa19283be984822ae98e3db43d1621a326cb0b"}
Jul 07 16:47:12.830008 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:12.829979    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9" event={"ID":"779e8dac-267c-4848-ac3f-4e8e0c9a0a0a","Type":"ContainerStarted","Data":"d70c4168dd28d97be7b0c4a10c8a69f2f8b87cafbe74645878cf3016521a116b"}
Jul 07 16:47:12.851031 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:12.850979    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdmv9" podStartSLOduration=5.015205938 podStartE2EDuration="25.850965075s" totalImagesPullingTime="20.835759137s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:46:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:12.85095005 +0000 UTC m=+25.863959870" watchObservedRunningTime="2026-07-07 16:47:12.850965075 +0000 UTC m=+25.863974895"
Jul 07 16:47:13.581429 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:13.581387    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:13.581599 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:13.581433    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:13.581599 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:13.581530    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:47:13.581691 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:13.581648    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:47:13.581726 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:13.581692    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:13.581758 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:13.581746    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:47:15.581509 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:15.581221    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:15.581509 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:15.581230    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:15.582205 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:15.581513    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:47:15.582205 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:15.581289    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:15.582205 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:15.581587    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:47:15.582205 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:15.581679    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:47:15.840257 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:15.840160    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgf45" event={"ID":"4541624c-a787-41a9-9d15-7b26c2aff7ec","Type":"ContainerStarted","Data":"57fded17dad1ff3bc75e563e741bac35576dc561464e30989bb7a31e10af1b2d"}
Jul 07 16:47:15.840463 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:15.840402    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:47:15.840463 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:15.840454    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:47:15.842154 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:15.842114    2583 generic.go:356] "Generic (PLEG): container finished" podID="99e5a676-4f78-4532-a5a4-87f1e55d2cba" containerID="5f28d518273f028c3172dd845ec755dcc0b5aa5910e0e6de077d56acd01a2f85" exitCode=0
Jul 07 16:47:15.842271 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:15.842170    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gbzdh" event={"ID":"99e5a676-4f78-4532-a5a4-87f1e55d2cba","Type":"ContainerDied","Data":"5f28d518273f028c3172dd845ec755dcc0b5aa5910e0e6de077d56acd01a2f85"}
Jul 07 16:47:15.858924 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:15.858897    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:47:15.893661 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:15.893602    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mgf45" podStartSLOduration=9.717080749 podStartE2EDuration="28.893583711s" totalImagesPullingTime="19.176502962s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:15.893330877 +0000 UTC m=+28.906340696" watchObservedRunningTime="2026-07-07 16:47:15.893583711 +0000 UTC m=+28.906593530"
Jul 07 16:47:16.545566 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:16.545513    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p6nfc_f7530471-b4ef-492d-b1bd-6393da0da34b/dns-node-resolver/0.log"
Jul 07 16:47:16.846841 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:16.846749    2583 generic.go:356] "Generic (PLEG): container finished" podID="99e5a676-4f78-4532-a5a4-87f1e55d2cba" containerID="2f7f8676c20288e201e258fe515c59dd9d47c7c3a4f75c3cf18ef3edcf63afb0" exitCode=0
Jul 07 16:47:16.847276 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:16.846839    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gbzdh" event={"ID":"99e5a676-4f78-4532-a5a4-87f1e55d2cba","Type":"ContainerDied","Data":"2f7f8676c20288e201e258fe515c59dd9d47c7c3a4f75c3cf18ef3edcf63afb0"}
Jul 07 16:47:16.847345 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:16.847295    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:47:16.863679 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:16.863648    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:47:17.512315 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:17.512292    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mj2p8_20811d55-4cd6-4c95-9e50-e53a2b3bf2c7/node-ca/0.log"
Jul 07 16:47:17.585235 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:17.585052    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:17.585349 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:17.585105    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:17.585349 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:17.585316    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:47:17.585431 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:17.585105    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:17.585476 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:17.585433    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:47:17.585476 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:17.585467    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:47:17.591045 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:17.591024    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l7frr"]
Jul 07 16:47:17.602117 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:17.602092    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x5chr"]
Jul 07 16:47:17.611663 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:17.611637    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vv52w"]
Jul 07 16:47:17.851286 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:17.851257    2583 generic.go:356] "Generic (PLEG): container finished" podID="99e5a676-4f78-4532-a5a4-87f1e55d2cba" containerID="b4404be700db47a95de6aff1bb852c23bf9ffd3dc71d7a3b534859db6738a9d5" exitCode=0
Jul 07 16:47:17.851766 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:17.851367    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gbzdh" event={"ID":"99e5a676-4f78-4532-a5a4-87f1e55d2cba","Type":"ContainerDied","Data":"b4404be700db47a95de6aff1bb852c23bf9ffd3dc71d7a3b534859db6738a9d5"}
Jul 07 16:47:17.851766 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:17.851407    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:17.851766 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:17.851564    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:47:17.852607 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:17.851988    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:17.852607 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:17.852072    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:47:17.852607 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:17.852114    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:17.852607 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:17.852206    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:47:19.580979 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:19.580889    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:19.580979 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:19.580889    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:19.581511 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:19.580991    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:47:19.581511 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:19.581010    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:19.581511 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:19.581094    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:47:19.581511 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:19.581196    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:47:21.150821 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:21.150791    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-original-pull-secret\") pod \"global-pull-secret-syncer-x5chr\" (UID: \"41c32952-21a9-4e19-bc8a-ceee6a08ceeb\") " pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:21.151453 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:21.150857    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b399a96b-729b-484d-8edf-e97618163cfa-metrics-certs\") pod \"network-metrics-daemon-vv52w\" (UID: \"b399a96b-729b-484d-8edf-e97618163cfa\") " pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:21.151453 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:21.150883    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26r9t\" (UniqueName: \"kubernetes.io/projected/9fc991d0-7b29-41d9-bf1a-7ba28833be28-kube-api-access-26r9t\") pod \"network-check-target-l7frr\" (UID: \"9fc991d0-7b29-41d9-bf1a-7ba28833be28\") " pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:21.151453 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:21.150964    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:47:21.151453 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:21.151000    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:47:21.151453 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:21.151001    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:47:21.151453 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:21.151018    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:47:21.151453 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:21.151030    2583 projected.go:196] Error preparing data for projected volume kube-api-access-26r9t for pod openshift-network-diagnostics/network-check-target-l7frr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:47:21.151453 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:21.151032    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-original-pull-secret podName:41c32952-21a9-4e19-bc8a-ceee6a08ceeb nodeName:}" failed. No retries permitted until 2026-07-07 16:47:53.151012509 +0000 UTC m=+66.164022326 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-original-pull-secret") pod "global-pull-secret-syncer-x5chr" (UID: "41c32952-21a9-4e19-bc8a-ceee6a08ceeb") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:47:21.151453 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:21.151056    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b399a96b-729b-484d-8edf-e97618163cfa-metrics-certs podName:b399a96b-729b-484d-8edf-e97618163cfa nodeName:}" failed. No retries permitted until 2026-07-07 16:47:53.151040745 +0000 UTC m=+66.164050558 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b399a96b-729b-484d-8edf-e97618163cfa-metrics-certs") pod "network-metrics-daemon-vv52w" (UID: "b399a96b-729b-484d-8edf-e97618163cfa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:47:21.151453 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:21.151077    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9fc991d0-7b29-41d9-bf1a-7ba28833be28-kube-api-access-26r9t podName:9fc991d0-7b29-41d9-bf1a-7ba28833be28 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:53.151064892 +0000 UTC m=+66.164074694 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-26r9t" (UniqueName: "kubernetes.io/projected/9fc991d0-7b29-41d9-bf1a-7ba28833be28-kube-api-access-26r9t") pod "network-check-target-l7frr" (UID: "9fc991d0-7b29-41d9-bf1a-7ba28833be28") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:47:21.581269 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:21.581223    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:21.581488 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:21.581243    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:21.581488 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:21.581358    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:21.581488 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:21.581356    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:47:21.581488 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:21.581444    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:47:21.581638 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:21.581511    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:47:23.581708 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:23.581609    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:23.582300 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:23.581617    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:23.582300 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:23.581738    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:47:23.582300 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:23.581617    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:23.582300 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:23.581802    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:47:23.582300 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:23.581868    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:47:23.867979 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:23.867899    2583 generic.go:356] "Generic (PLEG): container finished" podID="99e5a676-4f78-4532-a5a4-87f1e55d2cba" containerID="bb5385f9cc63a997744b28c14359716aad747ff0ab9c3aef5bc9fb5fa5f29e05" exitCode=0
Jul 07 16:47:23.867979 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:23.867961    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gbzdh" event={"ID":"99e5a676-4f78-4532-a5a4-87f1e55d2cba","Type":"ContainerDied","Data":"bb5385f9cc63a997744b28c14359716aad747ff0ab9c3aef5bc9fb5fa5f29e05"}
Jul 07 16:47:24.873635 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:24.873599    2583 generic.go:356] "Generic (PLEG): container finished" podID="99e5a676-4f78-4532-a5a4-87f1e55d2cba" containerID="4add23bf0ca9aaedd3e93648b6bf5dbf833183d4ab951855294acd15235ff47d" exitCode=0
Jul 07 16:47:24.874025 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:24.873648    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gbzdh" event={"ID":"99e5a676-4f78-4532-a5a4-87f1e55d2cba","Type":"ContainerDied","Data":"4add23bf0ca9aaedd3e93648b6bf5dbf833183d4ab951855294acd15235ff47d"}
Jul 07 16:47:25.583947 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:25.583918    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:25.583947 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:25.583938    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:25.584209 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:25.584000    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:25.584209 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:25.584011    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:47:25.584209 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:25.584092    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:47:25.584209 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:25.584166    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:47:25.879470 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:25.879392    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gbzdh" event={"ID":"99e5a676-4f78-4532-a5a4-87f1e55d2cba","Type":"ContainerStarted","Data":"71100bc116735d4762834f5aca209103ac862abcb954a0d0121fa1dbaecb1a86"}
Jul 07 16:47:25.910771 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:25.910719    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gbzdh" podStartSLOduration=8.675964451 podStartE2EDuration="38.910701887s" totalImagesPullingTime="30.234737436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:46:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:25.909364465 +0000 UTC m=+38.922374319" watchObservedRunningTime="2026-07-07 16:47:25.910701887 +0000 UTC m=+38.923711705"
Jul 07 16:47:27.581723 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:27.581690    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:27.582123 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:27.581798    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:47:27.582123 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:27.581877    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:27.582123 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:27.581954    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:27.582123 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:27.581966    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:47:27.582123 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:27.582006    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:47:29.581210 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:29.581174    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:29.581635 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:29.581174    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:29.581635 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:29.581278    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:47:29.581635 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:29.581187    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:29.581635 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:29.581372    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:47:29.581635 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:29.581418    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:47:31.584625 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:31.584592    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:31.585030 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:31.584716    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:47:31.585030 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:31.584781    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:31.585030 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:31.584871    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:47:31.585030 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:31.584900    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:31.585030 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:31.584992    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:47:33.584380 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:33.584348    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:33.584757 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:33.584437    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:33.584757 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:33.584442    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:47:33.584757 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:33.584464    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:33.584757 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:33.584550    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:47:33.584757 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:33.584628    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:47:35.583991 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:35.583940    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:35.583991 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:35.583977    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:35.584456 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:35.584058    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:47:35.584456 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:35.584109    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:35.584456 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:35.584215    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:47:35.584456 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:35.584323    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:47:37.581590 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:37.581550    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:37.582035 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:37.581626    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:37.582035 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:37.581645    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:47:37.582035 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:37.581747    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:47:37.582035 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:37.581783    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:37.582035 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:37.581962    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:47:39.581731 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:39.581689    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:39.582304 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:39.581689    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:39.582304 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:39.581809    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:47:39.582304 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:39.581896    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:47:39.582304 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:39.581693    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:39.582304 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:39.581994    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:47:41.581789 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:41.581755    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:41.581789 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:41.581772    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:41.581789 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:41.581783    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:41.582391 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:41.581850    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l7frr" podUID="9fc991d0-7b29-41d9-bf1a-7ba28833be28"
Jul 07 16:47:41.582391 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:41.581901    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5chr" podUID="41c32952-21a9-4e19-bc8a-ceee6a08ceeb"
Jul 07 16:47:41.582391 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:41.581981    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vv52w" podUID="b399a96b-729b-484d-8edf-e97618163cfa"
Jul 07 16:47:42.775116 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.775089    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-189.ec2.internal" event="NodeReady"
Jul 07 16:47:42.775495 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.775236    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 07 16:47:42.826829 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.826801    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hjctb"]
Jul 07 16:47:42.828544 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.828521    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hjctb"
Jul 07 16:47:42.829655 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.829633    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-58fvv"]
Jul 07 16:47:42.831408 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.831388    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 07 16:47:42.831499 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.831442    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rzd72\""
Jul 07 16:47:42.831499 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.831480    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-58fvv"
Jul 07 16:47:42.831821 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.831799    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 07 16:47:42.831939 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.831922    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 07 16:47:42.834381 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.834361    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 07 16:47:42.834476 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.834445    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 07 16:47:42.834522 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.834483    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 07 16:47:42.834557 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.834546    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 07 16:47:42.834656 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.834635    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7gk9x\""
Jul 07 16:47:42.839216 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.839197    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hjctb"]
Jul 07 16:47:42.845705 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.845686    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-58fvv"]
Jul 07 16:47:42.902666 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.902578    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szt8k\" (UniqueName: \"kubernetes.io/projected/ad0977ee-f824-48f4-8f9f-e15ac2c968aa-kube-api-access-szt8k\") pod \"insights-runtime-extractor-58fvv\" (UID: \"ad0977ee-f824-48f4-8f9f-e15ac2c968aa\") " pod="openshift-insights/insights-runtime-extractor-58fvv"
Jul 07 16:47:42.902666 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.902611    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f753b07-d621-412c-8ffb-764e1a09ab1e-cert\") pod \"ingress-canary-hjctb\" (UID: \"9f753b07-d621-412c-8ffb-764e1a09ab1e\") " pod="openshift-ingress-canary/ingress-canary-hjctb"
Jul 07 16:47:42.902666 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.902633    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad0977ee-f824-48f4-8f9f-e15ac2c968aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-58fvv\" (UID: \"ad0977ee-f824-48f4-8f9f-e15ac2c968aa\") " pod="openshift-insights/insights-runtime-extractor-58fvv"
Jul 07 16:47:42.902666 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.902651    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad0977ee-f824-48f4-8f9f-e15ac2c968aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-58fvv\" (UID: \"ad0977ee-f824-48f4-8f9f-e15ac2c968aa\") " pod="openshift-insights/insights-runtime-extractor-58fvv"
Jul 07 16:47:42.902666 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.902671    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr4rg\" (UniqueName: \"kubernetes.io/projected/9f753b07-d621-412c-8ffb-764e1a09ab1e-kube-api-access-hr4rg\") pod \"ingress-canary-hjctb\" (UID: \"9f753b07-d621-412c-8ffb-764e1a09ab1e\") " pod="openshift-ingress-canary/ingress-canary-hjctb"
Jul 07 16:47:42.902941 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.902745    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad0977ee-f824-48f4-8f9f-e15ac2c968aa-data-volume\") pod \"insights-runtime-extractor-58fvv\" (UID: \"ad0977ee-f824-48f4-8f9f-e15ac2c968aa\") " pod="openshift-insights/insights-runtime-extractor-58fvv"
Jul 07 16:47:42.902941 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.902787    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ad0977ee-f824-48f4-8f9f-e15ac2c968aa-crio-socket\") pod \"insights-runtime-extractor-58fvv\" (UID: \"ad0977ee-f824-48f4-8f9f-e15ac2c968aa\") " pod="openshift-insights/insights-runtime-extractor-58fvv"
Jul 07 16:47:42.925211 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.925184    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-grhmq"]
Jul 07 16:47:42.927326 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.927311    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-grhmq"
Jul 07 16:47:42.931257 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.931230    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 07 16:47:42.931257 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.931244    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mtbc6\""
Jul 07 16:47:42.931411 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.931311    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 07 16:47:42.941070 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:42.941047    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-grhmq"]
Jul 07 16:47:43.004155 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.004092    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad0977ee-f824-48f4-8f9f-e15ac2c968aa-data-volume\") pod \"insights-runtime-extractor-58fvv\" (UID: \"ad0977ee-f824-48f4-8f9f-e15ac2c968aa\") " pod="openshift-insights/insights-runtime-extractor-58fvv"
Jul 07 16:47:43.004155 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.004158    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ad0977ee-f824-48f4-8f9f-e15ac2c968aa-crio-socket\") pod \"insights-runtime-extractor-58fvv\" (UID: \"ad0977ee-f824-48f4-8f9f-e15ac2c968aa\") " pod="openshift-insights/insights-runtime-extractor-58fvv"
Jul 07 16:47:43.004400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.004190    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szt8k\" (UniqueName: \"kubernetes.io/projected/ad0977ee-f824-48f4-8f9f-e15ac2c968aa-kube-api-access-szt8k\") pod \"insights-runtime-extractor-58fvv\" (UID: \"ad0977ee-f824-48f4-8f9f-e15ac2c968aa\") " pod="openshift-insights/insights-runtime-extractor-58fvv"
Jul 07 16:47:43.004400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.004210    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqctt\" (UniqueName: \"kubernetes.io/projected/7c0e2b23-0949-4d6e-bd5e-11d1aff95701-kube-api-access-cqctt\") pod \"dns-default-grhmq\" (UID: \"7c0e2b23-0949-4d6e-bd5e-11d1aff95701\") " pod="openshift-dns/dns-default-grhmq"
Jul 07 16:47:43.004400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.004229    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f753b07-d621-412c-8ffb-764e1a09ab1e-cert\") pod \"ingress-canary-hjctb\" (UID: \"9f753b07-d621-412c-8ffb-764e1a09ab1e\") " pod="openshift-ingress-canary/ingress-canary-hjctb"
Jul 07 16:47:43.004400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.004243    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c0e2b23-0949-4d6e-bd5e-11d1aff95701-config-volume\") pod \"dns-default-grhmq\" (UID: \"7c0e2b23-0949-4d6e-bd5e-11d1aff95701\") " pod="openshift-dns/dns-default-grhmq"
Jul 07 16:47:43.004400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.004260    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c0e2b23-0949-4d6e-bd5e-11d1aff95701-metrics-tls\") pod \"dns-default-grhmq\" (UID: \"7c0e2b23-0949-4d6e-bd5e-11d1aff95701\") " pod="openshift-dns/dns-default-grhmq"
Jul 07 16:47:43.004635 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.004423    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ad0977ee-f824-48f4-8f9f-e15ac2c968aa-crio-socket\") pod \"insights-runtime-extractor-58fvv\" (UID: \"ad0977ee-f824-48f4-8f9f-e15ac2c968aa\") " pod="openshift-insights/insights-runtime-extractor-58fvv"
Jul 07 16:47:43.004635 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.004439    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad0977ee-f824-48f4-8f9f-e15ac2c968aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-58fvv\" (UID: \"ad0977ee-f824-48f4-8f9f-e15ac2c968aa\") " pod="openshift-insights/insights-runtime-extractor-58fvv"
Jul 07 16:47:43.004635 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.004466    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad0977ee-f824-48f4-8f9f-e15ac2c968aa-data-volume\") pod \"insights-runtime-extractor-58fvv\" (UID: \"ad0977ee-f824-48f4-8f9f-e15ac2c968aa\") " pod="openshift-insights/insights-runtime-extractor-58fvv"
Jul 07 16:47:43.004635 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.004476    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad0977ee-f824-48f4-8f9f-e15ac2c968aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-58fvv\" (UID: \"ad0977ee-f824-48f4-8f9f-e15ac2c968aa\") " pod="openshift-insights/insights-runtime-extractor-58fvv"
Jul 07 16:47:43.004635 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.004513    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c0e2b23-0949-4d6e-bd5e-11d1aff95701-tmp-dir\") pod \"dns-default-grhmq\" (UID: \"7c0e2b23-0949-4d6e-bd5e-11d1aff95701\") " pod="openshift-dns/dns-default-grhmq"
Jul 07 16:47:43.004635 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.004597    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr4rg\" (UniqueName: \"kubernetes.io/projected/9f753b07-d621-412c-8ffb-764e1a09ab1e-kube-api-access-hr4rg\") pod \"ingress-canary-hjctb\" (UID: \"9f753b07-d621-412c-8ffb-764e1a09ab1e\") " pod="openshift-ingress-canary/ingress-canary-hjctb"
Jul 07 16:47:43.005018 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.004986    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad0977ee-f824-48f4-8f9f-e15ac2c968aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-58fvv\" (UID: \"ad0977ee-f824-48f4-8f9f-e15ac2c968aa\") " pod="openshift-insights/insights-runtime-extractor-58fvv"
Jul 07 16:47:43.008401 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.008376    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f753b07-d621-412c-8ffb-764e1a09ab1e-cert\") pod \"ingress-canary-hjctb\" (UID: \"9f753b07-d621-412c-8ffb-764e1a09ab1e\") " pod="openshift-ingress-canary/ingress-canary-hjctb"
Jul 07 16:47:43.008488 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.008377    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad0977ee-f824-48f4-8f9f-e15ac2c968aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-58fvv\" (UID: \"ad0977ee-f824-48f4-8f9f-e15ac2c968aa\") " pod="openshift-insights/insights-runtime-extractor-58fvv"
Jul 07 16:47:43.019238 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.019208    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr4rg\" (UniqueName: \"kubernetes.io/projected/9f753b07-d621-412c-8ffb-764e1a09ab1e-kube-api-access-hr4rg\") pod \"ingress-canary-hjctb\" (UID: \"9f753b07-d621-412c-8ffb-764e1a09ab1e\") " pod="openshift-ingress-canary/ingress-canary-hjctb"
Jul 07 16:47:43.019400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.019383    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szt8k\" (UniqueName: \"kubernetes.io/projected/ad0977ee-f824-48f4-8f9f-e15ac2c968aa-kube-api-access-szt8k\") pod \"insights-runtime-extractor-58fvv\" (UID: \"ad0977ee-f824-48f4-8f9f-e15ac2c968aa\") " pod="openshift-insights/insights-runtime-extractor-58fvv"
Jul 07 16:47:43.105661 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.105618    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqctt\" (UniqueName: \"kubernetes.io/projected/7c0e2b23-0949-4d6e-bd5e-11d1aff95701-kube-api-access-cqctt\") pod \"dns-default-grhmq\" (UID: \"7c0e2b23-0949-4d6e-bd5e-11d1aff95701\") " pod="openshift-dns/dns-default-grhmq"
Jul 07 16:47:43.105661 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.105660    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c0e2b23-0949-4d6e-bd5e-11d1aff95701-config-volume\") pod \"dns-default-grhmq\" (UID: \"7c0e2b23-0949-4d6e-bd5e-11d1aff95701\") " pod="openshift-dns/dns-default-grhmq"
Jul 07 16:47:43.105850 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.105686    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c0e2b23-0949-4d6e-bd5e-11d1aff95701-metrics-tls\") pod \"dns-default-grhmq\" (UID: \"7c0e2b23-0949-4d6e-bd5e-11d1aff95701\") " pod="openshift-dns/dns-default-grhmq"
Jul 07 16:47:43.105850 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.105825    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c0e2b23-0949-4d6e-bd5e-11d1aff95701-tmp-dir\") pod \"dns-default-grhmq\" (UID: \"7c0e2b23-0949-4d6e-bd5e-11d1aff95701\") " pod="openshift-dns/dns-default-grhmq"
Jul 07 16:47:43.106178 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.106122    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c0e2b23-0949-4d6e-bd5e-11d1aff95701-tmp-dir\") pod \"dns-default-grhmq\" (UID: \"7c0e2b23-0949-4d6e-bd5e-11d1aff95701\") " pod="openshift-dns/dns-default-grhmq"
Jul 07 16:47:43.106313 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.106287    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c0e2b23-0949-4d6e-bd5e-11d1aff95701-config-volume\") pod \"dns-default-grhmq\" (UID: \"7c0e2b23-0949-4d6e-bd5e-11d1aff95701\") " pod="openshift-dns/dns-default-grhmq"
Jul 07 16:47:43.108228 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.108212    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c0e2b23-0949-4d6e-bd5e-11d1aff95701-metrics-tls\") pod \"dns-default-grhmq\" (UID: \"7c0e2b23-0949-4d6e-bd5e-11d1aff95701\") " pod="openshift-dns/dns-default-grhmq"
Jul 07 16:47:43.115392 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.115361    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqctt\" (UniqueName: \"kubernetes.io/projected/7c0e2b23-0949-4d6e-bd5e-11d1aff95701-kube-api-access-cqctt\") pod \"dns-default-grhmq\" (UID: \"7c0e2b23-0949-4d6e-bd5e-11d1aff95701\") " pod="openshift-dns/dns-default-grhmq"
Jul 07 16:47:43.139287 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.139263    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hjctb"
Jul 07 16:47:43.145126 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.145108    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-58fvv"
Jul 07 16:47:43.235693 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.235653    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-grhmq"
Jul 07 16:47:43.286344 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.286316    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-58fvv"]
Jul 07 16:47:43.292612 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:47:43.292472    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0977ee_f824_48f4_8f9f_e15ac2c968aa.slice/crio-f2a4346403468d407fc26bcd09e81d31609222e05f34f9dc9933c64e1a7bf5c9 WatchSource:0}: Error finding container f2a4346403468d407fc26bcd09e81d31609222e05f34f9dc9933c64e1a7bf5c9: Status 404 returned error can't find the container with id f2a4346403468d407fc26bcd09e81d31609222e05f34f9dc9933c64e1a7bf5c9
Jul 07 16:47:43.299397 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.299373    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hjctb"]
Jul 07 16:47:43.305333 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:47:43.303323    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f753b07_d621_412c_8ffb_764e1a09ab1e.slice/crio-218507d94022b278d185ed3a5683a4a46ea96490f3b23fe6ec3ba584759426c2 WatchSource:0}: Error finding container 218507d94022b278d185ed3a5683a4a46ea96490f3b23fe6ec3ba584759426c2: Status 404 returned error can't find the container with id 218507d94022b278d185ed3a5683a4a46ea96490f3b23fe6ec3ba584759426c2
Jul 07 16:47:43.373015 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.372990    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-grhmq"]
Jul 07 16:47:43.375168 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:47:43.375123    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c0e2b23_0949_4d6e_bd5e_11d1aff95701.slice/crio-5271514f4087ae58ad596cc57b10f8ee24807915bc2564eb7eacc6af4a189a96 WatchSource:0}: Error finding container 5271514f4087ae58ad596cc57b10f8ee24807915bc2564eb7eacc6af4a189a96: Status 404 returned error can't find the container with id 5271514f4087ae58ad596cc57b10f8ee24807915bc2564eb7eacc6af4a189a96
Jul 07 16:47:43.584225 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.584191    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:43.584225 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.584212    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:43.584472 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.584211    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:43.588114 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.588083    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 16:47:43.588114 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.588086    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 16:47:43.588320 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.588186    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5trbh\""
Jul 07 16:47:43.588320 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.588198    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 16:47:43.588320 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.588216    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n952x\""
Jul 07 16:47:43.588460 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.588425    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 16:47:43.925318 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.925219    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-58fvv" event={"ID":"ad0977ee-f824-48f4-8f9f-e15ac2c968aa","Type":"ContainerStarted","Data":"f2a4346403468d407fc26bcd09e81d31609222e05f34f9dc9933c64e1a7bf5c9"}
Jul 07 16:47:43.927181 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.926917    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-grhmq" event={"ID":"7c0e2b23-0949-4d6e-bd5e-11d1aff95701","Type":"ContainerStarted","Data":"5271514f4087ae58ad596cc57b10f8ee24807915bc2564eb7eacc6af4a189a96"}
Jul 07 16:47:43.929457 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:43.929433    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hjctb" event={"ID":"9f753b07-d621-412c-8ffb-764e1a09ab1e","Type":"ContainerStarted","Data":"218507d94022b278d185ed3a5683a4a46ea96490f3b23fe6ec3ba584759426c2"}
Jul 07 16:47:46.943217 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:46.943181    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bvlj2"]
Jul 07 16:47:46.945784 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:46.945759    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bvlj2"
Jul 07 16:47:46.949068 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:46.949045    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-kkp57\""
Jul 07 16:47:46.949313 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:46.949289    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 07 16:47:46.959190 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:46.959162    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bvlj2"]
Jul 07 16:47:47.039656 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:47.039616    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e39c3317-fd9e-44b3-8eaf-242513bbf517-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-bvlj2\" (UID: \"e39c3317-fd9e-44b3-8eaf-242513bbf517\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bvlj2"
Jul 07 16:47:47.140844 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:47.140295    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e39c3317-fd9e-44b3-8eaf-242513bbf517-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-bvlj2\" (UID: \"e39c3317-fd9e-44b3-8eaf-242513bbf517\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bvlj2"
Jul 07 16:47:47.140844 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:47.140532    2583 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 07 16:47:47.140844 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:47.140603    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e39c3317-fd9e-44b3-8eaf-242513bbf517-tls-certificates podName:e39c3317-fd9e-44b3-8eaf-242513bbf517 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:47.640581867 +0000 UTC m=+60.653591679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/e39c3317-fd9e-44b3-8eaf-242513bbf517-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-bvlj2" (UID: "e39c3317-fd9e-44b3-8eaf-242513bbf517") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 07 16:47:47.646314 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:47.646274    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e39c3317-fd9e-44b3-8eaf-242513bbf517-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-bvlj2\" (UID: \"e39c3317-fd9e-44b3-8eaf-242513bbf517\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bvlj2"
Jul 07 16:47:47.649253 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:47.649221    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e39c3317-fd9e-44b3-8eaf-242513bbf517-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-bvlj2\" (UID: \"e39c3317-fd9e-44b3-8eaf-242513bbf517\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bvlj2"
Jul 07 16:47:47.858548 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:47.858517    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-kkp57\""
Jul 07 16:47:47.866575 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:47.866544    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bvlj2"
Jul 07 16:47:48.435825 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:48.435753    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bvlj2"]
Jul 07 16:47:48.442839 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:47:48.442810    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39c3317_fd9e_44b3_8eaf_242513bbf517.slice/crio-088f64b124d9672d000e16dac6848ded2ea31c9b751bf253d42facd18e9d2501 WatchSource:0}: Error finding container 088f64b124d9672d000e16dac6848ded2ea31c9b751bf253d42facd18e9d2501: Status 404 returned error can't find the container with id 088f64b124d9672d000e16dac6848ded2ea31c9b751bf253d42facd18e9d2501
Jul 07 16:47:48.865643 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:48.865372    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mgf45"
Jul 07 16:47:48.946629 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:48.946586    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-grhmq" event={"ID":"7c0e2b23-0949-4d6e-bd5e-11d1aff95701","Type":"ContainerStarted","Data":"55f293755fcd0337b6d92b03540e2edc17f50a8c7b193b30ae15ec844e368d2f"}
Jul 07 16:47:48.946808 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:48.946688    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-grhmq" event={"ID":"7c0e2b23-0949-4d6e-bd5e-11d1aff95701","Type":"ContainerStarted","Data":"f4ad7da5d2849fba18e42057433ba18c22ca684d8fea0b869b1cc3ded1f06671"}
Jul 07 16:47:48.946808 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:48.946753    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-grhmq"
Jul 07 16:47:48.948679 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:48.948645    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-58fvv" event={"ID":"ad0977ee-f824-48f4-8f9f-e15ac2c968aa","Type":"ContainerStarted","Data":"09265083cc2cf1d4d7d494791ea1416e43389ac055474bfa696c3c56401efb39"}
Jul 07 16:47:48.950164 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:48.950119    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hjctb" event={"ID":"9f753b07-d621-412c-8ffb-764e1a09ab1e","Type":"ContainerStarted","Data":"3a306dff59ad87f37024e9c07895c454a939ded5c95453e60767e4a91e092c6e"}
Jul 07 16:47:48.951453 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:48.951430    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bvlj2" event={"ID":"e39c3317-fd9e-44b3-8eaf-242513bbf517","Type":"ContainerStarted","Data":"088f64b124d9672d000e16dac6848ded2ea31c9b751bf253d42facd18e9d2501"}
Jul 07 16:47:48.969891 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:48.969836    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-grhmq" podStartSLOduration=2.054529753 podStartE2EDuration="6.96981959s" totalImagesPullingTime="4.915289837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:48.968287952 +0000 UTC m=+61.981297771" watchObservedRunningTime="2026-07-07 16:47:48.96981959 +0000 UTC m=+61.982829409"
Jul 07 16:47:48.987749 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:48.987707    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hjctb" podStartSLOduration=1.997312688 podStartE2EDuration="6.987691779s" totalImagesPullingTime="4.990379091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:48.986462287 +0000 UTC m=+61.999472106" watchObservedRunningTime="2026-07-07 16:47:48.987691779 +0000 UTC m=+62.000701575"
Jul 07 16:47:49.955981 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:49.955945    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-58fvv" event={"ID":"ad0977ee-f824-48f4-8f9f-e15ac2c968aa","Type":"ContainerStarted","Data":"27bd139febc234c19468c08d17c3c505dd419c2b9d08df90c9f464d1104db8db"}
Jul 07 16:47:50.960878 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:50.960838    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bvlj2" event={"ID":"e39c3317-fd9e-44b3-8eaf-242513bbf517","Type":"ContainerStarted","Data":"5df0406b720fc3d1251bb867c1dfb4cdf5eb3f2e27784b18259871c9138ea581"}
Jul 07 16:47:50.961407 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:50.961051    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bvlj2"
Jul 07 16:47:50.966664 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:50.966639    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bvlj2"
Jul 07 16:47:51.009929 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:51.009864    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-bvlj2" podStartSLOduration=3.561582806 podStartE2EDuration="5.009841197s" totalImagesPullingTime="1.448258391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:50.985278413 +0000 UTC m=+63.998288232" watchObservedRunningTime="2026-07-07 16:47:51.009841197 +0000 UTC m=+64.022851015"
Jul 07 16:47:51.968514 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:51.968470    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-58fvv" event={"ID":"ad0977ee-f824-48f4-8f9f-e15ac2c968aa","Type":"ContainerStarted","Data":"c11f3e4933cbad31e72b1f75701dc884539b4cf45600041897f91716f82c769d"}
Jul 07 16:47:51.990761 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:51.990716    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-58fvv" podStartSLOduration=1.934382193 podStartE2EDuration="9.990699199s" totalImagesPullingTime="8.056317006s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:47:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:51.989712542 +0000 UTC m=+65.002722363" watchObservedRunningTime="2026-07-07 16:47:51.990699199 +0000 UTC m=+65.003708996"
Jul 07 16:47:52.007039 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:52.007005    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-ctshv"]
Jul 07 16:47:52.009170 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:52.009129    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ctshv"
Jul 07 16:47:52.011897 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:52.011872    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 07 16:47:52.012010 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:52.011934    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 07 16:47:52.012010 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:52.011873    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 07 16:47:52.012108 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:52.012022    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 07 16:47:52.012267 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:52.012254    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 07 16:47:52.012267 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:52.012260    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-xpwjm\""
Jul 07 16:47:52.021810 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:52.021787    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-ctshv"]
Jul 07 16:47:52.083158 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:52.083105    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f33e5bc9-45a5-4351-b72d-34c1c3c34a81-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-ctshv\" (UID: \"f33e5bc9-45a5-4351-b72d-34c1c3c34a81\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ctshv"
Jul 07 16:47:52.083355 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:52.083170    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f33e5bc9-45a5-4351-b72d-34c1c3c34a81-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-ctshv\" (UID: \"f33e5bc9-45a5-4351-b72d-34c1c3c34a81\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ctshv"
Jul 07 16:47:52.083355 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:52.083206    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxslz\" (UniqueName: \"kubernetes.io/projected/f33e5bc9-45a5-4351-b72d-34c1c3c34a81-kube-api-access-nxslz\") pod \"prometheus-operator-567f7b8b7c-ctshv\" (UID: \"f33e5bc9-45a5-4351-b72d-34c1c3c34a81\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ctshv"
Jul 07 16:47:52.083355 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:52.083235    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f33e5bc9-45a5-4351-b72d-34c1c3c34a81-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-ctshv\" (UID: \"f33e5bc9-45a5-4351-b72d-34c1c3c34a81\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ctshv"
Jul 07 16:47:52.184475 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:52.184445    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxslz\" (UniqueName: \"kubernetes.io/projected/f33e5bc9-45a5-4351-b72d-34c1c3c34a81-kube-api-access-nxslz\") pod \"prometheus-operator-567f7b8b7c-ctshv\" (UID: \"f33e5bc9-45a5-4351-b72d-34c1c3c34a81\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ctshv"
Jul 07 16:47:52.184618 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:52.184483    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f33e5bc9-45a5-4351-b72d-34c1c3c34a81-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-ctshv\" (UID: \"f33e5bc9-45a5-4351-b72d-34c1c3c34a81\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ctshv"
Jul 07 16:47:52.184618 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:52.184566    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f33e5bc9-45a5-4351-b72d-34c1c3c34a81-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-ctshv\" (UID: \"f33e5bc9-45a5-4351-b72d-34c1c3c34a81\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ctshv"
Jul 07 16:47:52.184720 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:52.184656    2583 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 07 16:47:52.184720 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:52.184701    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f33e5bc9-45a5-4351-b72d-34c1c3c34a81-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-ctshv\" (UID: \"f33e5bc9-45a5-4351-b72d-34c1c3c34a81\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ctshv"
Jul 07 16:47:52.184720 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:52.184712    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f33e5bc9-45a5-4351-b72d-34c1c3c34a81-prometheus-operator-tls podName:f33e5bc9-45a5-4351-b72d-34c1c3c34a81 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:52.684696863 +0000 UTC m=+65.697706664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/f33e5bc9-45a5-4351-b72d-34c1c3c34a81-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-ctshv" (UID: "f33e5bc9-45a5-4351-b72d-34c1c3c34a81") : secret "prometheus-operator-tls" not found
Jul 07 16:47:52.185301 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:52.185282    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f33e5bc9-45a5-4351-b72d-34c1c3c34a81-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-ctshv\" (UID: \"f33e5bc9-45a5-4351-b72d-34c1c3c34a81\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ctshv"
Jul 07 16:47:52.187004 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:52.186983    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f33e5bc9-45a5-4351-b72d-34c1c3c34a81-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-ctshv\" (UID: \"f33e5bc9-45a5-4351-b72d-34c1c3c34a81\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ctshv"
Jul 07 16:47:52.194158 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:52.194111    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxslz\" (UniqueName: \"kubernetes.io/projected/f33e5bc9-45a5-4351-b72d-34c1c3c34a81-kube-api-access-nxslz\") pod \"prometheus-operator-567f7b8b7c-ctshv\" (UID: \"f33e5bc9-45a5-4351-b72d-34c1c3c34a81\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ctshv"
Jul 07 16:47:52.688126 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:52.688082    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f33e5bc9-45a5-4351-b72d-34c1c3c34a81-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-ctshv\" (UID: \"f33e5bc9-45a5-4351-b72d-34c1c3c34a81\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ctshv"
Jul 07 16:47:52.690579 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:52.690554    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f33e5bc9-45a5-4351-b72d-34c1c3c34a81-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-ctshv\" (UID: \"f33e5bc9-45a5-4351-b72d-34c1c3c34a81\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ctshv"
Jul 07 16:47:52.918406 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:52.918357    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ctshv"
Jul 07 16:47:53.048362 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:53.048327    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-ctshv"]
Jul 07 16:47:53.051335 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:47:53.051307    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33e5bc9_45a5_4351_b72d_34c1c3c34a81.slice/crio-5806fc463209e751dcbe19476d45fff83045318e3229ba36838e6abda930de52 WatchSource:0}: Error finding container 5806fc463209e751dcbe19476d45fff83045318e3229ba36838e6abda930de52: Status 404 returned error can't find the container with id 5806fc463209e751dcbe19476d45fff83045318e3229ba36838e6abda930de52
Jul 07 16:47:53.193107 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:53.193072    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b399a96b-729b-484d-8edf-e97618163cfa-metrics-certs\") pod \"network-metrics-daemon-vv52w\" (UID: \"b399a96b-729b-484d-8edf-e97618163cfa\") " pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:53.193107 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:53.193111    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26r9t\" (UniqueName: \"kubernetes.io/projected/9fc991d0-7b29-41d9-bf1a-7ba28833be28-kube-api-access-26r9t\") pod \"network-check-target-l7frr\" (UID: \"9fc991d0-7b29-41d9-bf1a-7ba28833be28\") " pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:53.193344 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:53.193159    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-original-pull-secret\") pod \"global-pull-secret-syncer-x5chr\" (UID: \"41c32952-21a9-4e19-bc8a-ceee6a08ceeb\") " pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:53.195985 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:53.195963    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 16:47:53.196099 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:53.195988    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 16:47:53.196099 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:53.195964    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 16:47:53.206687 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:53.206663    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/41c32952-21a9-4e19-bc8a-ceee6a08ceeb-original-pull-secret\") pod \"global-pull-secret-syncer-x5chr\" (UID: \"41c32952-21a9-4e19-bc8a-ceee6a08ceeb\") " pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:53.206837 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:53.206719    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 16:47:53.206837 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:53.206751    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b399a96b-729b-484d-8edf-e97618163cfa-metrics-certs\") pod \"network-metrics-daemon-vv52w\" (UID: \"b399a96b-729b-484d-8edf-e97618163cfa\") " pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:53.216984 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:53.216932    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26r9t\" (UniqueName: \"kubernetes.io/projected/9fc991d0-7b29-41d9-bf1a-7ba28833be28-kube-api-access-26r9t\") pod \"network-check-target-l7frr\" (UID: \"9fc991d0-7b29-41d9-bf1a-7ba28833be28\") " pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:53.496479 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:53.496398    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5trbh\""
Jul 07 16:47:53.501554 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:53.501536    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n952x\""
Jul 07 16:47:53.504422 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:53.504402    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vv52w"
Jul 07 16:47:53.504500 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:53.504415    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5chr"
Jul 07 16:47:53.509993 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:53.509961    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:53.651362 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:53.651326    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x5chr"]
Jul 07 16:47:53.656310 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:47:53.656284    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c32952_21a9_4e19_bc8a_ceee6a08ceeb.slice/crio-0ef36e253cf5f86fffc08be8bee42a78abba45bc8a669308465f37bcd5c8f959 WatchSource:0}: Error finding container 0ef36e253cf5f86fffc08be8bee42a78abba45bc8a669308465f37bcd5c8f959: Status 404 returned error can't find the container with id 0ef36e253cf5f86fffc08be8bee42a78abba45bc8a669308465f37bcd5c8f959
Jul 07 16:47:53.672393 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:53.672363    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vv52w"]
Jul 07 16:47:53.676456 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:47:53.676389    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb399a96b_729b_484d_8edf_e97618163cfa.slice/crio-13218e7d5305e8bd7cc6369bfeb61a2e1d87f75b6f35aaa1a55694148333811d WatchSource:0}: Error finding container 13218e7d5305e8bd7cc6369bfeb61a2e1d87f75b6f35aaa1a55694148333811d: Status 404 returned error can't find the container with id 13218e7d5305e8bd7cc6369bfeb61a2e1d87f75b6f35aaa1a55694148333811d
Jul 07 16:47:53.879531 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:53.879410    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l7frr"]
Jul 07 16:47:53.882680 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:47:53.882644    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc991d0_7b29_41d9_bf1a_7ba28833be28.slice/crio-39b9de354e4b589319498e08076a3bf29fbff8e130e5200563aa3eb8a36945f8 WatchSource:0}: Error finding container 39b9de354e4b589319498e08076a3bf29fbff8e130e5200563aa3eb8a36945f8: Status 404 returned error can't find the container with id 39b9de354e4b589319498e08076a3bf29fbff8e130e5200563aa3eb8a36945f8
Jul 07 16:47:53.975334 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:53.975293    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l7frr" event={"ID":"9fc991d0-7b29-41d9-bf1a-7ba28833be28","Type":"ContainerStarted","Data":"39b9de354e4b589319498e08076a3bf29fbff8e130e5200563aa3eb8a36945f8"}
Jul 07 16:47:53.976541 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:53.976504    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x5chr" event={"ID":"41c32952-21a9-4e19-bc8a-ceee6a08ceeb","Type":"ContainerStarted","Data":"0ef36e253cf5f86fffc08be8bee42a78abba45bc8a669308465f37bcd5c8f959"}
Jul 07 16:47:53.977557 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:53.977532    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ctshv" event={"ID":"f33e5bc9-45a5-4351-b72d-34c1c3c34a81","Type":"ContainerStarted","Data":"5806fc463209e751dcbe19476d45fff83045318e3229ba36838e6abda930de52"}
Jul 07 16:47:53.978456 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:53.978435    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vv52w" event={"ID":"b399a96b-729b-484d-8edf-e97618163cfa","Type":"ContainerStarted","Data":"13218e7d5305e8bd7cc6369bfeb61a2e1d87f75b6f35aaa1a55694148333811d"}
Jul 07 16:47:54.992684 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:54.991843    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ctshv" event={"ID":"f33e5bc9-45a5-4351-b72d-34c1c3c34a81","Type":"ContainerStarted","Data":"71d1e0ab184ecaad8f7de87750ba77c3ff6ab8db2f1ffc98d28ed7048412414a"}
Jul 07 16:47:54.992684 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:54.991911    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ctshv" event={"ID":"f33e5bc9-45a5-4351-b72d-34c1c3c34a81","Type":"ContainerStarted","Data":"edac7c1d6301dde481e913fa9d25a9610338339aa78946d74fd567f21b0bb3cd"}
Jul 07 16:47:54.996917 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:54.996335    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vv52w" event={"ID":"b399a96b-729b-484d-8edf-e97618163cfa","Type":"ContainerStarted","Data":"6bcf320460a98eb11b88b13027e587b7d6ceeb7c3b5683b96e0ca48c8af9177c"}
Jul 07 16:47:55.018094 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:55.014856    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-ctshv" podStartSLOduration=2.267197885 podStartE2EDuration="4.014832777s" totalImagesPullingTime="1.747634892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:47:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:55.011835879 +0000 UTC m=+68.024845698" watchObservedRunningTime="2026-07-07 16:47:55.014832777 +0000 UTC m=+68.027842601"
Jul 07 16:47:56.001514 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:56.001468    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vv52w" event={"ID":"b399a96b-729b-484d-8edf-e97618163cfa","Type":"ContainerStarted","Data":"db8d4981da9e1a5a1a54a733550e71a5881205d5371b2b5639f0609b7ffe1e9e"}
Jul 07 16:47:56.027111 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:56.027048    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vv52w" podStartSLOduration=67.901923447 podStartE2EDuration="1m9.027025557s" totalImagesPullingTime="1.12510211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:56.026677416 +0000 UTC m=+69.039687235" watchObservedRunningTime="2026-07-07 16:47:56.027025557 +0000 UTC m=+69.040035377"
Jul 07 16:47:57.426032 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.425993    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-j4j5l"]
Jul 07 16:47:57.444583 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.444555    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-j4j5l"
Jul 07 16:47:57.444868 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.444797    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-j4j5l"]
Jul 07 16:47:57.447375 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.447347    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 07 16:47:57.448637 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.448320    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 07 16:47:57.448637 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.448327    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-pm7wb\""
Jul 07 16:47:57.449531 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.449507    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8"]
Jul 07 16:47:57.470930 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.470897    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8"
Jul 07 16:47:57.474939 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.474913    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8"]
Jul 07 16:47:57.475955 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.475928    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 07 16:47:57.476358 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.476315    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-rdqj7\""
Jul 07 16:47:57.476839 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.476818    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 07 16:47:57.485111 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.485091    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 07 16:47:57.513300 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.513268    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dthp7"]
Jul 07 16:47:57.526601 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.526566    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/799dd927-af40-4e34-91a8-afabf2916b3c-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-j4rm8\" (UID: \"799dd927-af40-4e34-91a8-afabf2916b3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8"
Jul 07 16:47:57.526833 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.526611    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfn9l\" (UniqueName: \"kubernetes.io/projected/ef657466-f5a1-445b-8a06-b950868b9843-kube-api-access-vfn9l\") pod \"openshift-state-metrics-54cf96dfcf-j4j5l\" (UID: \"ef657466-f5a1-445b-8a06-b950868b9843\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-j4j5l"
Jul 07 16:47:57.526833 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.526679    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwz5p\" (UniqueName: \"kubernetes.io/projected/799dd927-af40-4e34-91a8-afabf2916b3c-kube-api-access-pwz5p\") pod \"kube-state-metrics-7c9dbb8b7d-j4rm8\" (UID: \"799dd927-af40-4e34-91a8-afabf2916b3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8"
Jul 07 16:47:57.526833 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.526725    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.526833 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.526743    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef657466-f5a1-445b-8a06-b950868b9843-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-j4j5l\" (UID: \"ef657466-f5a1-445b-8a06-b950868b9843\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-j4j5l"
Jul 07 16:47:57.526833 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.526820    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef657466-f5a1-445b-8a06-b950868b9843-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-j4j5l\" (UID: \"ef657466-f5a1-445b-8a06-b950868b9843\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-j4j5l"
Jul 07 16:47:57.528520 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.526853    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/799dd927-af40-4e34-91a8-afabf2916b3c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-j4rm8\" (UID: \"799dd927-af40-4e34-91a8-afabf2916b3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8"
Jul 07 16:47:57.528520 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.526915    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/799dd927-af40-4e34-91a8-afabf2916b3c-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-j4rm8\" (UID: \"799dd927-af40-4e34-91a8-afabf2916b3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8"
Jul 07 16:47:57.528520 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.526953    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/799dd927-af40-4e34-91a8-afabf2916b3c-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-j4rm8\" (UID: \"799dd927-af40-4e34-91a8-afabf2916b3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8"
Jul 07 16:47:57.528520 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.526977    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/799dd927-af40-4e34-91a8-afabf2916b3c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-j4rm8\" (UID: \"799dd927-af40-4e34-91a8-afabf2916b3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8"
Jul 07 16:47:57.528520 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.527000    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef657466-f5a1-445b-8a06-b950868b9843-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-j4j5l\" (UID: \"ef657466-f5a1-445b-8a06-b950868b9843\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-j4j5l"
Jul 07 16:47:57.530426 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.530404    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6pqjv\""
Jul 07 16:47:57.531284 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.530922    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 07 16:47:57.531284 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.531112    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 07 16:47:57.531955 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.531937    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 07 16:47:57.627833 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.627795    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef657466-f5a1-445b-8a06-b950868b9843-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-j4j5l\" (UID: \"ef657466-f5a1-445b-8a06-b950868b9843\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-j4j5l"
Jul 07 16:47:57.628026 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.627846    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/799dd927-af40-4e34-91a8-afabf2916b3c-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-j4rm8\" (UID: \"799dd927-af40-4e34-91a8-afabf2916b3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8"
Jul 07 16:47:57.628026 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.627878    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef657466-f5a1-445b-8a06-b950868b9843-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-j4j5l\" (UID: \"ef657466-f5a1-445b-8a06-b950868b9843\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-j4j5l"
Jul 07 16:47:57.628026 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.627909    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfn9l\" (UniqueName: \"kubernetes.io/projected/ef657466-f5a1-445b-8a06-b950868b9843-kube-api-access-vfn9l\") pod \"openshift-state-metrics-54cf96dfcf-j4j5l\" (UID: \"ef657466-f5a1-445b-8a06-b950868b9843\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-j4j5l"
Jul 07 16:47:57.628026 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.627935    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/799dd927-af40-4e34-91a8-afabf2916b3c-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-j4rm8\" (UID: \"799dd927-af40-4e34-91a8-afabf2916b3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8"
Jul 07 16:47:57.628026 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.627966    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/98cd7ce4-4275-4530-8b3d-94cd6ee8f305-node-exporter-textfile\") pod \"node-exporter-dthp7\" (UID: \"98cd7ce4-4275-4530-8b3d-94cd6ee8f305\") " pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.628026 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.628002    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/98cd7ce4-4275-4530-8b3d-94cd6ee8f305-root\") pod \"node-exporter-dthp7\" (UID: \"98cd7ce4-4275-4530-8b3d-94cd6ee8f305\") " pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.628362 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.628031    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/799dd927-af40-4e34-91a8-afabf2916b3c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-j4rm8\" (UID: \"799dd927-af40-4e34-91a8-afabf2916b3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8"
Jul 07 16:47:57.628362 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.628064    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef657466-f5a1-445b-8a06-b950868b9843-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-j4j5l\" (UID: \"ef657466-f5a1-445b-8a06-b950868b9843\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-j4j5l"
Jul 07 16:47:57.628362 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.628090    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/799dd927-af40-4e34-91a8-afabf2916b3c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-j4rm8\" (UID: \"799dd927-af40-4e34-91a8-afabf2916b3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8"
Jul 07 16:47:57.628362 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.628119    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/799dd927-af40-4e34-91a8-afabf2916b3c-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-j4rm8\" (UID: \"799dd927-af40-4e34-91a8-afabf2916b3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8"
Jul 07 16:47:57.628362 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.628162    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwz5p\" (UniqueName: \"kubernetes.io/projected/799dd927-af40-4e34-91a8-afabf2916b3c-kube-api-access-pwz5p\") pod \"kube-state-metrics-7c9dbb8b7d-j4rm8\" (UID: \"799dd927-af40-4e34-91a8-afabf2916b3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8"
Jul 07 16:47:57.628362 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.628198    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/98cd7ce4-4275-4530-8b3d-94cd6ee8f305-node-exporter-tls\") pod \"node-exporter-dthp7\" (UID: \"98cd7ce4-4275-4530-8b3d-94cd6ee8f305\") " pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.628362 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.628233    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/98cd7ce4-4275-4530-8b3d-94cd6ee8f305-node-exporter-wtmp\") pod \"node-exporter-dthp7\" (UID: \"98cd7ce4-4275-4530-8b3d-94cd6ee8f305\") " pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.628362 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.628261    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98cd7ce4-4275-4530-8b3d-94cd6ee8f305-metrics-client-ca\") pod \"node-exporter-dthp7\" (UID: \"98cd7ce4-4275-4530-8b3d-94cd6ee8f305\") " pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.628362 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.628287    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98cd7ce4-4275-4530-8b3d-94cd6ee8f305-sys\") pod \"node-exporter-dthp7\" (UID: \"98cd7ce4-4275-4530-8b3d-94cd6ee8f305\") " pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.628362 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.628318    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/98cd7ce4-4275-4530-8b3d-94cd6ee8f305-node-exporter-accelerators-collector-config\") pod \"node-exporter-dthp7\" (UID: \"98cd7ce4-4275-4530-8b3d-94cd6ee8f305\") " pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.628362 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.628346    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98cd7ce4-4275-4530-8b3d-94cd6ee8f305-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dthp7\" (UID: \"98cd7ce4-4275-4530-8b3d-94cd6ee8f305\") " pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.628875 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.628370    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgqh9\" (UniqueName: \"kubernetes.io/projected/98cd7ce4-4275-4530-8b3d-94cd6ee8f305-kube-api-access-dgqh9\") pod \"node-exporter-dthp7\" (UID: \"98cd7ce4-4275-4530-8b3d-94cd6ee8f305\") " pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.628875 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.628556    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef657466-f5a1-445b-8a06-b950868b9843-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-j4j5l\" (UID: \"ef657466-f5a1-445b-8a06-b950868b9843\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-j4j5l"
Jul 07 16:47:57.628875 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.628609    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/799dd927-af40-4e34-91a8-afabf2916b3c-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-j4rm8\" (UID: \"799dd927-af40-4e34-91a8-afabf2916b3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8"
Jul 07 16:47:57.628875 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.628721    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/799dd927-af40-4e34-91a8-afabf2916b3c-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-j4rm8\" (UID: \"799dd927-af40-4e34-91a8-afabf2916b3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8"
Jul 07 16:47:57.629045 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.628921    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/799dd927-af40-4e34-91a8-afabf2916b3c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-j4rm8\" (UID: \"799dd927-af40-4e34-91a8-afabf2916b3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8"
Jul 07 16:47:57.631932 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.631230    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef657466-f5a1-445b-8a06-b950868b9843-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-j4j5l\" (UID: \"ef657466-f5a1-445b-8a06-b950868b9843\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-j4j5l"
Jul 07 16:47:57.631932 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.631527    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef657466-f5a1-445b-8a06-b950868b9843-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-j4j5l\" (UID: \"ef657466-f5a1-445b-8a06-b950868b9843\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-j4j5l"
Jul 07 16:47:57.631932 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.631715    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/799dd927-af40-4e34-91a8-afabf2916b3c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-j4rm8\" (UID: \"799dd927-af40-4e34-91a8-afabf2916b3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8"
Jul 07 16:47:57.632531 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.632511    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/799dd927-af40-4e34-91a8-afabf2916b3c-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-j4rm8\" (UID: \"799dd927-af40-4e34-91a8-afabf2916b3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8"
Jul 07 16:47:57.645157 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.645111    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwz5p\" (UniqueName: \"kubernetes.io/projected/799dd927-af40-4e34-91a8-afabf2916b3c-kube-api-access-pwz5p\") pod \"kube-state-metrics-7c9dbb8b7d-j4rm8\" (UID: \"799dd927-af40-4e34-91a8-afabf2916b3c\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8"
Jul 07 16:47:57.646362 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.646336    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfn9l\" (UniqueName: \"kubernetes.io/projected/ef657466-f5a1-445b-8a06-b950868b9843-kube-api-access-vfn9l\") pod \"openshift-state-metrics-54cf96dfcf-j4j5l\" (UID: \"ef657466-f5a1-445b-8a06-b950868b9843\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-j4j5l"
Jul 07 16:47:57.728799 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.728762    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98cd7ce4-4275-4530-8b3d-94cd6ee8f305-sys\") pod \"node-exporter-dthp7\" (UID: \"98cd7ce4-4275-4530-8b3d-94cd6ee8f305\") " pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.728974 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.728816    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/98cd7ce4-4275-4530-8b3d-94cd6ee8f305-node-exporter-accelerators-collector-config\") pod \"node-exporter-dthp7\" (UID: \"98cd7ce4-4275-4530-8b3d-94cd6ee8f305\") " pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.728974 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.728849    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98cd7ce4-4275-4530-8b3d-94cd6ee8f305-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dthp7\" (UID: \"98cd7ce4-4275-4530-8b3d-94cd6ee8f305\") " pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.728974 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.728878    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgqh9\" (UniqueName: \"kubernetes.io/projected/98cd7ce4-4275-4530-8b3d-94cd6ee8f305-kube-api-access-dgqh9\") pod \"node-exporter-dthp7\" (UID: \"98cd7ce4-4275-4530-8b3d-94cd6ee8f305\") " pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.728974 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.728931    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/98cd7ce4-4275-4530-8b3d-94cd6ee8f305-node-exporter-textfile\") pod \"node-exporter-dthp7\" (UID: \"98cd7ce4-4275-4530-8b3d-94cd6ee8f305\") " pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.728974 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.728970    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/98cd7ce4-4275-4530-8b3d-94cd6ee8f305-root\") pod \"node-exporter-dthp7\" (UID: \"98cd7ce4-4275-4530-8b3d-94cd6ee8f305\") " pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.729260 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.729007    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/98cd7ce4-4275-4530-8b3d-94cd6ee8f305-node-exporter-tls\") pod \"node-exporter-dthp7\" (UID: \"98cd7ce4-4275-4530-8b3d-94cd6ee8f305\") " pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.729260 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.729037    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/98cd7ce4-4275-4530-8b3d-94cd6ee8f305-node-exporter-wtmp\") pod \"node-exporter-dthp7\" (UID: \"98cd7ce4-4275-4530-8b3d-94cd6ee8f305\") " pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.729260 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.729063    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98cd7ce4-4275-4530-8b3d-94cd6ee8f305-metrics-client-ca\") pod \"node-exporter-dthp7\" (UID: \"98cd7ce4-4275-4530-8b3d-94cd6ee8f305\") " pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.729636 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.729612    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98cd7ce4-4275-4530-8b3d-94cd6ee8f305-metrics-client-ca\") pod \"node-exporter-dthp7\" (UID: \"98cd7ce4-4275-4530-8b3d-94cd6ee8f305\") " pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.729789 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.729691    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98cd7ce4-4275-4530-8b3d-94cd6ee8f305-sys\") pod \"node-exporter-dthp7\" (UID: \"98cd7ce4-4275-4530-8b3d-94cd6ee8f305\") " pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.730151 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.730114    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/98cd7ce4-4275-4530-8b3d-94cd6ee8f305-node-exporter-accelerators-collector-config\") pod \"node-exporter-dthp7\" (UID: \"98cd7ce4-4275-4530-8b3d-94cd6ee8f305\") " pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.730268 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.730245    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/98cd7ce4-4275-4530-8b3d-94cd6ee8f305-root\") pod \"node-exporter-dthp7\" (UID: \"98cd7ce4-4275-4530-8b3d-94cd6ee8f305\") " pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.730391 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.730372    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/98cd7ce4-4275-4530-8b3d-94cd6ee8f305-node-exporter-textfile\") pod \"node-exporter-dthp7\" (UID: \"98cd7ce4-4275-4530-8b3d-94cd6ee8f305\") " pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.730530 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.730511    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/98cd7ce4-4275-4530-8b3d-94cd6ee8f305-node-exporter-wtmp\") pod \"node-exporter-dthp7\" (UID: \"98cd7ce4-4275-4530-8b3d-94cd6ee8f305\") " pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.732949 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.732908    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98cd7ce4-4275-4530-8b3d-94cd6ee8f305-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dthp7\" (UID: \"98cd7ce4-4275-4530-8b3d-94cd6ee8f305\") " pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.733298 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.733271    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/98cd7ce4-4275-4530-8b3d-94cd6ee8f305-node-exporter-tls\") pod \"node-exporter-dthp7\" (UID: \"98cd7ce4-4275-4530-8b3d-94cd6ee8f305\") " pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.744965 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.744939    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgqh9\" (UniqueName: \"kubernetes.io/projected/98cd7ce4-4275-4530-8b3d-94cd6ee8f305-kube-api-access-dgqh9\") pod \"node-exporter-dthp7\" (UID: \"98cd7ce4-4275-4530-8b3d-94cd6ee8f305\") " pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:57.755910 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.755882    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-j4j5l"
Jul 07 16:47:57.783167 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.783116    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8"
Jul 07 16:47:57.838442 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:57.838406    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dthp7"
Jul 07 16:47:58.528512 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.528475    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 16:47:58.531754 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.531733    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.535360 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.535326    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 07 16:47:58.535542 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.535327    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 07 16:47:58.535542 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.535329    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 07 16:47:58.535691 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.535668    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 07 16:47:58.536553 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.536327    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 07 16:47:58.536553 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.536368    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 07 16:47:58.536553 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.536394    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mff2l\""
Jul 07 16:47:58.536766 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.536685    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 07 16:47:58.536766 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.536699    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 07 16:47:58.536766 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.536712    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 07 16:47:58.549593 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.549553    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 16:47:58.638227 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.636548    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-web-config\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.638227 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.636593    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.638227 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.636648    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5hww\" (UniqueName: \"kubernetes.io/projected/c216e6fb-0687-4a74-b41b-982f4ef8902b-kube-api-access-l5hww\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.638227 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.636679    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c216e6fb-0687-4a74-b41b-982f4ef8902b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.638227 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.636791    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.638227 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.636833    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c216e6fb-0687-4a74-b41b-982f4ef8902b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.638227 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.636880    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c216e6fb-0687-4a74-b41b-982f4ef8902b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.638227 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.636939    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.638227 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.636971    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-config-volume\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.638227 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.636999    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c216e6fb-0687-4a74-b41b-982f4ef8902b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.638227 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.637029    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.638227 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.637054    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c216e6fb-0687-4a74-b41b-982f4ef8902b-config-out\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.638227 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.637085    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.738303 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.738267    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.738504 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.738315    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c216e6fb-0687-4a74-b41b-982f4ef8902b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.738504 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.738354    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c216e6fb-0687-4a74-b41b-982f4ef8902b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.738504 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.738383    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.738504 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.738411    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-config-volume\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.738504 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.738438    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c216e6fb-0687-4a74-b41b-982f4ef8902b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.738504 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.738466    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.738504 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.738491    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c216e6fb-0687-4a74-b41b-982f4ef8902b-config-out\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.738837 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.738519    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.738837 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.738562    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-web-config\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.738837 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.738587    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.738837 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.738613    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5hww\" (UniqueName: \"kubernetes.io/projected/c216e6fb-0687-4a74-b41b-982f4ef8902b-kube-api-access-l5hww\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.738837 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.738638    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c216e6fb-0687-4a74-b41b-982f4ef8902b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.739208 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.739173    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c216e6fb-0687-4a74-b41b-982f4ef8902b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.739333 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:58.739295    2583 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 07 16:47:58.739333 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:47:58.739356    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-secret-alertmanager-main-tls podName:c216e6fb-0687-4a74-b41b-982f4ef8902b nodeName:}" failed. No retries permitted until 2026-07-07 16:47:59.239337808 +0000 UTC m=+72.252347621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "c216e6fb-0687-4a74-b41b-982f4ef8902b") : secret "alertmanager-main-tls" not found
Jul 07 16:47:58.739643 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.739404    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c216e6fb-0687-4a74-b41b-982f4ef8902b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.739643 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.739435    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c216e6fb-0687-4a74-b41b-982f4ef8902b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.741711 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.741685    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c216e6fb-0687-4a74-b41b-982f4ef8902b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.742074 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.742023    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.742635 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.742610    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-config-volume\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.742720 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.742673    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.742905 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.742888    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c216e6fb-0687-4a74-b41b-982f4ef8902b-config-out\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.743187 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.743163    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.743634 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.743603    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-web-config\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.743802 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.743785    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.779568 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.779494    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5hww\" (UniqueName: \"kubernetes.io/projected/c216e6fb-0687-4a74-b41b-982f4ef8902b-kube-api-access-l5hww\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:58.961178 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:58.960916    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-grhmq"
Jul 07 16:47:59.028014 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:59.027931    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dthp7" event={"ID":"98cd7ce4-4275-4530-8b3d-94cd6ee8f305","Type":"ContainerStarted","Data":"5a03939c38e4dbe572f24474f7a164b3fa9b9d5903d890e1aa3f7a31148d5230"}
Jul 07 16:47:59.031265 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:59.031114    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l7frr" event={"ID":"9fc991d0-7b29-41d9-bf1a-7ba28833be28","Type":"ContainerStarted","Data":"a7b84a0848f9940dd384702102bc97a742572b8235e73d64fd4060d9cc383ddc"}
Jul 07 16:47:59.031513 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:59.031322    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:47:59.055489 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:59.055431    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-l7frr" podStartSLOduration=67.015910795 podStartE2EDuration="1m12.055409841s" totalImagesPullingTime="5.039499046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:59.055376814 +0000 UTC m=+72.068386643" watchObservedRunningTime="2026-07-07 16:47:59.055409841 +0000 UTC m=+72.068419660"
Jul 07 16:47:59.083850 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:59.083821    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8"]
Jul 07 16:47:59.085265 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:47:59.084940    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799dd927_af40_4e34_91a8_afabf2916b3c.slice/crio-55ae8bdfa13ae92cee3a73222a88e3200b6ed43bf8c684c9ab14a2e5c9416220 WatchSource:0}: Error finding container 55ae8bdfa13ae92cee3a73222a88e3200b6ed43bf8c684c9ab14a2e5c9416220: Status 404 returned error can't find the container with id 55ae8bdfa13ae92cee3a73222a88e3200b6ed43bf8c684c9ab14a2e5c9416220
Jul 07 16:47:59.103066 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:59.102904    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-j4j5l"]
Jul 07 16:47:59.106349 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:47:59.106321    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef657466_f5a1_445b_8a06_b950868b9843.slice/crio-11054bf5fe2c82d226d55d8e1564754b795b1189ff9ea85461170b236d18c3cf WatchSource:0}: Error finding container 11054bf5fe2c82d226d55d8e1564754b795b1189ff9ea85461170b236d18c3cf: Status 404 returned error can't find the container with id 11054bf5fe2c82d226d55d8e1564754b795b1189ff9ea85461170b236d18c3cf
Jul 07 16:47:59.246861 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:59.246826    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:59.249537 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:59.249510    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:59.443703 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:59.443667    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:47:59.636362 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:47:59.636329    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 16:47:59.797104 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:47:59.797030    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc216e6fb_0687_4a74_b41b_982f4ef8902b.slice/crio-005360d769336839450531a454430744ce7ee918d403d7eeccc3617f72a35d83 WatchSource:0}: Error finding container 005360d769336839450531a454430744ce7ee918d403d7eeccc3617f72a35d83: Status 404 returned error can't find the container with id 005360d769336839450531a454430744ce7ee918d403d7eeccc3617f72a35d83
Jul 07 16:48:00.037938 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.037894    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c216e6fb-0687-4a74-b41b-982f4ef8902b","Type":"ContainerStarted","Data":"005360d769336839450531a454430744ce7ee918d403d7eeccc3617f72a35d83"}
Jul 07 16:48:00.040067 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.040040    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x5chr" event={"ID":"41c32952-21a9-4e19-bc8a-ceee6a08ceeb","Type":"ContainerStarted","Data":"5f81dbfebe2412e3f40fc76a37fea465036a9fb7d07be1e46a2aea9094cf6839"}
Jul 07 16:48:00.041534 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.041512    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8" event={"ID":"799dd927-af40-4e34-91a8-afabf2916b3c","Type":"ContainerStarted","Data":"55ae8bdfa13ae92cee3a73222a88e3200b6ed43bf8c684c9ab14a2e5c9416220"}
Jul 07 16:48:00.043575 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.043550    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dthp7" event={"ID":"98cd7ce4-4275-4530-8b3d-94cd6ee8f305","Type":"ContainerStarted","Data":"8fa8c01658531c6a4a01f3e16100bb7f64140cae6ec215be03f1821e89ab0c92"}
Jul 07 16:48:00.046604 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.046574    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-j4j5l" event={"ID":"ef657466-f5a1-445b-8a06-b950868b9843","Type":"ContainerStarted","Data":"e6e44947207aa0f87c40f80ac691a9c3fd63a5d4d682485741c1df94c65523c7"}
Jul 07 16:48:00.046711 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.046628    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-j4j5l" event={"ID":"ef657466-f5a1-445b-8a06-b950868b9843","Type":"ContainerStarted","Data":"992fdf4726ee440b69d7c18bc7707cfefc1696c268ba497f7d95029202277e39"}
Jul 07 16:48:00.046711 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.046656    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-j4j5l" event={"ID":"ef657466-f5a1-445b-8a06-b950868b9843","Type":"ContainerStarted","Data":"11054bf5fe2c82d226d55d8e1564754b795b1189ff9ea85461170b236d18c3cf"}
Jul 07 16:48:00.070840 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.070720    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-x5chr" podStartSLOduration=67.798660477 podStartE2EDuration="1m13.070700398s" totalImagesPullingTime="5.272039921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:00.070200985 +0000 UTC m=+73.083210804" watchObservedRunningTime="2026-07-07 16:48:00.070700398 +0000 UTC m=+73.083710201"
Jul 07 16:48:00.557513 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.557472    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"]
Jul 07 16:48:00.560478 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.560447    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:00.563505 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.563240    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 07 16:48:00.563505 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.563272    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 07 16:48:00.563969 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.563754    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 07 16:48:00.563969 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.563808    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-cejota9394oq2\""
Jul 07 16:48:00.563969 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.563840    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 07 16:48:00.563969 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.563868    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 07 16:48:00.564271 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.564255    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-jxvfh\""
Jul 07 16:48:00.579444 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.579414    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"]
Jul 07 16:48:00.659591 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.659571    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63a50a99-77c3-477d-9e21-cd1066fb698d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bb46bd4d-s5qzv\" (UID: \"63a50a99-77c3-477d-9e21-cd1066fb698d\") " pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:00.659942 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.659611    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63a50a99-77c3-477d-9e21-cd1066fb698d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bb46bd4d-s5qzv\" (UID: \"63a50a99-77c3-477d-9e21-cd1066fb698d\") " pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:00.659942 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.659638    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/63a50a99-77c3-477d-9e21-cd1066fb698d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bb46bd4d-s5qzv\" (UID: \"63a50a99-77c3-477d-9e21-cd1066fb698d\") " pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:00.659942 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.659663    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63a50a99-77c3-477d-9e21-cd1066fb698d-metrics-client-ca\") pod \"thanos-querier-5bb46bd4d-s5qzv\" (UID: \"63a50a99-77c3-477d-9e21-cd1066fb698d\") " pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:00.659942 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.659708    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/63a50a99-77c3-477d-9e21-cd1066fb698d-secret-grpc-tls\") pod \"thanos-querier-5bb46bd4d-s5qzv\" (UID: \"63a50a99-77c3-477d-9e21-cd1066fb698d\") " pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:00.659942 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.659803    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/63a50a99-77c3-477d-9e21-cd1066fb698d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bb46bd4d-s5qzv\" (UID: \"63a50a99-77c3-477d-9e21-cd1066fb698d\") " pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:00.659942 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.659895    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g825z\" (UniqueName: \"kubernetes.io/projected/63a50a99-77c3-477d-9e21-cd1066fb698d-kube-api-access-g825z\") pod \"thanos-querier-5bb46bd4d-s5qzv\" (UID: \"63a50a99-77c3-477d-9e21-cd1066fb698d\") " pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:00.659942 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.659922    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/63a50a99-77c3-477d-9e21-cd1066fb698d-secret-thanos-querier-tls\") pod \"thanos-querier-5bb46bd4d-s5qzv\" (UID: \"63a50a99-77c3-477d-9e21-cd1066fb698d\") " pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:00.760340 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.760304    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/63a50a99-77c3-477d-9e21-cd1066fb698d-secret-thanos-querier-tls\") pod \"thanos-querier-5bb46bd4d-s5qzv\" (UID: \"63a50a99-77c3-477d-9e21-cd1066fb698d\") " pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:00.760501 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.760379    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63a50a99-77c3-477d-9e21-cd1066fb698d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bb46bd4d-s5qzv\" (UID: \"63a50a99-77c3-477d-9e21-cd1066fb698d\") " pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:00.760501 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.760410    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63a50a99-77c3-477d-9e21-cd1066fb698d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bb46bd4d-s5qzv\" (UID: \"63a50a99-77c3-477d-9e21-cd1066fb698d\") " pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:00.760501 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.760437    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/63a50a99-77c3-477d-9e21-cd1066fb698d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bb46bd4d-s5qzv\" (UID: \"63a50a99-77c3-477d-9e21-cd1066fb698d\") " pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:00.760501 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.760463    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63a50a99-77c3-477d-9e21-cd1066fb698d-metrics-client-ca\") pod \"thanos-querier-5bb46bd4d-s5qzv\" (UID: \"63a50a99-77c3-477d-9e21-cd1066fb698d\") " pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:00.760501 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.760488    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/63a50a99-77c3-477d-9e21-cd1066fb698d-secret-grpc-tls\") pod \"thanos-querier-5bb46bd4d-s5qzv\" (UID: \"63a50a99-77c3-477d-9e21-cd1066fb698d\") " pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:00.760742 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.760548    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/63a50a99-77c3-477d-9e21-cd1066fb698d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bb46bd4d-s5qzv\" (UID: \"63a50a99-77c3-477d-9e21-cd1066fb698d\") " pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:00.760742 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.760609    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g825z\" (UniqueName: \"kubernetes.io/projected/63a50a99-77c3-477d-9e21-cd1066fb698d-kube-api-access-g825z\") pod \"thanos-querier-5bb46bd4d-s5qzv\" (UID: \"63a50a99-77c3-477d-9e21-cd1066fb698d\") " pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:00.761621 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.761591    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63a50a99-77c3-477d-9e21-cd1066fb698d-metrics-client-ca\") pod \"thanos-querier-5bb46bd4d-s5qzv\" (UID: \"63a50a99-77c3-477d-9e21-cd1066fb698d\") " pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:00.764478 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.764453    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63a50a99-77c3-477d-9e21-cd1066fb698d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bb46bd4d-s5qzv\" (UID: \"63a50a99-77c3-477d-9e21-cd1066fb698d\") " pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:00.764638 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.764616    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/63a50a99-77c3-477d-9e21-cd1066fb698d-secret-thanos-querier-tls\") pod \"thanos-querier-5bb46bd4d-s5qzv\" (UID: \"63a50a99-77c3-477d-9e21-cd1066fb698d\") " pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:00.764762 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.764726    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63a50a99-77c3-477d-9e21-cd1066fb698d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bb46bd4d-s5qzv\" (UID: \"63a50a99-77c3-477d-9e21-cd1066fb698d\") " pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:00.765076 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.765037    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/63a50a99-77c3-477d-9e21-cd1066fb698d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bb46bd4d-s5qzv\" (UID: \"63a50a99-77c3-477d-9e21-cd1066fb698d\") " pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:00.765911 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.765869    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/63a50a99-77c3-477d-9e21-cd1066fb698d-secret-grpc-tls\") pod \"thanos-querier-5bb46bd4d-s5qzv\" (UID: \"63a50a99-77c3-477d-9e21-cd1066fb698d\") " pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:00.767442 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.767421    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/63a50a99-77c3-477d-9e21-cd1066fb698d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bb46bd4d-s5qzv\" (UID: \"63a50a99-77c3-477d-9e21-cd1066fb698d\") " pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:00.770851 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.770657    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g825z\" (UniqueName: \"kubernetes.io/projected/63a50a99-77c3-477d-9e21-cd1066fb698d-kube-api-access-g825z\") pod \"thanos-querier-5bb46bd4d-s5qzv\" (UID: \"63a50a99-77c3-477d-9e21-cd1066fb698d\") " pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:00.872461 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:00.872345    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:01.055916 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:01.055876    2583 generic.go:356] "Generic (PLEG): container finished" podID="98cd7ce4-4275-4530-8b3d-94cd6ee8f305" containerID="8fa8c01658531c6a4a01f3e16100bb7f64140cae6ec215be03f1821e89ab0c92" exitCode=0
Jul 07 16:48:01.056094 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:01.055978    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dthp7" event={"ID":"98cd7ce4-4275-4530-8b3d-94cd6ee8f305","Type":"ContainerDied","Data":"8fa8c01658531c6a4a01f3e16100bb7f64140cae6ec215be03f1821e89ab0c92"}
Jul 07 16:48:01.059863 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:01.059833    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-j4j5l" event={"ID":"ef657466-f5a1-445b-8a06-b950868b9843","Type":"ContainerStarted","Data":"26d42865cb7be9e9ad154ec5e0ec08db0bfda0275757e2ae65c77458333b8604"}
Jul 07 16:48:01.062306 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:01.062241    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8" event={"ID":"799dd927-af40-4e34-91a8-afabf2916b3c","Type":"ContainerStarted","Data":"056a697d12221e8fe4d7b22568af1c2c85f7b9d3c9f37b16ce4805f74f34e0e9"}
Jul 07 16:48:01.062306 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:01.062298    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8" event={"ID":"799dd927-af40-4e34-91a8-afabf2916b3c","Type":"ContainerStarted","Data":"e1ca553e8184039e4f5c54561a8eaf3856aa2bfbca095281c9e7fb21f79aa31b"}
Jul 07 16:48:01.062456 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:01.062325    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8" event={"ID":"799dd927-af40-4e34-91a8-afabf2916b3c","Type":"ContainerStarted","Data":"90d175e864817bc293547d79c51f602de6461ff3bbca827124d3f65d41b66331"}
Jul 07 16:48:01.115930 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:01.115863    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-j4rm8" podStartSLOduration=2.550135981 podStartE2EDuration="4.115842473s" totalImagesPullingTime="1.565706492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:01.113533752 +0000 UTC m=+74.126543571" watchObservedRunningTime="2026-07-07 16:48:01.115842473 +0000 UTC m=+74.128852292"
Jul 07 16:48:01.151098 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:01.151034    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-j4j5l" podStartSLOduration=3.111878615 podStartE2EDuration="4.151012025s" totalImagesPullingTime="1.03913341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:01.150613313 +0000 UTC m=+74.163623132" watchObservedRunningTime="2026-07-07 16:48:01.151012025 +0000 UTC m=+74.164021843"
Jul 07 16:48:01.302419 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:01.302388    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"]
Jul 07 16:48:01.307009 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:48:01.306984    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63a50a99_77c3_477d_9e21_cd1066fb698d.slice/crio-9df0b1765d402b495bafd3bce2bedd23c0fb7f45fb54e894464f0e4c09178823 WatchSource:0}: Error finding container 9df0b1765d402b495bafd3bce2bedd23c0fb7f45fb54e894464f0e4c09178823: Status 404 returned error can't find the container with id 9df0b1765d402b495bafd3bce2bedd23c0fb7f45fb54e894464f0e4c09178823
Jul 07 16:48:01.842709 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:01.842670    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-78665f68c4-cj9jt"]
Jul 07 16:48:01.845028 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:01.845010    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt"
Jul 07 16:48:01.847881 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:01.847855    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 07 16:48:01.848787 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:01.848762    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-31qcofbf6c9hj\""
Jul 07 16:48:01.848900 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:01.848817    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 07 16:48:01.848964 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:01.848943    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 07 16:48:01.849023 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:01.848960    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 07 16:48:01.849388 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:01.849165    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4fwf8\""
Jul 07 16:48:01.862451 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:01.862424    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-78665f68c4-cj9jt"]
Jul 07 16:48:01.969597 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:01.969555    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d86c8264-ee31-4fd3-a988-79bc42548e51-client-ca-bundle\") pod \"metrics-server-78665f68c4-cj9jt\" (UID: \"d86c8264-ee31-4fd3-a988-79bc42548e51\") " pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt"
Jul 07 16:48:01.969794 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:01.969610    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkgxx\" (UniqueName: \"kubernetes.io/projected/d86c8264-ee31-4fd3-a988-79bc42548e51-kube-api-access-qkgxx\") pod \"metrics-server-78665f68c4-cj9jt\" (UID: \"d86c8264-ee31-4fd3-a988-79bc42548e51\") " pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt"
Jul 07 16:48:01.969794 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:01.969640    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86c8264-ee31-4fd3-a988-79bc42548e51-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78665f68c4-cj9jt\" (UID: \"d86c8264-ee31-4fd3-a988-79bc42548e51\") " pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt"
Jul 07 16:48:01.969794 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:01.969682    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d86c8264-ee31-4fd3-a988-79bc42548e51-secret-metrics-server-client-certs\") pod \"metrics-server-78665f68c4-cj9jt\" (UID: \"d86c8264-ee31-4fd3-a988-79bc42548e51\") " pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt"
Jul 07 16:48:01.969794 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:01.969702    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d86c8264-ee31-4fd3-a988-79bc42548e51-audit-log\") pod \"metrics-server-78665f68c4-cj9jt\" (UID: \"d86c8264-ee31-4fd3-a988-79bc42548e51\") " pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt"
Jul 07 16:48:01.969794 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:01.969729    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d86c8264-ee31-4fd3-a988-79bc42548e51-secret-metrics-server-tls\") pod \"metrics-server-78665f68c4-cj9jt\" (UID: \"d86c8264-ee31-4fd3-a988-79bc42548e51\") " pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt"
Jul 07 16:48:01.969794 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:01.969776    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d86c8264-ee31-4fd3-a988-79bc42548e51-metrics-server-audit-profiles\") pod \"metrics-server-78665f68c4-cj9jt\" (UID: \"d86c8264-ee31-4fd3-a988-79bc42548e51\") " pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt"
Jul 07 16:48:02.068635 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.068599    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dthp7" event={"ID":"98cd7ce4-4275-4530-8b3d-94cd6ee8f305","Type":"ContainerStarted","Data":"cb7510d50b86df777d2b7787240f5a449d6bc5aa941a3eab284381cd5d4ecc0e"}
Jul 07 16:48:02.068833 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.068664    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dthp7" event={"ID":"98cd7ce4-4275-4530-8b3d-94cd6ee8f305","Type":"ContainerStarted","Data":"8966234f73e8d94c6eeeb18750d0039e41455a6e915f7ca75f10077d00673182"}
Jul 07 16:48:02.070158 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.070111    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv" event={"ID":"63a50a99-77c3-477d-9e21-cd1066fb698d","Type":"ContainerStarted","Data":"9df0b1765d402b495bafd3bce2bedd23c0fb7f45fb54e894464f0e4c09178823"}
Jul 07 16:48:02.070299 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.070182    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkgxx\" (UniqueName: \"kubernetes.io/projected/d86c8264-ee31-4fd3-a988-79bc42548e51-kube-api-access-qkgxx\") pod \"metrics-server-78665f68c4-cj9jt\" (UID: \"d86c8264-ee31-4fd3-a988-79bc42548e51\") " pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt"
Jul 07 16:48:02.070299 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.070225    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86c8264-ee31-4fd3-a988-79bc42548e51-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78665f68c4-cj9jt\" (UID: \"d86c8264-ee31-4fd3-a988-79bc42548e51\") " pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt"
Jul 07 16:48:02.070299 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.070269    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d86c8264-ee31-4fd3-a988-79bc42548e51-secret-metrics-server-client-certs\") pod \"metrics-server-78665f68c4-cj9jt\" (UID: \"d86c8264-ee31-4fd3-a988-79bc42548e51\") " pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt"
Jul 07 16:48:02.070299 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.070294    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d86c8264-ee31-4fd3-a988-79bc42548e51-audit-log\") pod \"metrics-server-78665f68c4-cj9jt\" (UID: \"d86c8264-ee31-4fd3-a988-79bc42548e51\") " pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt"
Jul 07 16:48:02.070518 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.070314    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d86c8264-ee31-4fd3-a988-79bc42548e51-secret-metrics-server-tls\") pod \"metrics-server-78665f68c4-cj9jt\" (UID: \"d86c8264-ee31-4fd3-a988-79bc42548e51\") " pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt"
Jul 07 16:48:02.070518 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.070343    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d86c8264-ee31-4fd3-a988-79bc42548e51-metrics-server-audit-profiles\") pod \"metrics-server-78665f68c4-cj9jt\" (UID: \"d86c8264-ee31-4fd3-a988-79bc42548e51\") " pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt"
Jul 07 16:48:02.070518 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.070372    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d86c8264-ee31-4fd3-a988-79bc42548e51-client-ca-bundle\") pod \"metrics-server-78665f68c4-cj9jt\" (UID: \"d86c8264-ee31-4fd3-a988-79bc42548e51\") " pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt"
Jul 07 16:48:02.070835 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.070805    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d86c8264-ee31-4fd3-a988-79bc42548e51-audit-log\") pod \"metrics-server-78665f68c4-cj9jt\" (UID: \"d86c8264-ee31-4fd3-a988-79bc42548e51\") " pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt"
Jul 07 16:48:02.070936 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.070894    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86c8264-ee31-4fd3-a988-79bc42548e51-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78665f68c4-cj9jt\" (UID: \"d86c8264-ee31-4fd3-a988-79bc42548e51\") " pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt"
Jul 07 16:48:02.071642 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.071585    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d86c8264-ee31-4fd3-a988-79bc42548e51-metrics-server-audit-profiles\") pod \"metrics-server-78665f68c4-cj9jt\" (UID: \"d86c8264-ee31-4fd3-a988-79bc42548e51\") " pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt"
Jul 07 16:48:02.072665 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.072634    2583 generic.go:356] "Generic (PLEG): container finished" podID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerID="d21b74688cce656d467239b6d3717e6b1c6d4d98f8dfb7dbbf2716a092310f94" exitCode=0
Jul 07 16:48:02.073437 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.073416    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d86c8264-ee31-4fd3-a988-79bc42548e51-client-ca-bundle\") pod \"metrics-server-78665f68c4-cj9jt\" (UID: \"d86c8264-ee31-4fd3-a988-79bc42548e51\") " pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt"
Jul 07 16:48:02.073545 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.073461    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c216e6fb-0687-4a74-b41b-982f4ef8902b","Type":"ContainerDied","Data":"d21b74688cce656d467239b6d3717e6b1c6d4d98f8dfb7dbbf2716a092310f94"}
Jul 07 16:48:02.073611 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.073583    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d86c8264-ee31-4fd3-a988-79bc42548e51-secret-metrics-server-tls\") pod \"metrics-server-78665f68c4-cj9jt\" (UID: \"d86c8264-ee31-4fd3-a988-79bc42548e51\") " pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt"
Jul 07 16:48:02.073819 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.073798    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d86c8264-ee31-4fd3-a988-79bc42548e51-secret-metrics-server-client-certs\") pod \"metrics-server-78665f68c4-cj9jt\" (UID: \"d86c8264-ee31-4fd3-a988-79bc42548e51\") " pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt"
Jul 07 16:48:02.088003 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.087975    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkgxx\" (UniqueName: \"kubernetes.io/projected/d86c8264-ee31-4fd3-a988-79bc42548e51-kube-api-access-qkgxx\") pod \"metrics-server-78665f68c4-cj9jt\" (UID: \"d86c8264-ee31-4fd3-a988-79bc42548e51\") " pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt"
Jul 07 16:48:02.102739 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.102662    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dthp7" podStartSLOduration=4.114404679 podStartE2EDuration="5.102644813s" totalImagesPullingTime="905.216578ms" totalInitContainerRuntime="83.023556ms" isStatefulPod=true podCreationTimestamp="2026-07-07 16:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:02.101274095 +0000 UTC m=+75.114283915" watchObservedRunningTime="2026-07-07 16:48:02.102644813 +0000 UTC m=+75.115654633"
Jul 07 16:48:02.156863 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.156832    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt"
Jul 07 16:48:02.280448 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.280418    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-5kgbd"]
Jul 07 16:48:02.283655 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.283632    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5kgbd"
Jul 07 16:48:02.287152 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.287114    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 07 16:48:02.291393 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.287465    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-cvhts\""
Jul 07 16:48:02.295415 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.295378    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-5kgbd"]
Jul 07 16:48:02.315158 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.315092    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-78665f68c4-cj9jt"]
Jul 07 16:48:02.320029 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:48:02.320000    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86c8264_ee31_4fd3_a988_79bc42548e51.slice/crio-dde783e793a3ab6fea966b66693c12ae6a06557371dcb76cec62c1c7b073d140 WatchSource:0}: Error finding container dde783e793a3ab6fea966b66693c12ae6a06557371dcb76cec62c1c7b073d140: Status 404 returned error can't find the container with id dde783e793a3ab6fea966b66693c12ae6a06557371dcb76cec62c1c7b073d140
Jul 07 16:48:02.373776 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.373688    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsvwv\" (UniqueName: \"kubernetes.io/projected/2f3a1ce1-eb1e-43e3-a0e0-a7f28c63e227-kube-api-access-dsvwv\") pod \"monitoring-plugin-d5dcff7cf-5kgbd\" (UID: \"2f3a1ce1-eb1e-43e3-a0e0-a7f28c63e227\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5kgbd"
Jul 07 16:48:02.373776 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.373732    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2f3a1ce1-eb1e-43e3-a0e0-a7f28c63e227-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-5kgbd\" (UID: \"2f3a1ce1-eb1e-43e3-a0e0-a7f28c63e227\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5kgbd"
Jul 07 16:48:02.474316 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.474282    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsvwv\" (UniqueName: \"kubernetes.io/projected/2f3a1ce1-eb1e-43e3-a0e0-a7f28c63e227-kube-api-access-dsvwv\") pod \"monitoring-plugin-d5dcff7cf-5kgbd\" (UID: \"2f3a1ce1-eb1e-43e3-a0e0-a7f28c63e227\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5kgbd"
Jul 07 16:48:02.474316 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.474319    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2f3a1ce1-eb1e-43e3-a0e0-a7f28c63e227-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-5kgbd\" (UID: \"2f3a1ce1-eb1e-43e3-a0e0-a7f28c63e227\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5kgbd"
Jul 07 16:48:02.477383 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.477353    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2f3a1ce1-eb1e-43e3-a0e0-a7f28c63e227-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-5kgbd\" (UID: \"2f3a1ce1-eb1e-43e3-a0e0-a7f28c63e227\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5kgbd"
Jul 07 16:48:02.487633 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.487596    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsvwv\" (UniqueName: \"kubernetes.io/projected/2f3a1ce1-eb1e-43e3-a0e0-a7f28c63e227-kube-api-access-dsvwv\") pod \"monitoring-plugin-d5dcff7cf-5kgbd\" (UID: \"2f3a1ce1-eb1e-43e3-a0e0-a7f28c63e227\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5kgbd"
Jul 07 16:48:02.600366 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.600314    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5kgbd"
Jul 07 16:48:02.755383 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:02.755356    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-5kgbd"]
Jul 07 16:48:02.758200 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:48:02.758172    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f3a1ce1_eb1e_43e3_a0e0_a7f28c63e227.slice/crio-8e25dd8609599b187cd353787f7b23d0ddcb58c4c99cbdcf12c7c1458c34428b WatchSource:0}: Error finding container 8e25dd8609599b187cd353787f7b23d0ddcb58c4c99cbdcf12c7c1458c34428b: Status 404 returned error can't find the container with id 8e25dd8609599b187cd353787f7b23d0ddcb58c4c99cbdcf12c7c1458c34428b
Jul 07 16:48:03.078356 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.078318    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt" event={"ID":"d86c8264-ee31-4fd3-a988-79bc42548e51","Type":"ContainerStarted","Data":"dde783e793a3ab6fea966b66693c12ae6a06557371dcb76cec62c1c7b073d140"}
Jul 07 16:48:03.080246 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.080208    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5kgbd" event={"ID":"2f3a1ce1-eb1e-43e3-a0e0-a7f28c63e227","Type":"ContainerStarted","Data":"8e25dd8609599b187cd353787f7b23d0ddcb58c4c99cbdcf12c7c1458c34428b"}
Jul 07 16:48:03.806401 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.806360    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 16:48:03.810521 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.810476    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.821151 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.821106    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 07 16:48:03.821323 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.821302    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 07 16:48:03.821386 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.821302    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-248q1c10rg6lp\""
Jul 07 16:48:03.821429 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.821306    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 07 16:48:03.821646 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.821631    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 07 16:48:03.824622 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.824468    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 07 16:48:03.824622 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.824502    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5w2mx\""
Jul 07 16:48:03.824740 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.824716    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 07 16:48:03.825025 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.824882    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 07 16:48:03.827241 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.826829    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 07 16:48:03.827241 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.826949    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 07 16:48:03.827241 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.827018    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 07 16:48:03.827241 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.827114    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 07 16:48:03.828333 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.828311    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 07 16:48:03.828863 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.828836    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 07 16:48:03.835284 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.835263    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 07 16:48:03.838821 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.838784    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 16:48:03.889621 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.889570    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.889829 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.889633    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.889829 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.889684    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.889829 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.889721    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.889829 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.889746    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwtz6\" (UniqueName: \"kubernetes.io/projected/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-kube-api-access-fwtz6\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.889829 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.889769    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.889829 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.889794    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.889829 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.889825    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-config-out\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.890247 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.889849    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.890247 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.889894    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.890247 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.889941    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.890247 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.889967    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.890247 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.889988    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-web-config\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.890247 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.890015    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.890247 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.890045    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.890247 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.890074    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.890247 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.890115    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.890247 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.890166    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.890247 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.890201    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-config\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.890247 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.890240    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.991691 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.991243    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.991691 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.991292    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.991691 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.991323    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.991691 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.991348    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.991691 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.991371    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.991691 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.991393    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-config\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.991691 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.991428    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.991691 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.991457    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.991691 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.991479    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.991691 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.991520    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.991691 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.991551    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.991691 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.991571    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwtz6\" (UniqueName: \"kubernetes.io/projected/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-kube-api-access-fwtz6\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.991691 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.991594    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.991691 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.991626    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.991691 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.991656    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-config-out\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.991691 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.991680    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.995878 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.991725    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.995878 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.991774    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.995878 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.991802    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.995878 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.991822    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-web-config\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.995878 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.993916    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.995878 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.995291    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.995878 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.995874    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.997778 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.996576    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.997778 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.997175    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.997778 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.997201    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-config\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.998186 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.997947    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.998186 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.998023    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.999245 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.998612    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.999245 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.998990    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.999245 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.999106    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.999776 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.999570    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:03.999969 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:03.999946    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:04.000338 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:04.000297    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-web-config\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:04.001095 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:04.001070    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:04.002165 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:04.002119    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:04.002563 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:04.002542    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:04.002903 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:04.002875    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-config-out\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:04.003470 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:04.003442    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:04.012332 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:04.012274    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwtz6\" (UniqueName: \"kubernetes.io/projected/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-kube-api-access-fwtz6\") pod \"prometheus-k8s-0\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:04.128418 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:04.128325    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:04.703488 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:04.703411    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 16:48:04.706532 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:48:04.706440    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b5c233_4ae7_4e77_95f1_d195d5bafb4b.slice/crio-906fe9a93e36093ecc44b728bd0a1cbc514d15b210c7b5f16d1043d5f3f07d87 WatchSource:0}: Error finding container 906fe9a93e36093ecc44b728bd0a1cbc514d15b210c7b5f16d1043d5f3f07d87: Status 404 returned error can't find the container with id 906fe9a93e36093ecc44b728bd0a1cbc514d15b210c7b5f16d1043d5f3f07d87
Jul 07 16:48:05.094512 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:05.092911    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c216e6fb-0687-4a74-b41b-982f4ef8902b","Type":"ContainerStarted","Data":"04ed6d9438650ac71c48742f07b40dbebc795183a86d0a3b735272e2f7a8cbae"}
Jul 07 16:48:05.094512 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:05.092993    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c216e6fb-0687-4a74-b41b-982f4ef8902b","Type":"ContainerStarted","Data":"eefcb6946709584332616c15da45bfccc88185c4fbe8e5e93f0c8ce5bde1484f"}
Jul 07 16:48:05.094512 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:05.093030    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c216e6fb-0687-4a74-b41b-982f4ef8902b","Type":"ContainerStarted","Data":"69a95ec0fe2a593731f4b85698e20b3d22d293fbfd5af2c358464ae29546bb94"}
Jul 07 16:48:05.094876 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:05.094852    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"09b5c233-4ae7-4e77-95f1-d195d5bafb4b","Type":"ContainerStarted","Data":"906fe9a93e36093ecc44b728bd0a1cbc514d15b210c7b5f16d1043d5f3f07d87"}
Jul 07 16:48:05.096817 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:05.096792    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt" event={"ID":"d86c8264-ee31-4fd3-a988-79bc42548e51","Type":"ContainerStarted","Data":"4d6a2a8229af85ba379a857cf299e7f3d2dc0c0486636c0cb4f835f7de0aa209"}
Jul 07 16:48:05.098638 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:05.098617    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv" event={"ID":"63a50a99-77c3-477d-9e21-cd1066fb698d","Type":"ContainerStarted","Data":"5496a404e0ed902844e771cb39bf615e016a2b838137a7290211fe8182847e54"}
Jul 07 16:48:05.098714 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:05.098666    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv" event={"ID":"63a50a99-77c3-477d-9e21-cd1066fb698d","Type":"ContainerStarted","Data":"8cf052cbcadbace6fc39665b573f707ba98cd4ef3555119480253d6e7a7e6cfa"}
Jul 07 16:48:05.098714 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:05.098693    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv" event={"ID":"63a50a99-77c3-477d-9e21-cd1066fb698d","Type":"ContainerStarted","Data":"dade90084892dd0fd40f1ecb2554e9b6964ec9f434cec0e675a2c10b8c69d4c3"}
Jul 07 16:48:05.121627 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:05.121572    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt" podStartSLOduration=1.936952472 podStartE2EDuration="4.121554304s" totalImagesPullingTime="2.184601832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:05.120402515 +0000 UTC m=+78.133412343" watchObservedRunningTime="2026-07-07 16:48:05.121554304 +0000 UTC m=+78.134564100"
Jul 07 16:48:06.106761 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:06.106718    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c216e6fb-0687-4a74-b41b-982f4ef8902b","Type":"ContainerStarted","Data":"6dd5e2f3b955d347c9f75ab4e34ad5b4564b4a08ee03ba911bfb7a053b66b20a"}
Jul 07 16:48:06.107207 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:06.106832    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c216e6fb-0687-4a74-b41b-982f4ef8902b","Type":"ContainerStarted","Data":"2c637c79894416e8196ccfda38091016e65128fd51f124a7fad9e714be8458f1"}
Jul 07 16:48:06.107207 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:06.106871    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c216e6fb-0687-4a74-b41b-982f4ef8902b","Type":"ContainerStarted","Data":"f7b535b1e8a7f039dcc0ed619b1ba98ff2b00837aa1f20bfdf7181293a3a9272"}
Jul 07 16:48:06.108404 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:06.108382    2583 generic.go:356] "Generic (PLEG): container finished" podID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerID="cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c" exitCode=0
Jul 07 16:48:06.108519 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:06.108416    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"09b5c233-4ae7-4e77-95f1-d195d5bafb4b","Type":"ContainerDied","Data":"cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c"}
Jul 07 16:48:06.111689 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:06.111658    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv" event={"ID":"63a50a99-77c3-477d-9e21-cd1066fb698d","Type":"ContainerStarted","Data":"6e4507a241058a4591e869d9dcaf7f4932d97b6892d4a07bb34085956616fd1e"}
Jul 07 16:48:06.111762 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:06.111716    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv" event={"ID":"63a50a99-77c3-477d-9e21-cd1066fb698d","Type":"ContainerStarted","Data":"67d5c25dee4a095ead25b2940b355408c437212bf757c2f340fea777eddf6c2e"}
Jul 07 16:48:06.111762 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:06.111752    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:06.111838 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:06.111786    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv" event={"ID":"63a50a99-77c3-477d-9e21-cd1066fb698d","Type":"ContainerStarted","Data":"bc3ce83b50296a79166a21a8deb46a6eee169c1d8eb356f945fa041b5b4a02fc"}
Jul 07 16:48:06.113216 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:06.113193    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5kgbd" event={"ID":"2f3a1ce1-eb1e-43e3-a0e0-a7f28c63e227","Type":"ContainerStarted","Data":"d537c1b600cea446c132bb9eb1e9436d1e9692a392683d1962aa203e3c3426f9"}
Jul 07 16:48:06.139290 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:06.139238    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.9479246630000002 podStartE2EDuration="8.139216688s" totalImagesPullingTime="4.16588068s" totalInitContainerRuntime="25.411345ms" isStatefulPod=false podCreationTimestamp="2026-07-07 16:47:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:06.13766371 +0000 UTC m=+79.150673529" watchObservedRunningTime="2026-07-07 16:48:06.139216688 +0000 UTC m=+79.152226507"
Jul 07 16:48:06.164726 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:06.164666    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv" podStartSLOduration=2.197457519 podStartE2EDuration="6.164642312s" totalImagesPullingTime="3.967184793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:06.162777031 +0000 UTC m=+79.175786863" watchObservedRunningTime="2026-07-07 16:48:06.164642312 +0000 UTC m=+79.177652133"
Jul 07 16:48:06.235607 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:06.235553    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5kgbd" podStartSLOduration=1.835647961 podStartE2EDuration="4.235537172s" totalImagesPullingTime="2.399889211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:06.235424194 +0000 UTC m=+79.248434013" watchObservedRunningTime="2026-07-07 16:48:06.235537172 +0000 UTC m=+79.248546990"
Jul 07 16:48:07.118677 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:07.118502    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5kgbd"
Jul 07 16:48:07.123795 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:07.123770    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5kgbd"
Jul 07 16:48:10.135398 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:10.135365    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"09b5c233-4ae7-4e77-95f1-d195d5bafb4b","Type":"ContainerStarted","Data":"df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61"}
Jul 07 16:48:10.135767 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:10.135428    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"09b5c233-4ae7-4e77-95f1-d195d5bafb4b","Type":"ContainerStarted","Data":"d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f"}
Jul 07 16:48:10.135767 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:10.135462    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"09b5c233-4ae7-4e77-95f1-d195d5bafb4b","Type":"ContainerStarted","Data":"577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965"}
Jul 07 16:48:10.135767 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:10.135487    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"09b5c233-4ae7-4e77-95f1-d195d5bafb4b","Type":"ContainerStarted","Data":"ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f"}
Jul 07 16:48:10.135767 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:10.135511    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"09b5c233-4ae7-4e77-95f1-d195d5bafb4b","Type":"ContainerStarted","Data":"ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc"}
Jul 07 16:48:10.135767 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:10.135532    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"09b5c233-4ae7-4e77-95f1-d195d5bafb4b","Type":"ContainerStarted","Data":"5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5"}
Jul 07 16:48:10.181579 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:10.181521    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.00054447 podStartE2EDuration="7.181504142s" totalImagesPullingTime="3.157649284s" totalInitContainerRuntime="23.310388ms" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:10.180229395 +0000 UTC m=+83.193239214" watchObservedRunningTime="2026-07-07 16:48:10.181504142 +0000 UTC m=+83.194513954"
Jul 07 16:48:12.123808 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:12.123780    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5bb46bd4d-s5qzv"
Jul 07 16:48:14.128927 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:14.128886    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:22.157029 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:22.156993    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt"
Jul 07 16:48:22.157508 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:22.157076    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt"
Jul 07 16:48:30.049101 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:30.049070    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-l7frr"
Jul 07 16:48:42.163379 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:42.163351    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt"
Jul 07 16:48:42.167398 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:48:42.167370    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-78665f68c4-cj9jt"
Jul 07 16:49:04.129396 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:04.129349    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:04.149505 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:04.149473    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:04.374601 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:04.374574    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:17.997747 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:17.997711    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 16:49:17.998767 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:17.998711    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerName="alertmanager" containerID="cri-o://69a95ec0fe2a593731f4b85698e20b3d22d293fbfd5af2c358464ae29546bb94" gracePeriod=120
Jul 07 16:49:17.999370 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:17.998916    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerName="kube-rbac-proxy" containerID="cri-o://f7b535b1e8a7f039dcc0ed619b1ba98ff2b00837aa1f20bfdf7181293a3a9272" gracePeriod=120
Jul 07 16:49:17.999535 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:17.999049    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerName="prom-label-proxy" containerID="cri-o://6dd5e2f3b955d347c9f75ab4e34ad5b4564b4a08ee03ba911bfb7a053b66b20a" gracePeriod=120
Jul 07 16:49:17.999657 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:17.999068    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerName="kube-rbac-proxy-metric" containerID="cri-o://2c637c79894416e8196ccfda38091016e65128fd51f124a7fad9e714be8458f1" gracePeriod=120
Jul 07 16:49:17.999786 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:17.999083    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerName="config-reloader" containerID="cri-o://eefcb6946709584332616c15da45bfccc88185c4fbe8e5e93f0c8ce5bde1484f" gracePeriod=120
Jul 07 16:49:17.999894 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:17.999098    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerName="kube-rbac-proxy-web" containerID="cri-o://04ed6d9438650ac71c48742f07b40dbebc795183a86d0a3b735272e2f7a8cbae" gracePeriod=120
Jul 07 16:49:18.414462 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:18.414378    2583 generic.go:356] "Generic (PLEG): container finished" podID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerID="6dd5e2f3b955d347c9f75ab4e34ad5b4564b4a08ee03ba911bfb7a053b66b20a" exitCode=0
Jul 07 16:49:18.414462 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:18.414403    2583 generic.go:356] "Generic (PLEG): container finished" podID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerID="f7b535b1e8a7f039dcc0ed619b1ba98ff2b00837aa1f20bfdf7181293a3a9272" exitCode=0
Jul 07 16:49:18.414462 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:18.414411    2583 generic.go:356] "Generic (PLEG): container finished" podID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerID="eefcb6946709584332616c15da45bfccc88185c4fbe8e5e93f0c8ce5bde1484f" exitCode=0
Jul 07 16:49:18.414462 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:18.414416    2583 generic.go:356] "Generic (PLEG): container finished" podID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerID="69a95ec0fe2a593731f4b85698e20b3d22d293fbfd5af2c358464ae29546bb94" exitCode=0
Jul 07 16:49:18.414730 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:18.414451    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c216e6fb-0687-4a74-b41b-982f4ef8902b","Type":"ContainerDied","Data":"6dd5e2f3b955d347c9f75ab4e34ad5b4564b4a08ee03ba911bfb7a053b66b20a"}
Jul 07 16:49:18.414730 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:18.414531    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c216e6fb-0687-4a74-b41b-982f4ef8902b","Type":"ContainerDied","Data":"f7b535b1e8a7f039dcc0ed619b1ba98ff2b00837aa1f20bfdf7181293a3a9272"}
Jul 07 16:49:18.414730 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:18.414570    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c216e6fb-0687-4a74-b41b-982f4ef8902b","Type":"ContainerDied","Data":"eefcb6946709584332616c15da45bfccc88185c4fbe8e5e93f0c8ce5bde1484f"}
Jul 07 16:49:18.414730 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:18.414606    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c216e6fb-0687-4a74-b41b-982f4ef8902b","Type":"ContainerDied","Data":"69a95ec0fe2a593731f4b85698e20b3d22d293fbfd5af2c358464ae29546bb94"}
Jul 07 16:49:19.254655 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.254625    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.256550 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.256532    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-config-volume\") pod \"c216e6fb-0687-4a74-b41b-982f4ef8902b\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") "
Jul 07 16:49:19.256654 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.256562    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-secret-alertmanager-main-tls\") pod \"c216e6fb-0687-4a74-b41b-982f4ef8902b\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") "
Jul 07 16:49:19.256654 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.256592    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c216e6fb-0687-4a74-b41b-982f4ef8902b-tls-assets\") pod \"c216e6fb-0687-4a74-b41b-982f4ef8902b\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") "
Jul 07 16:49:19.256654 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.256609    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-cluster-tls-config\") pod \"c216e6fb-0687-4a74-b41b-982f4ef8902b\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") "
Jul 07 16:49:19.256809 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.256731    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c216e6fb-0687-4a74-b41b-982f4ef8902b-metrics-client-ca\") pod \"c216e6fb-0687-4a74-b41b-982f4ef8902b\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") "
Jul 07 16:49:19.256809 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.256774    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c216e6fb-0687-4a74-b41b-982f4ef8902b-config-out\") pod \"c216e6fb-0687-4a74-b41b-982f4ef8902b\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") "
Jul 07 16:49:19.256923 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.256806    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5hww\" (UniqueName: \"kubernetes.io/projected/c216e6fb-0687-4a74-b41b-982f4ef8902b-kube-api-access-l5hww\") pod \"c216e6fb-0687-4a74-b41b-982f4ef8902b\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") "
Jul 07 16:49:19.256923 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.256834    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-secret-alertmanager-kube-rbac-proxy\") pod \"c216e6fb-0687-4a74-b41b-982f4ef8902b\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") "
Jul 07 16:49:19.256923 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.256879    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-web-config\") pod \"c216e6fb-0687-4a74-b41b-982f4ef8902b\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") "
Jul 07 16:49:19.256923 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.256919    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c216e6fb-0687-4a74-b41b-982f4ef8902b-alertmanager-trusted-ca-bundle\") pod \"c216e6fb-0687-4a74-b41b-982f4ef8902b\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") "
Jul 07 16:49:19.257107 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.256961    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c216e6fb-0687-4a74-b41b-982f4ef8902b\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") "
Jul 07 16:49:19.257107 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.256989    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c216e6fb-0687-4a74-b41b-982f4ef8902b-alertmanager-main-db\") pod \"c216e6fb-0687-4a74-b41b-982f4ef8902b\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") "
Jul 07 16:49:19.257399 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.257095    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c216e6fb-0687-4a74-b41b-982f4ef8902b-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "c216e6fb-0687-4a74-b41b-982f4ef8902b" (UID: "c216e6fb-0687-4a74-b41b-982f4ef8902b"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:49:19.257472 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.257454    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c216e6fb-0687-4a74-b41b-982f4ef8902b-metrics-client-ca\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:19.257760 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.257690    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c216e6fb-0687-4a74-b41b-982f4ef8902b-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "c216e6fb-0687-4a74-b41b-982f4ef8902b" (UID: "c216e6fb-0687-4a74-b41b-982f4ef8902b"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:49:19.257760 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.257747    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c216e6fb-0687-4a74-b41b-982f4ef8902b-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "c216e6fb-0687-4a74-b41b-982f4ef8902b" (UID: "c216e6fb-0687-4a74-b41b-982f4ef8902b"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:49:19.260013 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.259983    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c216e6fb-0687-4a74-b41b-982f4ef8902b-config-out" (OuterVolumeSpecName: "config-out") pod "c216e6fb-0687-4a74-b41b-982f4ef8902b" (UID: "c216e6fb-0687-4a74-b41b-982f4ef8902b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:49:19.260374 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.260350    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c216e6fb-0687-4a74-b41b-982f4ef8902b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c216e6fb-0687-4a74-b41b-982f4ef8902b" (UID: "c216e6fb-0687-4a74-b41b-982f4ef8902b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:49:19.260537 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.260421    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "c216e6fb-0687-4a74-b41b-982f4ef8902b" (UID: "c216e6fb-0687-4a74-b41b-982f4ef8902b"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:49:19.260693 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.260664    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-config-volume" (OuterVolumeSpecName: "config-volume") pod "c216e6fb-0687-4a74-b41b-982f4ef8902b" (UID: "c216e6fb-0687-4a74-b41b-982f4ef8902b"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:49:19.260831 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.260796    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "c216e6fb-0687-4a74-b41b-982f4ef8902b" (UID: "c216e6fb-0687-4a74-b41b-982f4ef8902b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:49:19.261063 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.261040    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c216e6fb-0687-4a74-b41b-982f4ef8902b-kube-api-access-l5hww" (OuterVolumeSpecName: "kube-api-access-l5hww") pod "c216e6fb-0687-4a74-b41b-982f4ef8902b" (UID: "c216e6fb-0687-4a74-b41b-982f4ef8902b"). InnerVolumeSpecName "kube-api-access-l5hww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:49:19.261344 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.261319    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "c216e6fb-0687-4a74-b41b-982f4ef8902b" (UID: "c216e6fb-0687-4a74-b41b-982f4ef8902b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:49:19.264722 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.264605    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "c216e6fb-0687-4a74-b41b-982f4ef8902b" (UID: "c216e6fb-0687-4a74-b41b-982f4ef8902b"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:49:19.272162 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.272123    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-web-config" (OuterVolumeSpecName: "web-config") pod "c216e6fb-0687-4a74-b41b-982f4ef8902b" (UID: "c216e6fb-0687-4a74-b41b-982f4ef8902b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:49:19.358543 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.358453    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-secret-alertmanager-kube-rbac-proxy-web\") pod \"c216e6fb-0687-4a74-b41b-982f4ef8902b\" (UID: \"c216e6fb-0687-4a74-b41b-982f4ef8902b\") "
Jul 07 16:49:19.358702 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.358676    2583 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c216e6fb-0687-4a74-b41b-982f4ef8902b-config-out\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:19.358702 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.358694    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5hww\" (UniqueName: \"kubernetes.io/projected/c216e6fb-0687-4a74-b41b-982f4ef8902b-kube-api-access-l5hww\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:19.358829 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.358711    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:19.358829 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.358724    2583 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-web-config\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:19.358829 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.358738    2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c216e6fb-0687-4a74-b41b-982f4ef8902b-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:19.358829 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.358753    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:19.358829 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.358768    2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c216e6fb-0687-4a74-b41b-982f4ef8902b-alertmanager-main-db\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:19.358829 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.358782    2583 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-config-volume\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:19.358829 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.358796    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-secret-alertmanager-main-tls\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:19.358829 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.358811    2583 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c216e6fb-0687-4a74-b41b-982f4ef8902b-tls-assets\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:19.358829 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.358825    2583 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-cluster-tls-config\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:19.360685 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.360656    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "c216e6fb-0687-4a74-b41b-982f4ef8902b" (UID: "c216e6fb-0687-4a74-b41b-982f4ef8902b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:49:19.423729 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.423687    2583 generic.go:356] "Generic (PLEG): container finished" podID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerID="2c637c79894416e8196ccfda38091016e65128fd51f124a7fad9e714be8458f1" exitCode=0
Jul 07 16:49:19.423729 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.423723    2583 generic.go:356] "Generic (PLEG): container finished" podID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerID="04ed6d9438650ac71c48742f07b40dbebc795183a86d0a3b735272e2f7a8cbae" exitCode=0
Jul 07 16:49:19.423970 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.423775    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.423970 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.423771    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c216e6fb-0687-4a74-b41b-982f4ef8902b","Type":"ContainerDied","Data":"2c637c79894416e8196ccfda38091016e65128fd51f124a7fad9e714be8458f1"}
Jul 07 16:49:19.423970 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.423893    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c216e6fb-0687-4a74-b41b-982f4ef8902b","Type":"ContainerDied","Data":"04ed6d9438650ac71c48742f07b40dbebc795183a86d0a3b735272e2f7a8cbae"}
Jul 07 16:49:19.423970 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.423936    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c216e6fb-0687-4a74-b41b-982f4ef8902b","Type":"ContainerDied","Data":"005360d769336839450531a454430744ce7ee918d403d7eeccc3617f72a35d83"}
Jul 07 16:49:19.423970 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.423958    2583 scope.go:117] "RemoveContainer" containerID="6dd5e2f3b955d347c9f75ab4e34ad5b4564b4a08ee03ba911bfb7a053b66b20a"
Jul 07 16:49:19.432317 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.432285    2583 scope.go:117] "RemoveContainer" containerID="2c637c79894416e8196ccfda38091016e65128fd51f124a7fad9e714be8458f1"
Jul 07 16:49:19.439722 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.439704    2583 scope.go:117] "RemoveContainer" containerID="f7b535b1e8a7f039dcc0ed619b1ba98ff2b00837aa1f20bfdf7181293a3a9272"
Jul 07 16:49:19.447031 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.447015    2583 scope.go:117] "RemoveContainer" containerID="04ed6d9438650ac71c48742f07b40dbebc795183a86d0a3b735272e2f7a8cbae"
Jul 07 16:49:19.453902 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.453861    2583 scope.go:117] "RemoveContainer" containerID="eefcb6946709584332616c15da45bfccc88185c4fbe8e5e93f0c8ce5bde1484f"
Jul 07 16:49:19.459631 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.459611    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c216e6fb-0687-4a74-b41b-982f4ef8902b-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:19.460890 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.460873    2583 scope.go:117] "RemoveContainer" containerID="69a95ec0fe2a593731f4b85698e20b3d22d293fbfd5af2c358464ae29546bb94"
Jul 07 16:49:19.468066 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.468046    2583 scope.go:117] "RemoveContainer" containerID="d21b74688cce656d467239b6d3717e6b1c6d4d98f8dfb7dbbf2716a092310f94"
Jul 07 16:49:19.471776 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.471754    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 16:49:19.476872 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.476853    2583 scope.go:117] "RemoveContainer" containerID="6dd5e2f3b955d347c9f75ab4e34ad5b4564b4a08ee03ba911bfb7a053b66b20a"
Jul 07 16:49:19.477185 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:49:19.477164    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dd5e2f3b955d347c9f75ab4e34ad5b4564b4a08ee03ba911bfb7a053b66b20a\": container with ID starting with 6dd5e2f3b955d347c9f75ab4e34ad5b4564b4a08ee03ba911bfb7a053b66b20a not found: ID does not exist" containerID="6dd5e2f3b955d347c9f75ab4e34ad5b4564b4a08ee03ba911bfb7a053b66b20a"
Jul 07 16:49:19.477252 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.477195    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dd5e2f3b955d347c9f75ab4e34ad5b4564b4a08ee03ba911bfb7a053b66b20a"} err="failed to get container status \"6dd5e2f3b955d347c9f75ab4e34ad5b4564b4a08ee03ba911bfb7a053b66b20a\": rpc error: code = NotFound desc = could not find container \"6dd5e2f3b955d347c9f75ab4e34ad5b4564b4a08ee03ba911bfb7a053b66b20a\": container with ID starting with 6dd5e2f3b955d347c9f75ab4e34ad5b4564b4a08ee03ba911bfb7a053b66b20a not found: ID does not exist"
Jul 07 16:49:19.477252 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.477229    2583 scope.go:117] "RemoveContainer" containerID="2c637c79894416e8196ccfda38091016e65128fd51f124a7fad9e714be8458f1"
Jul 07 16:49:19.477496 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:49:19.477480    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c637c79894416e8196ccfda38091016e65128fd51f124a7fad9e714be8458f1\": container with ID starting with 2c637c79894416e8196ccfda38091016e65128fd51f124a7fad9e714be8458f1 not found: ID does not exist" containerID="2c637c79894416e8196ccfda38091016e65128fd51f124a7fad9e714be8458f1"
Jul 07 16:49:19.477536 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.477502    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c637c79894416e8196ccfda38091016e65128fd51f124a7fad9e714be8458f1"} err="failed to get container status \"2c637c79894416e8196ccfda38091016e65128fd51f124a7fad9e714be8458f1\": rpc error: code = NotFound desc = could not find container \"2c637c79894416e8196ccfda38091016e65128fd51f124a7fad9e714be8458f1\": container with ID starting with 2c637c79894416e8196ccfda38091016e65128fd51f124a7fad9e714be8458f1 not found: ID does not exist"
Jul 07 16:49:19.477536 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.477519    2583 scope.go:117] "RemoveContainer" containerID="f7b535b1e8a7f039dcc0ed619b1ba98ff2b00837aa1f20bfdf7181293a3a9272"
Jul 07 16:49:19.477752 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:49:19.477736    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b535b1e8a7f039dcc0ed619b1ba98ff2b00837aa1f20bfdf7181293a3a9272\": container with ID starting with f7b535b1e8a7f039dcc0ed619b1ba98ff2b00837aa1f20bfdf7181293a3a9272 not found: ID does not exist" containerID="f7b535b1e8a7f039dcc0ed619b1ba98ff2b00837aa1f20bfdf7181293a3a9272"
Jul 07 16:49:19.477794 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.477757    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b535b1e8a7f039dcc0ed619b1ba98ff2b00837aa1f20bfdf7181293a3a9272"} err="failed to get container status \"f7b535b1e8a7f039dcc0ed619b1ba98ff2b00837aa1f20bfdf7181293a3a9272\": rpc error: code = NotFound desc = could not find container \"f7b535b1e8a7f039dcc0ed619b1ba98ff2b00837aa1f20bfdf7181293a3a9272\": container with ID starting with f7b535b1e8a7f039dcc0ed619b1ba98ff2b00837aa1f20bfdf7181293a3a9272 not found: ID does not exist"
Jul 07 16:49:19.477794 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.477778    2583 scope.go:117] "RemoveContainer" containerID="04ed6d9438650ac71c48742f07b40dbebc795183a86d0a3b735272e2f7a8cbae"
Jul 07 16:49:19.478008 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:49:19.477993    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04ed6d9438650ac71c48742f07b40dbebc795183a86d0a3b735272e2f7a8cbae\": container with ID starting with 04ed6d9438650ac71c48742f07b40dbebc795183a86d0a3b735272e2f7a8cbae not found: ID does not exist" containerID="04ed6d9438650ac71c48742f07b40dbebc795183a86d0a3b735272e2f7a8cbae"
Jul 07 16:49:19.478048 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.478012    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04ed6d9438650ac71c48742f07b40dbebc795183a86d0a3b735272e2f7a8cbae"} err="failed to get container status \"04ed6d9438650ac71c48742f07b40dbebc795183a86d0a3b735272e2f7a8cbae\": rpc error: code = NotFound desc = could not find container \"04ed6d9438650ac71c48742f07b40dbebc795183a86d0a3b735272e2f7a8cbae\": container with ID starting with 04ed6d9438650ac71c48742f07b40dbebc795183a86d0a3b735272e2f7a8cbae not found: ID does not exist"
Jul 07 16:49:19.478048 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.478035    2583 scope.go:117] "RemoveContainer" containerID="eefcb6946709584332616c15da45bfccc88185c4fbe8e5e93f0c8ce5bde1484f"
Jul 07 16:49:19.478270 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:49:19.478255    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eefcb6946709584332616c15da45bfccc88185c4fbe8e5e93f0c8ce5bde1484f\": container with ID starting with eefcb6946709584332616c15da45bfccc88185c4fbe8e5e93f0c8ce5bde1484f not found: ID does not exist" containerID="eefcb6946709584332616c15da45bfccc88185c4fbe8e5e93f0c8ce5bde1484f"
Jul 07 16:49:19.478332 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.478272    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eefcb6946709584332616c15da45bfccc88185c4fbe8e5e93f0c8ce5bde1484f"} err="failed to get container status \"eefcb6946709584332616c15da45bfccc88185c4fbe8e5e93f0c8ce5bde1484f\": rpc error: code = NotFound desc = could not find container \"eefcb6946709584332616c15da45bfccc88185c4fbe8e5e93f0c8ce5bde1484f\": container with ID starting with eefcb6946709584332616c15da45bfccc88185c4fbe8e5e93f0c8ce5bde1484f not found: ID does not exist"
Jul 07 16:49:19.478332 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.478284    2583 scope.go:117] "RemoveContainer" containerID="69a95ec0fe2a593731f4b85698e20b3d22d293fbfd5af2c358464ae29546bb94"
Jul 07 16:49:19.478507 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:49:19.478488    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69a95ec0fe2a593731f4b85698e20b3d22d293fbfd5af2c358464ae29546bb94\": container with ID starting with 69a95ec0fe2a593731f4b85698e20b3d22d293fbfd5af2c358464ae29546bb94 not found: ID does not exist" containerID="69a95ec0fe2a593731f4b85698e20b3d22d293fbfd5af2c358464ae29546bb94"
Jul 07 16:49:19.478555 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.478510    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69a95ec0fe2a593731f4b85698e20b3d22d293fbfd5af2c358464ae29546bb94"} err="failed to get container status \"69a95ec0fe2a593731f4b85698e20b3d22d293fbfd5af2c358464ae29546bb94\": rpc error: code = NotFound desc = could not find container \"69a95ec0fe2a593731f4b85698e20b3d22d293fbfd5af2c358464ae29546bb94\": container with ID starting with 69a95ec0fe2a593731f4b85698e20b3d22d293fbfd5af2c358464ae29546bb94 not found: ID does not exist"
Jul 07 16:49:19.478555 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.478525    2583 scope.go:117] "RemoveContainer" containerID="d21b74688cce656d467239b6d3717e6b1c6d4d98f8dfb7dbbf2716a092310f94"
Jul 07 16:49:19.478734 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:49:19.478714    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d21b74688cce656d467239b6d3717e6b1c6d4d98f8dfb7dbbf2716a092310f94\": container with ID starting with d21b74688cce656d467239b6d3717e6b1c6d4d98f8dfb7dbbf2716a092310f94 not found: ID does not exist" containerID="d21b74688cce656d467239b6d3717e6b1c6d4d98f8dfb7dbbf2716a092310f94"
Jul 07 16:49:19.478769 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.478739    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d21b74688cce656d467239b6d3717e6b1c6d4d98f8dfb7dbbf2716a092310f94"} err="failed to get container status \"d21b74688cce656d467239b6d3717e6b1c6d4d98f8dfb7dbbf2716a092310f94\": rpc error: code = NotFound desc = could not find container \"d21b74688cce656d467239b6d3717e6b1c6d4d98f8dfb7dbbf2716a092310f94\": container with ID starting with d21b74688cce656d467239b6d3717e6b1c6d4d98f8dfb7dbbf2716a092310f94 not found: ID does not exist"
Jul 07 16:49:19.478769 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.478755    2583 scope.go:117] "RemoveContainer" containerID="6dd5e2f3b955d347c9f75ab4e34ad5b4564b4a08ee03ba911bfb7a053b66b20a"
Jul 07 16:49:19.478999 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.478984    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dd5e2f3b955d347c9f75ab4e34ad5b4564b4a08ee03ba911bfb7a053b66b20a"} err="failed to get container status \"6dd5e2f3b955d347c9f75ab4e34ad5b4564b4a08ee03ba911bfb7a053b66b20a\": rpc error: code = NotFound desc = could not find container \"6dd5e2f3b955d347c9f75ab4e34ad5b4564b4a08ee03ba911bfb7a053b66b20a\": container with ID starting with 6dd5e2f3b955d347c9f75ab4e34ad5b4564b4a08ee03ba911bfb7a053b66b20a not found: ID does not exist"
Jul 07 16:49:19.479040 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.479000    2583 scope.go:117] "RemoveContainer" containerID="2c637c79894416e8196ccfda38091016e65128fd51f124a7fad9e714be8458f1"
Jul 07 16:49:19.479273 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.479253    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c637c79894416e8196ccfda38091016e65128fd51f124a7fad9e714be8458f1"} err="failed to get container status \"2c637c79894416e8196ccfda38091016e65128fd51f124a7fad9e714be8458f1\": rpc error: code = NotFound desc = could not find container \"2c637c79894416e8196ccfda38091016e65128fd51f124a7fad9e714be8458f1\": container with ID starting with 2c637c79894416e8196ccfda38091016e65128fd51f124a7fad9e714be8458f1 not found: ID does not exist"
Jul 07 16:49:19.479332 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.479284    2583 scope.go:117] "RemoveContainer" containerID="f7b535b1e8a7f039dcc0ed619b1ba98ff2b00837aa1f20bfdf7181293a3a9272"
Jul 07 16:49:19.479541 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.479524    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b535b1e8a7f039dcc0ed619b1ba98ff2b00837aa1f20bfdf7181293a3a9272"} err="failed to get container status \"f7b535b1e8a7f039dcc0ed619b1ba98ff2b00837aa1f20bfdf7181293a3a9272\": rpc error: code = NotFound desc = could not find container \"f7b535b1e8a7f039dcc0ed619b1ba98ff2b00837aa1f20bfdf7181293a3a9272\": container with ID starting with f7b535b1e8a7f039dcc0ed619b1ba98ff2b00837aa1f20bfdf7181293a3a9272 not found: ID does not exist"
Jul 07 16:49:19.479583 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.479543    2583 scope.go:117] "RemoveContainer" containerID="04ed6d9438650ac71c48742f07b40dbebc795183a86d0a3b735272e2f7a8cbae"
Jul 07 16:49:19.479739 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.479723    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04ed6d9438650ac71c48742f07b40dbebc795183a86d0a3b735272e2f7a8cbae"} err="failed to get container status \"04ed6d9438650ac71c48742f07b40dbebc795183a86d0a3b735272e2f7a8cbae\": rpc error: code = NotFound desc = could not find container \"04ed6d9438650ac71c48742f07b40dbebc795183a86d0a3b735272e2f7a8cbae\": container with ID starting with 04ed6d9438650ac71c48742f07b40dbebc795183a86d0a3b735272e2f7a8cbae not found: ID does not exist"
Jul 07 16:49:19.479786 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.479738    2583 scope.go:117] "RemoveContainer" containerID="eefcb6946709584332616c15da45bfccc88185c4fbe8e5e93f0c8ce5bde1484f"
Jul 07 16:49:19.479967 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.479949    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eefcb6946709584332616c15da45bfccc88185c4fbe8e5e93f0c8ce5bde1484f"} err="failed to get container status \"eefcb6946709584332616c15da45bfccc88185c4fbe8e5e93f0c8ce5bde1484f\": rpc error: code = NotFound desc = could not find container \"eefcb6946709584332616c15da45bfccc88185c4fbe8e5e93f0c8ce5bde1484f\": container with ID starting with eefcb6946709584332616c15da45bfccc88185c4fbe8e5e93f0c8ce5bde1484f not found: ID does not exist"
Jul 07 16:49:19.480017 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.479968    2583 scope.go:117] "RemoveContainer" containerID="69a95ec0fe2a593731f4b85698e20b3d22d293fbfd5af2c358464ae29546bb94"
Jul 07 16:49:19.480209 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.480193    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69a95ec0fe2a593731f4b85698e20b3d22d293fbfd5af2c358464ae29546bb94"} err="failed to get container status \"69a95ec0fe2a593731f4b85698e20b3d22d293fbfd5af2c358464ae29546bb94\": rpc error: code = NotFound desc = could not find container \"69a95ec0fe2a593731f4b85698e20b3d22d293fbfd5af2c358464ae29546bb94\": container with ID starting with 69a95ec0fe2a593731f4b85698e20b3d22d293fbfd5af2c358464ae29546bb94 not found: ID does not exist"
Jul 07 16:49:19.480209 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.480209    2583 scope.go:117] "RemoveContainer" containerID="d21b74688cce656d467239b6d3717e6b1c6d4d98f8dfb7dbbf2716a092310f94"
Jul 07 16:49:19.480449 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.480430    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d21b74688cce656d467239b6d3717e6b1c6d4d98f8dfb7dbbf2716a092310f94"} err="failed to get container status \"d21b74688cce656d467239b6d3717e6b1c6d4d98f8dfb7dbbf2716a092310f94\": rpc error: code = NotFound desc = could not find container \"d21b74688cce656d467239b6d3717e6b1c6d4d98f8dfb7dbbf2716a092310f94\": container with ID starting with d21b74688cce656d467239b6d3717e6b1c6d4d98f8dfb7dbbf2716a092310f94 not found: ID does not exist"
Jul 07 16:49:19.483888 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.483816    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 16:49:19.527579 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.527546    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 16:49:19.527947 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.527929    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerName="alertmanager"
Jul 07 16:49:19.528040 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.527951    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerName="alertmanager"
Jul 07 16:49:19.528040 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.527967    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerName="kube-rbac-proxy"
Jul 07 16:49:19.528040 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.527975    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerName="kube-rbac-proxy"
Jul 07 16:49:19.528040 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.527986    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerName="prom-label-proxy"
Jul 07 16:49:19.528040 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.527994    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerName="prom-label-proxy"
Jul 07 16:49:19.528040 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.528005    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerName="config-reloader"
Jul 07 16:49:19.528040 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.528015    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerName="config-reloader"
Jul 07 16:49:19.528040 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.528034    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerName="init-config-reloader"
Jul 07 16:49:19.528040 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.528042    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerName="init-config-reloader"
Jul 07 16:49:19.528491 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.528055    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerName="kube-rbac-proxy-metric"
Jul 07 16:49:19.528491 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.528063    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerName="kube-rbac-proxy-metric"
Jul 07 16:49:19.528491 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.528080    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerName="kube-rbac-proxy-web"
Jul 07 16:49:19.528491 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.528089    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerName="kube-rbac-proxy-web"
Jul 07 16:49:19.528491 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.528173    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerName="prom-label-proxy"
Jul 07 16:49:19.528491 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.528187    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerName="config-reloader"
Jul 07 16:49:19.528491 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.528198    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerName="kube-rbac-proxy"
Jul 07 16:49:19.528491 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.528207    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerName="kube-rbac-proxy-metric"
Jul 07 16:49:19.528491 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.528215    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerName="kube-rbac-proxy-web"
Jul 07 16:49:19.528491 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.528225    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c216e6fb-0687-4a74-b41b-982f4ef8902b" containerName="alertmanager"
Jul 07 16:49:19.532338 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.532315    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.535454 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.535422    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 07 16:49:19.535593 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.535497    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 07 16:49:19.535593 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.535519    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mff2l\""
Jul 07 16:49:19.535593 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.535421    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 07 16:49:19.535593 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.535422    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 07 16:49:19.535830 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.535628    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 07 16:49:19.535830 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.535767    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 07 16:49:19.535923 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.535911    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 07 16:49:19.536093 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.536075    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 07 16:49:19.542178 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.542156    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 07 16:49:19.550806 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.550777    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 16:49:19.560375 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.560353    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/118fa8c4-16ad-4526-88dc-57746af1a633-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.560375 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.560379    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/118fa8c4-16ad-4526-88dc-57746af1a633-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.560503 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.560395    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/118fa8c4-16ad-4526-88dc-57746af1a633-tls-assets\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.560503 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.560413    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/118fa8c4-16ad-4526-88dc-57746af1a633-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.560503 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.560429    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/118fa8c4-16ad-4526-88dc-57746af1a633-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.560503 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.560446    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/118fa8c4-16ad-4526-88dc-57746af1a633-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.560503 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.560490    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/118fa8c4-16ad-4526-88dc-57746af1a633-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.560652 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.560558    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmj7w\" (UniqueName: \"kubernetes.io/projected/118fa8c4-16ad-4526-88dc-57746af1a633-kube-api-access-zmj7w\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.560652 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.560595    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/118fa8c4-16ad-4526-88dc-57746af1a633-web-config\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.560652 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.560619    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/118fa8c4-16ad-4526-88dc-57746af1a633-config-volume\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.560652 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.560634    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/118fa8c4-16ad-4526-88dc-57746af1a633-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.560806 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.560657    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/118fa8c4-16ad-4526-88dc-57746af1a633-config-out\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.560806 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.560727    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/118fa8c4-16ad-4526-88dc-57746af1a633-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.587022 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.586991    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c216e6fb-0687-4a74-b41b-982f4ef8902b" path="/var/lib/kubelet/pods/c216e6fb-0687-4a74-b41b-982f4ef8902b/volumes"
Jul 07 16:49:19.661651 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.661575    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/118fa8c4-16ad-4526-88dc-57746af1a633-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.661651 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.661614    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/118fa8c4-16ad-4526-88dc-57746af1a633-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.661651 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.661638    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/118fa8c4-16ad-4526-88dc-57746af1a633-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.661891 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.661664    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/118fa8c4-16ad-4526-88dc-57746af1a633-tls-assets\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.661891 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.661690    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/118fa8c4-16ad-4526-88dc-57746af1a633-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.661891 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.661716    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/118fa8c4-16ad-4526-88dc-57746af1a633-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.661891 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.661744    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/118fa8c4-16ad-4526-88dc-57746af1a633-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.662080 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.661894    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/118fa8c4-16ad-4526-88dc-57746af1a633-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.662080 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.661979    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmj7w\" (UniqueName: \"kubernetes.io/projected/118fa8c4-16ad-4526-88dc-57746af1a633-kube-api-access-zmj7w\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.662080 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.662046    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/118fa8c4-16ad-4526-88dc-57746af1a633-web-config\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.662274 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.662086    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/118fa8c4-16ad-4526-88dc-57746af1a633-config-volume\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.662274 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.662111    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/118fa8c4-16ad-4526-88dc-57746af1a633-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.662274 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.662170    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/118fa8c4-16ad-4526-88dc-57746af1a633-config-out\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.662497 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.662473    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/118fa8c4-16ad-4526-88dc-57746af1a633-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.663235 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.663209    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/118fa8c4-16ad-4526-88dc-57746af1a633-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.663827 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.663796    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/118fa8c4-16ad-4526-88dc-57746af1a633-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.664907 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.664879    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/118fa8c4-16ad-4526-88dc-57746af1a633-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.664907 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.664899    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/118fa8c4-16ad-4526-88dc-57746af1a633-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.665066 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.664939    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/118fa8c4-16ad-4526-88dc-57746af1a633-config-out\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.665066 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.665057    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/118fa8c4-16ad-4526-88dc-57746af1a633-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.665201 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.665071    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/118fa8c4-16ad-4526-88dc-57746af1a633-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.665260 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.665209    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/118fa8c4-16ad-4526-88dc-57746af1a633-config-volume\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.665780 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.665754    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/118fa8c4-16ad-4526-88dc-57746af1a633-tls-assets\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.666271 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.666253    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/118fa8c4-16ad-4526-88dc-57746af1a633-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.666870 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.666853    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/118fa8c4-16ad-4526-88dc-57746af1a633-web-config\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.673402 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.673385    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmj7w\" (UniqueName: \"kubernetes.io/projected/118fa8c4-16ad-4526-88dc-57746af1a633-kube-api-access-zmj7w\") pod \"alertmanager-main-0\" (UID: \"118fa8c4-16ad-4526-88dc-57746af1a633\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.842464 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.842434    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:49:19.985668 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:19.985526    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 16:49:19.989746 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:49:19.989717    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod118fa8c4_16ad_4526_88dc_57746af1a633.slice/crio-078ae5f034efddb25fd2e19952dbbf606d4cc18e06319323b5e5d0c90cc41733 WatchSource:0}: Error finding container 078ae5f034efddb25fd2e19952dbbf606d4cc18e06319323b5e5d0c90cc41733: Status 404 returned error can't find the container with id 078ae5f034efddb25fd2e19952dbbf606d4cc18e06319323b5e5d0c90cc41733
Jul 07 16:49:20.432159 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:20.432109    2583 generic.go:356] "Generic (PLEG): container finished" podID="118fa8c4-16ad-4526-88dc-57746af1a633" containerID="320d286c227e4d5c303ef7ae72f002439a4b3ca94b9f595153e69dce58c77eb1" exitCode=0
Jul 07 16:49:20.432559 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:20.432166    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"118fa8c4-16ad-4526-88dc-57746af1a633","Type":"ContainerDied","Data":"320d286c227e4d5c303ef7ae72f002439a4b3ca94b9f595153e69dce58c77eb1"}
Jul 07 16:49:20.432559 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:20.432233    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"118fa8c4-16ad-4526-88dc-57746af1a633","Type":"ContainerStarted","Data":"078ae5f034efddb25fd2e19952dbbf606d4cc18e06319323b5e5d0c90cc41733"}
Jul 07 16:49:21.439292 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:21.439256    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"118fa8c4-16ad-4526-88dc-57746af1a633","Type":"ContainerStarted","Data":"2773f751d5c57ddbe1ea57a4da78b65cce795066a68aff516c37ac4881f3424c"}
Jul 07 16:49:21.439717 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:21.439334    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"118fa8c4-16ad-4526-88dc-57746af1a633","Type":"ContainerStarted","Data":"08844026327d19ff7bad3f03d8bc0821796879f20e26a4d87e05d19b8440da18"}
Jul 07 16:49:21.439717 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:21.439373    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"118fa8c4-16ad-4526-88dc-57746af1a633","Type":"ContainerStarted","Data":"f90d49ae923885a7f5abc138ba5ccb4e2b2dfd9477492ff7473f70f9763c1148"}
Jul 07 16:49:21.439717 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:21.439414    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"118fa8c4-16ad-4526-88dc-57746af1a633","Type":"ContainerStarted","Data":"838b80b4ed53b7e05ea24a6622041e69eec3b64de18452ee2fbde285f625fc97"}
Jul 07 16:49:21.439717 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:21.439457    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"118fa8c4-16ad-4526-88dc-57746af1a633","Type":"ContainerStarted","Data":"a5dad18eaffb3cf940ea286fcbffe3e71f690429e582b445f4c000c2018da5f4"}
Jul 07 16:49:21.439717 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:21.439496    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"118fa8c4-16ad-4526-88dc-57746af1a633","Type":"ContainerStarted","Data":"9ff2b599dd91acdf9003d9b6fd6d3d2d14a5875df4a5343939d755f40bf2350f"}
Jul 07 16:49:21.472899 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:21.472845    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.458658984 podStartE2EDuration="2.472824211s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.165227ms" isStatefulPod=false podCreationTimestamp="2026-07-07 16:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:49:21.471441628 +0000 UTC m=+154.484451475" watchObservedRunningTime="2026-07-07 16:49:21.472824211 +0000 UTC m=+154.485834093"
Jul 07 16:49:22.038414 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.038367    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7f766df6b9-f899l"]
Jul 07 16:49:22.040982 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.040962    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l"
Jul 07 16:49:22.044973 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.044951    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 07 16:49:22.045333 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.045314    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 07 16:49:22.045442 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.045419    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 07 16:49:22.045442 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.045431    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 07 16:49:22.045549 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.045436    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 07 16:49:22.045549 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.045508    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-qngqn\""
Jul 07 16:49:22.051380 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.051358    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 07 16:49:22.057187 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.057161    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f766df6b9-f899l"]
Jul 07 16:49:22.086888 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.086856    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d2201672-aa8c-4b3b-9c55-49ea72d212db-secret-telemeter-client\") pod \"telemeter-client-7f766df6b9-f899l\" (UID: \"d2201672-aa8c-4b3b-9c55-49ea72d212db\") " pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l"
Jul 07 16:49:22.086888 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.086891    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2201672-aa8c-4b3b-9c55-49ea72d212db-serving-certs-ca-bundle\") pod \"telemeter-client-7f766df6b9-f899l\" (UID: \"d2201672-aa8c-4b3b-9c55-49ea72d212db\") " pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l"
Jul 07 16:49:22.087096 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.086915    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d2201672-aa8c-4b3b-9c55-49ea72d212db-federate-client-tls\") pod \"telemeter-client-7f766df6b9-f899l\" (UID: \"d2201672-aa8c-4b3b-9c55-49ea72d212db\") " pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l"
Jul 07 16:49:22.087096 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.086933    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqjjc\" (UniqueName: \"kubernetes.io/projected/d2201672-aa8c-4b3b-9c55-49ea72d212db-kube-api-access-hqjjc\") pod \"telemeter-client-7f766df6b9-f899l\" (UID: \"d2201672-aa8c-4b3b-9c55-49ea72d212db\") " pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l"
Jul 07 16:49:22.087096 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.086969    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2201672-aa8c-4b3b-9c55-49ea72d212db-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f766df6b9-f899l\" (UID: \"d2201672-aa8c-4b3b-9c55-49ea72d212db\") " pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l"
Jul 07 16:49:22.087096 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.087052    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2201672-aa8c-4b3b-9c55-49ea72d212db-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f766df6b9-f899l\" (UID: \"d2201672-aa8c-4b3b-9c55-49ea72d212db\") " pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l"
Jul 07 16:49:22.087096 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.087087    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d2201672-aa8c-4b3b-9c55-49ea72d212db-telemeter-client-tls\") pod \"telemeter-client-7f766df6b9-f899l\" (UID: \"d2201672-aa8c-4b3b-9c55-49ea72d212db\") " pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l"
Jul 07 16:49:22.087276 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.087106    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2201672-aa8c-4b3b-9c55-49ea72d212db-metrics-client-ca\") pod \"telemeter-client-7f766df6b9-f899l\" (UID: \"d2201672-aa8c-4b3b-9c55-49ea72d212db\") " pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l"
Jul 07 16:49:22.188693 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.188647    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d2201672-aa8c-4b3b-9c55-49ea72d212db-secret-telemeter-client\") pod \"telemeter-client-7f766df6b9-f899l\" (UID: \"d2201672-aa8c-4b3b-9c55-49ea72d212db\") " pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l"
Jul 07 16:49:22.188865 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.188712    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2201672-aa8c-4b3b-9c55-49ea72d212db-serving-certs-ca-bundle\") pod \"telemeter-client-7f766df6b9-f899l\" (UID: \"d2201672-aa8c-4b3b-9c55-49ea72d212db\") " pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l"
Jul 07 16:49:22.188865 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.188760    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d2201672-aa8c-4b3b-9c55-49ea72d212db-federate-client-tls\") pod \"telemeter-client-7f766df6b9-f899l\" (UID: \"d2201672-aa8c-4b3b-9c55-49ea72d212db\") " pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l"
Jul 07 16:49:22.188865 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.188791    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqjjc\" (UniqueName: \"kubernetes.io/projected/d2201672-aa8c-4b3b-9c55-49ea72d212db-kube-api-access-hqjjc\") pod \"telemeter-client-7f766df6b9-f899l\" (UID: \"d2201672-aa8c-4b3b-9c55-49ea72d212db\") " pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l"
Jul 07 16:49:22.188865 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.188826    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2201672-aa8c-4b3b-9c55-49ea72d212db-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f766df6b9-f899l\" (UID: \"d2201672-aa8c-4b3b-9c55-49ea72d212db\") " pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l"
Jul 07 16:49:22.189071 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.188934    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2201672-aa8c-4b3b-9c55-49ea72d212db-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f766df6b9-f899l\" (UID: \"d2201672-aa8c-4b3b-9c55-49ea72d212db\") " pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l"
Jul 07 16:49:22.189071 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.188988    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d2201672-aa8c-4b3b-9c55-49ea72d212db-telemeter-client-tls\") pod \"telemeter-client-7f766df6b9-f899l\" (UID: \"d2201672-aa8c-4b3b-9c55-49ea72d212db\") " pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l"
Jul 07 16:49:22.189071 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.189018    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2201672-aa8c-4b3b-9c55-49ea72d212db-metrics-client-ca\") pod \"telemeter-client-7f766df6b9-f899l\" (UID: \"d2201672-aa8c-4b3b-9c55-49ea72d212db\") " pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l"
Jul 07 16:49:22.189936 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.189901    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2201672-aa8c-4b3b-9c55-49ea72d212db-metrics-client-ca\") pod \"telemeter-client-7f766df6b9-f899l\" (UID: \"d2201672-aa8c-4b3b-9c55-49ea72d212db\") " pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l"
Jul 07 16:49:22.190723 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.190699    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2201672-aa8c-4b3b-9c55-49ea72d212db-serving-certs-ca-bundle\") pod \"telemeter-client-7f766df6b9-f899l\" (UID: \"d2201672-aa8c-4b3b-9c55-49ea72d212db\") " pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l"
Jul 07 16:49:22.191002 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.190978    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2201672-aa8c-4b3b-9c55-49ea72d212db-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f766df6b9-f899l\" (UID: \"d2201672-aa8c-4b3b-9c55-49ea72d212db\") " pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l"
Jul 07 16:49:22.192606 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.192573    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2201672-aa8c-4b3b-9c55-49ea72d212db-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f766df6b9-f899l\" (UID: \"d2201672-aa8c-4b3b-9c55-49ea72d212db\") " pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l"
Jul 07 16:49:22.192691 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.192644    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d2201672-aa8c-4b3b-9c55-49ea72d212db-federate-client-tls\") pod \"telemeter-client-7f766df6b9-f899l\" (UID: \"d2201672-aa8c-4b3b-9c55-49ea72d212db\") " pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l"
Jul 07 16:49:22.192828 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.192799    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d2201672-aa8c-4b3b-9c55-49ea72d212db-secret-telemeter-client\") pod \"telemeter-client-7f766df6b9-f899l\" (UID: \"d2201672-aa8c-4b3b-9c55-49ea72d212db\") " pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l"
Jul 07 16:49:22.192938 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.192837    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d2201672-aa8c-4b3b-9c55-49ea72d212db-telemeter-client-tls\") pod \"telemeter-client-7f766df6b9-f899l\" (UID: \"d2201672-aa8c-4b3b-9c55-49ea72d212db\") " pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l"
Jul 07 16:49:22.200036 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.200010    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqjjc\" (UniqueName: \"kubernetes.io/projected/d2201672-aa8c-4b3b-9c55-49ea72d212db-kube-api-access-hqjjc\") pod \"telemeter-client-7f766df6b9-f899l\" (UID: \"d2201672-aa8c-4b3b-9c55-49ea72d212db\") " pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l"
Jul 07 16:49:22.352845 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.352760    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l"
Jul 07 16:49:22.384047 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.382657    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 16:49:22.384047 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.383010    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerName="prometheus" containerID="cri-o://5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5" gracePeriod=600
Jul 07 16:49:22.384047 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.383428    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61" gracePeriod=600
Jul 07 16:49:22.384047 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.383553    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerName="kube-rbac-proxy" containerID="cri-o://d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f" gracePeriod=600
Jul 07 16:49:22.384047 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.383628    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerName="kube-rbac-proxy-web" containerID="cri-o://577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965" gracePeriod=600
Jul 07 16:49:22.384047 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.383718    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerName="thanos-sidecar" containerID="cri-o://ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f" gracePeriod=600
Jul 07 16:49:22.384047 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.383788    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerName="config-reloader" containerID="cri-o://ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc" gracePeriod=600
Jul 07 16:49:22.530858 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.530825    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f766df6b9-f899l"]
Jul 07 16:49:22.535055 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:49:22.535026    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2201672_aa8c_4b3b_9c55_49ea72d212db.slice/crio-b5d4f5af068bd25cfdc13f328762d1a45a39a666702a6517d6a8f49d25846a69 WatchSource:0}: Error finding container b5d4f5af068bd25cfdc13f328762d1a45a39a666702a6517d6a8f49d25846a69: Status 404 returned error can't find the container with id b5d4f5af068bd25cfdc13f328762d1a45a39a666702a6517d6a8f49d25846a69
Jul 07 16:49:22.641417 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.641391    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:22.694913 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.694880    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-grpc-tls\") pod \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") "
Jul 07 16:49:22.694913 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.694916    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-k8s-rulefiles-0\") pod \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") "
Jul 07 16:49:22.695216 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.694940    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-config\") pod \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") "
Jul 07 16:49:22.695282 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.695239    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-tls-assets\") pod \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") "
Jul 07 16:49:22.695342 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.695288    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-configmap-kubelet-serving-ca-bundle\") pod \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") "
Jul 07 16:49:22.695342 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.695320    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-kube-rbac-proxy\") pod \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") "
Jul 07 16:49:22.695439 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.695364    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-config-out\") pod \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") "
Jul 07 16:49:22.695439 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.695390    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-k8s-rulefiles-2\") pod \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") "
Jul 07 16:49:22.695439 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.695423    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-configmap-metrics-client-ca\") pod \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") "
Jul 07 16:49:22.695580 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.695449    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-k8s-rulefiles-1\") pod \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") "
Jul 07 16:49:22.695580 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.695499    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-k8s-db\") pod \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") "
Jul 07 16:49:22.695580 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.695525    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-prometheus-k8s-tls\") pod \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") "
Jul 07 16:49:22.695580 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.695563    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") "
Jul 07 16:49:22.695790 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.695594    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-web-config\") pod \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") "
Jul 07 16:49:22.695790 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.695629    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-thanos-prometheus-http-client-file\") pod \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") "
Jul 07 16:49:22.695790 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.695666    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-configmap-serving-certs-ca-bundle\") pod \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") "
Jul 07 16:49:22.695790 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.695701    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwtz6\" (UniqueName: \"kubernetes.io/projected/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-kube-api-access-fwtz6\") pod \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") "
Jul 07 16:49:22.695790 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.695727    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") "
Jul 07 16:49:22.695790 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.695754    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-trusted-ca-bundle\") pod \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") "
Jul 07 16:49:22.696086 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.695796    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-metrics-client-certs\") pod \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\" (UID: \"09b5c233-4ae7-4e77-95f1-d195d5bafb4b\") "
Jul 07 16:49:22.696171 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.696104    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "09b5c233-4ae7-4e77-95f1-d195d5bafb4b" (UID: "09b5c233-4ae7-4e77-95f1-d195d5bafb4b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:49:22.697166 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.696414    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "09b5c233-4ae7-4e77-95f1-d195d5bafb4b" (UID: "09b5c233-4ae7-4e77-95f1-d195d5bafb4b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:49:22.697166 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.696568    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "09b5c233-4ae7-4e77-95f1-d195d5bafb4b" (UID: "09b5c233-4ae7-4e77-95f1-d195d5bafb4b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:49:22.697166 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.696884    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "09b5c233-4ae7-4e77-95f1-d195d5bafb4b" (UID: "09b5c233-4ae7-4e77-95f1-d195d5bafb4b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:49:22.698470 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.698006    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "09b5c233-4ae7-4e77-95f1-d195d5bafb4b" (UID: "09b5c233-4ae7-4e77-95f1-d195d5bafb4b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:49:22.698470 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.698391    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "09b5c233-4ae7-4e77-95f1-d195d5bafb4b" (UID: "09b5c233-4ae7-4e77-95f1-d195d5bafb4b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:49:22.701032 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.699687    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-config" (OuterVolumeSpecName: "config") pod "09b5c233-4ae7-4e77-95f1-d195d5bafb4b" (UID: "09b5c233-4ae7-4e77-95f1-d195d5bafb4b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:49:22.701032 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.699938    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "09b5c233-4ae7-4e77-95f1-d195d5bafb4b" (UID: "09b5c233-4ae7-4e77-95f1-d195d5bafb4b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:49:22.701032 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.700781    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "09b5c233-4ae7-4e77-95f1-d195d5bafb4b" (UID: "09b5c233-4ae7-4e77-95f1-d195d5bafb4b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:49:22.701276 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.701102    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "09b5c233-4ae7-4e77-95f1-d195d5bafb4b" (UID: "09b5c233-4ae7-4e77-95f1-d195d5bafb4b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:49:22.701276 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.701196    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-config-out" (OuterVolumeSpecName: "config-out") pod "09b5c233-4ae7-4e77-95f1-d195d5bafb4b" (UID: "09b5c233-4ae7-4e77-95f1-d195d5bafb4b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:49:22.701398 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.701271    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-kube-api-access-fwtz6" (OuterVolumeSpecName: "kube-api-access-fwtz6") pod "09b5c233-4ae7-4e77-95f1-d195d5bafb4b" (UID: "09b5c233-4ae7-4e77-95f1-d195d5bafb4b"). InnerVolumeSpecName "kube-api-access-fwtz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:49:22.701830 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.701775    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "09b5c233-4ae7-4e77-95f1-d195d5bafb4b" (UID: "09b5c233-4ae7-4e77-95f1-d195d5bafb4b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:49:22.701934 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.701830    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "09b5c233-4ae7-4e77-95f1-d195d5bafb4b" (UID: "09b5c233-4ae7-4e77-95f1-d195d5bafb4b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:49:22.702026 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.702005    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "09b5c233-4ae7-4e77-95f1-d195d5bafb4b" (UID: "09b5c233-4ae7-4e77-95f1-d195d5bafb4b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:49:22.702118 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.702094    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "09b5c233-4ae7-4e77-95f1-d195d5bafb4b" (UID: "09b5c233-4ae7-4e77-95f1-d195d5bafb4b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:49:22.702567 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.702534    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "09b5c233-4ae7-4e77-95f1-d195d5bafb4b" (UID: "09b5c233-4ae7-4e77-95f1-d195d5bafb4b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:49:22.703264 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.703220    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "09b5c233-4ae7-4e77-95f1-d195d5bafb4b" (UID: "09b5c233-4ae7-4e77-95f1-d195d5bafb4b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:49:22.703830 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.703786    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "09b5c233-4ae7-4e77-95f1-d195d5bafb4b" (UID: "09b5c233-4ae7-4e77-95f1-d195d5bafb4b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:49:22.715886 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.715845    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-web-config" (OuterVolumeSpecName: "web-config") pod "09b5c233-4ae7-4e77-95f1-d195d5bafb4b" (UID: "09b5c233-4ae7-4e77-95f1-d195d5bafb4b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:49:22.797476 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.797435    2583 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-config\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:22.797476 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.797466    2583 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-tls-assets\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:22.797476 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.797478    2583 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:22.797697 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.797489    2583 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-kube-rbac-proxy\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:22.797697 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.797499    2583 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-config-out\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:22.797697 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.797509    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:22.797697 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.797517    2583 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-configmap-metrics-client-ca\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:22.797697 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.797526    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:22.797697 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.797534    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-k8s-db\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:22.797697 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.797541    2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:22.797697 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.797549    2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:22.797697 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.797558    2583 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-web-config\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:22.797697 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.797567    2583 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:22.797697 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.797575    2583 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:22.797697 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.797583    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwtz6\" (UniqueName: \"kubernetes.io/projected/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-kube-api-access-fwtz6\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:22.797697 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.797591    2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:22.797697 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.797599    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:22.797697 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.797606    2583 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-metrics-client-certs\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:22.797697 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.797614    2583 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-secret-grpc-tls\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:22.797697 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:22.797621    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/09b5c233-4ae7-4e77-95f1-d195d5bafb4b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:49:23.451766 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.451676    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l" event={"ID":"d2201672-aa8c-4b3b-9c55-49ea72d212db","Type":"ContainerStarted","Data":"b5d4f5af068bd25cfdc13f328762d1a45a39a666702a6517d6a8f49d25846a69"}
Jul 07 16:49:23.455054 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.455022    2583 generic.go:356] "Generic (PLEG): container finished" podID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerID="df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61" exitCode=0
Jul 07 16:49:23.455054 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.455052    2583 generic.go:356] "Generic (PLEG): container finished" podID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerID="d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f" exitCode=0
Jul 07 16:49:23.455278 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.455062    2583 generic.go:356] "Generic (PLEG): container finished" podID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerID="577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965" exitCode=0
Jul 07 16:49:23.455278 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.455073    2583 generic.go:356] "Generic (PLEG): container finished" podID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerID="ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f" exitCode=0
Jul 07 16:49:23.455278 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.455080    2583 generic.go:356] "Generic (PLEG): container finished" podID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerID="ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc" exitCode=0
Jul 07 16:49:23.455278 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.455088    2583 generic.go:356] "Generic (PLEG): container finished" podID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerID="5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5" exitCode=0
Jul 07 16:49:23.455278 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.455091    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"09b5c233-4ae7-4e77-95f1-d195d5bafb4b","Type":"ContainerDied","Data":"df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61"}
Jul 07 16:49:23.455278 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.455128    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.455278 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.455194    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"09b5c233-4ae7-4e77-95f1-d195d5bafb4b","Type":"ContainerDied","Data":"d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f"}
Jul 07 16:49:23.455278 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.455241    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"09b5c233-4ae7-4e77-95f1-d195d5bafb4b","Type":"ContainerDied","Data":"577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965"}
Jul 07 16:49:23.455684 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.455285    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"09b5c233-4ae7-4e77-95f1-d195d5bafb4b","Type":"ContainerDied","Data":"ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f"}
Jul 07 16:49:23.455684 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.455266    2583 scope.go:117] "RemoveContainer" containerID="df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61"
Jul 07 16:49:23.455684 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.455324    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"09b5c233-4ae7-4e77-95f1-d195d5bafb4b","Type":"ContainerDied","Data":"ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc"}
Jul 07 16:49:23.455684 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.455366    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"09b5c233-4ae7-4e77-95f1-d195d5bafb4b","Type":"ContainerDied","Data":"5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5"}
Jul 07 16:49:23.455684 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.455390    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"09b5c233-4ae7-4e77-95f1-d195d5bafb4b","Type":"ContainerDied","Data":"906fe9a93e36093ecc44b728bd0a1cbc514d15b210c7b5f16d1043d5f3f07d87"}
Jul 07 16:49:23.465944 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.465922    2583 scope.go:117] "RemoveContainer" containerID="d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f"
Jul 07 16:49:23.474442 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.474425    2583 scope.go:117] "RemoveContainer" containerID="577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965"
Jul 07 16:49:23.482800 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.482777    2583 scope.go:117] "RemoveContainer" containerID="ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f"
Jul 07 16:49:23.488368 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.488340    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 16:49:23.491981 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.491955    2583 scope.go:117] "RemoveContainer" containerID="ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc"
Jul 07 16:49:23.498371 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.498343    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 16:49:23.501300 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.501279    2583 scope.go:117] "RemoveContainer" containerID="5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5"
Jul 07 16:49:23.510932 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.510910    2583 scope.go:117] "RemoveContainer" containerID="cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c"
Jul 07 16:49:23.519281 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.519258    2583 scope.go:117] "RemoveContainer" containerID="df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61"
Jul 07 16:49:23.519590 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:49:23.519561    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61\": container with ID starting with df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61 not found: ID does not exist" containerID="df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61"
Jul 07 16:49:23.519700 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.519599    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61"} err="failed to get container status \"df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61\": rpc error: code = NotFound desc = could not find container \"df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61\": container with ID starting with df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61 not found: ID does not exist"
Jul 07 16:49:23.519700 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.519620    2583 scope.go:117] "RemoveContainer" containerID="d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f"
Jul 07 16:49:23.519964 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:49:23.519935    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f\": container with ID starting with d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f not found: ID does not exist" containerID="d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f"
Jul 07 16:49:23.520060 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.519973    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f"} err="failed to get container status \"d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f\": rpc error: code = NotFound desc = could not find container \"d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f\": container with ID starting with d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f not found: ID does not exist"
Jul 07 16:49:23.520060 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.519999    2583 scope.go:117] "RemoveContainer" containerID="577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965"
Jul 07 16:49:23.520275 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:49:23.520252    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965\": container with ID starting with 577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965 not found: ID does not exist" containerID="577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965"
Jul 07 16:49:23.520357 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.520279    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965"} err="failed to get container status \"577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965\": rpc error: code = NotFound desc = could not find container \"577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965\": container with ID starting with 577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965 not found: ID does not exist"
Jul 07 16:49:23.520357 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.520293    2583 scope.go:117] "RemoveContainer" containerID="ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f"
Jul 07 16:49:23.520522 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:49:23.520506    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f\": container with ID starting with ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f not found: ID does not exist" containerID="ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f"
Jul 07 16:49:23.520563 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.520531    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f"} err="failed to get container status \"ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f\": rpc error: code = NotFound desc = could not find container \"ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f\": container with ID starting with ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f not found: ID does not exist"
Jul 07 16:49:23.520563 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.520543    2583 scope.go:117] "RemoveContainer" containerID="ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc"
Jul 07 16:49:23.520792 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:49:23.520772    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc\": container with ID starting with ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc not found: ID does not exist" containerID="ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc"
Jul 07 16:49:23.520842 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.520798    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc"} err="failed to get container status \"ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc\": rpc error: code = NotFound desc = could not find container \"ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc\": container with ID starting with ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc not found: ID does not exist"
Jul 07 16:49:23.520842 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.520814    2583 scope.go:117] "RemoveContainer" containerID="5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5"
Jul 07 16:49:23.521030 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:49:23.521013    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5\": container with ID starting with 5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5 not found: ID does not exist" containerID="5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5"
Jul 07 16:49:23.521092 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.521038    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5"} err="failed to get container status \"5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5\": rpc error: code = NotFound desc = could not find container \"5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5\": container with ID starting with 5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5 not found: ID does not exist"
Jul 07 16:49:23.521092 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.521058    2583 scope.go:117] "RemoveContainer" containerID="cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c"
Jul 07 16:49:23.521299 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:49:23.521284    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c\": container with ID starting with cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c not found: ID does not exist" containerID="cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c"
Jul 07 16:49:23.521338 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.521303    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c"} err="failed to get container status \"cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c\": rpc error: code = NotFound desc = could not find container \"cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c\": container with ID starting with cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c not found: ID does not exist"
Jul 07 16:49:23.521338 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.521316    2583 scope.go:117] "RemoveContainer" containerID="df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61"
Jul 07 16:49:23.521500 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.521485    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61"} err="failed to get container status \"df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61\": rpc error: code = NotFound desc = could not find container \"df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61\": container with ID starting with df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61 not found: ID does not exist"
Jul 07 16:49:23.521543 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.521500    2583 scope.go:117] "RemoveContainer" containerID="d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f"
Jul 07 16:49:23.521720 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.521702    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f"} err="failed to get container status \"d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f\": rpc error: code = NotFound desc = could not find container \"d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f\": container with ID starting with d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f not found: ID does not exist"
Jul 07 16:49:23.521757 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.521722    2583 scope.go:117] "RemoveContainer" containerID="577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965"
Jul 07 16:49:23.521984 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.521963    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965"} err="failed to get container status \"577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965\": rpc error: code = NotFound desc = could not find container \"577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965\": container with ID starting with 577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965 not found: ID does not exist"
Jul 07 16:49:23.521984 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.521983    2583 scope.go:117] "RemoveContainer" containerID="ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f"
Jul 07 16:49:23.522218 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.522201    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f"} err="failed to get container status \"ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f\": rpc error: code = NotFound desc = could not find container \"ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f\": container with ID starting with ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f not found: ID does not exist"
Jul 07 16:49:23.522266 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.522219    2583 scope.go:117] "RemoveContainer" containerID="ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc"
Jul 07 16:49:23.522439 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.522423    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc"} err="failed to get container status \"ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc\": rpc error: code = NotFound desc = could not find container \"ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc\": container with ID starting with ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc not found: ID does not exist"
Jul 07 16:49:23.522483 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.522441    2583 scope.go:117] "RemoveContainer" containerID="5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5"
Jul 07 16:49:23.522642 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.522626    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5"} err="failed to get container status \"5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5\": rpc error: code = NotFound desc = could not find container \"5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5\": container with ID starting with 5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5 not found: ID does not exist"
Jul 07 16:49:23.522642 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.522640    2583 scope.go:117] "RemoveContainer" containerID="cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c"
Jul 07 16:49:23.522837 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.522821    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c"} err="failed to get container status \"cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c\": rpc error: code = NotFound desc = could not find container \"cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c\": container with ID starting with cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c not found: ID does not exist"
Jul 07 16:49:23.522877 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.522839    2583 scope.go:117] "RemoveContainer" containerID="df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61"
Jul 07 16:49:23.523023 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.523008    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61"} err="failed to get container status \"df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61\": rpc error: code = NotFound desc = could not find container \"df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61\": container with ID starting with df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61 not found: ID does not exist"
Jul 07 16:49:23.523023 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.523022    2583 scope.go:117] "RemoveContainer" containerID="d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f"
Jul 07 16:49:23.523259 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.523240    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f"} err="failed to get container status \"d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f\": rpc error: code = NotFound desc = could not find container \"d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f\": container with ID starting with d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f not found: ID does not exist"
Jul 07 16:49:23.523259 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.523259    2583 scope.go:117] "RemoveContainer" containerID="577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965"
Jul 07 16:49:23.523450 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.523433    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965"} err="failed to get container status \"577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965\": rpc error: code = NotFound desc = could not find container \"577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965\": container with ID starting with 577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965 not found: ID does not exist"
Jul 07 16:49:23.523508 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.523450    2583 scope.go:117] "RemoveContainer" containerID="ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f"
Jul 07 16:49:23.523632 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.523616    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f"} err="failed to get container status \"ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f\": rpc error: code = NotFound desc = could not find container \"ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f\": container with ID starting with ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f not found: ID does not exist"
Jul 07 16:49:23.523682 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.523633    2583 scope.go:117] "RemoveContainer" containerID="ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc"
Jul 07 16:49:23.523803 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.523787    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc"} err="failed to get container status \"ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc\": rpc error: code = NotFound desc = could not find container \"ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc\": container with ID starting with ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc not found: ID does not exist"
Jul 07 16:49:23.523850 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.523802    2583 scope.go:117] "RemoveContainer" containerID="5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5"
Jul 07 16:49:23.524005 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.523989    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5"} err="failed to get container status \"5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5\": rpc error: code = NotFound desc = could not find container \"5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5\": container with ID starting with 5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5 not found: ID does not exist"
Jul 07 16:49:23.524005 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.524004    2583 scope.go:117] "RemoveContainer" containerID="cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c"
Jul 07 16:49:23.524195 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.524179    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c"} err="failed to get container status \"cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c\": rpc error: code = NotFound desc = could not find container \"cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c\": container with ID starting with cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c not found: ID does not exist"
Jul 07 16:49:23.524195 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.524195    2583 scope.go:117] "RemoveContainer" containerID="df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61"
Jul 07 16:49:23.524401 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.524385    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61"} err="failed to get container status \"df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61\": rpc error: code = NotFound desc = could not find container \"df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61\": container with ID starting with df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61 not found: ID does not exist"
Jul 07 16:49:23.524401 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.524401    2583 scope.go:117] "RemoveContainer" containerID="d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f"
Jul 07 16:49:23.524595 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.524578    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f"} err="failed to get container status \"d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f\": rpc error: code = NotFound desc = could not find container \"d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f\": container with ID starting with d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f not found: ID does not exist"
Jul 07 16:49:23.524595 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.524595    2583 scope.go:117] "RemoveContainer" containerID="577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965"
Jul 07 16:49:23.524803 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.524787    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965"} err="failed to get container status \"577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965\": rpc error: code = NotFound desc = could not find container \"577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965\": container with ID starting with 577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965 not found: ID does not exist"
Jul 07 16:49:23.524844 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.524803    2583 scope.go:117] "RemoveContainer" containerID="ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f"
Jul 07 16:49:23.524986 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.524971    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f"} err="failed to get container status \"ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f\": rpc error: code = NotFound desc = could not find container \"ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f\": container with ID starting with ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f not found: ID does not exist"
Jul 07 16:49:23.525025 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.524986    2583 scope.go:117] "RemoveContainer" containerID="ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc"
Jul 07 16:49:23.525195 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.525179    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc"} err="failed to get container status \"ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc\": rpc error: code = NotFound desc = could not find container \"ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc\": container with ID starting with ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc not found: ID does not exist"
Jul 07 16:49:23.525243 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.525196    2583 scope.go:117] "RemoveContainer" containerID="5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5"
Jul 07 16:49:23.525388 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.525363    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5"} err="failed to get container status \"5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5\": rpc error: code = NotFound desc = could not find container \"5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5\": container with ID starting with 5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5 not found: ID does not exist"
Jul 07 16:49:23.525427 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.525390    2583 scope.go:117] "RemoveContainer" containerID="cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c"
Jul 07 16:49:23.525607 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.525592    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c"} err="failed to get container status \"cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c\": rpc error: code = NotFound desc = could not find container \"cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c\": container with ID starting with cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c not found: ID does not exist"
Jul 07 16:49:23.525676 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.525609    2583 scope.go:117] "RemoveContainer" containerID="df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61"
Jul 07 16:49:23.525820 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.525802    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61"} err="failed to get container status \"df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61\": rpc error: code = NotFound desc = could not find container \"df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61\": container with ID starting with df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61 not found: ID does not exist"
Jul 07 16:49:23.525881 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.525820    2583 scope.go:117] "RemoveContainer" containerID="d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f"
Jul 07 16:49:23.526025 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.526009    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f"} err="failed to get container status \"d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f\": rpc error: code = NotFound desc = could not find container \"d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f\": container with ID starting with d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f not found: ID does not exist"
Jul 07 16:49:23.526080 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.526025    2583 scope.go:117] "RemoveContainer" containerID="577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965"
Jul 07 16:49:23.526255 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.526228    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965"} err="failed to get container status \"577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965\": rpc error: code = NotFound desc = could not find container \"577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965\": container with ID starting with 577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965 not found: ID does not exist"
Jul 07 16:49:23.526309 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.526254    2583 scope.go:117] "RemoveContainer" containerID="ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f"
Jul 07 16:49:23.526451 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.526436    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f"} err="failed to get container status \"ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f\": rpc error: code = NotFound desc = could not find container \"ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f\": container with ID starting with ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f not found: ID does not exist"
Jul 07 16:49:23.526496 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.526452    2583 scope.go:117] "RemoveContainer" containerID="ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc"
Jul 07 16:49:23.526635 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.526621    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc"} err="failed to get container status \"ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc\": rpc error: code = NotFound desc = could not find container \"ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc\": container with ID starting with ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc not found: ID does not exist"
Jul 07 16:49:23.526684 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.526636    2583 scope.go:117] "RemoveContainer" containerID="5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5"
Jul 07 16:49:23.526852 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.526829    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5"} err="failed to get container status \"5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5\": rpc error: code = NotFound desc = could not find container \"5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5\": container with ID starting with 5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5 not found: ID does not exist"
Jul 07 16:49:23.526852 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.526848    2583 scope.go:117] "RemoveContainer" containerID="cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c"
Jul 07 16:49:23.527058 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.527041    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c"} err="failed to get container status \"cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c\": rpc error: code = NotFound desc = could not find container \"cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c\": container with ID starting with cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c not found: ID does not exist"
Jul 07 16:49:23.527115 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.527058    2583 scope.go:117] "RemoveContainer" containerID="df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61"
Jul 07 16:49:23.527258 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.527243    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61"} err="failed to get container status \"df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61\": rpc error: code = NotFound desc = could not find container \"df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61\": container with ID starting with df3997d4fb48ed9cebeda7a8ca009ec48b4edf69edef5c86af9ff60fcf659a61 not found: ID does not exist"
Jul 07 16:49:23.527303 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.527258    2583 scope.go:117] "RemoveContainer" containerID="d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f"
Jul 07 16:49:23.527444 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.527431    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f"} err="failed to get container status \"d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f\": rpc error: code = NotFound desc = could not find container \"d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f\": container with ID starting with d97ef0b598cd3309a90289f09200ac7d15434144096470270ddfc151461ae53f not found: ID does not exist"
Jul 07 16:49:23.527487 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.527445    2583 scope.go:117] "RemoveContainer" containerID="577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965"
Jul 07 16:49:23.527637 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.527621    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965"} err="failed to get container status \"577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965\": rpc error: code = NotFound desc = could not find container \"577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965\": container with ID starting with 577b2634cc69d3351d221fbefbdbe1a91f8fe4f11ac2e10e28e6f373dce5f965 not found: ID does not exist"
Jul 07 16:49:23.527637 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.527635    2583 scope.go:117] "RemoveContainer" containerID="ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f"
Jul 07 16:49:23.527831 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.527816    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f"} err="failed to get container status \"ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f\": rpc error: code = NotFound desc = could not find container \"ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f\": container with ID starting with ded4cb51ee5e61b693bc0f82a71a4fb133d4e9a83e3244f4eab8ff90d521947f not found: ID does not exist"
Jul 07 16:49:23.527888 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.527838    2583 scope.go:117] "RemoveContainer" containerID="ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc"
Jul 07 16:49:23.528113 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.528070    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc"} err="failed to get container status \"ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc\": rpc error: code = NotFound desc = could not find container \"ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc\": container with ID starting with ab33963ee910eeece2864283b3ce3badee72ecf926642824a2d4f2b70292eadc not found: ID does not exist"
Jul 07 16:49:23.528113 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.528097    2583 scope.go:117] "RemoveContainer" containerID="5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5"
Jul 07 16:49:23.528824 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.528515    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5"} err="failed to get container status \"5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5\": rpc error: code = NotFound desc = could not find container \"5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5\": container with ID starting with 5eb4f8168e09a0c60da073844297231e978aaa07cf12cbf3179f630dfcb824d5 not found: ID does not exist"
Jul 07 16:49:23.528824 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.528538    2583 scope.go:117] "RemoveContainer" containerID="cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c"
Jul 07 16:49:23.528824 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.528790    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c"} err="failed to get container status \"cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c\": rpc error: code = NotFound desc = could not find container \"cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c\": container with ID starting with cfd7c7d82e03a05b8a831e86892bb50cb516d0fb1a520b62b60cc45433776b2c not found: ID does not exist"
Jul 07 16:49:23.530255 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.530236    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 16:49:23.530540 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.530528    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerName="init-config-reloader"
Jul 07 16:49:23.530585 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.530542    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerName="init-config-reloader"
Jul 07 16:49:23.530585 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.530553    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerName="config-reloader"
Jul 07 16:49:23.530585 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.530559    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerName="config-reloader"
Jul 07 16:49:23.530585 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.530565    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerName="kube-rbac-proxy"
Jul 07 16:49:23.530585 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.530573    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerName="kube-rbac-proxy"
Jul 07 16:49:23.530585 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.530580    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerName="kube-rbac-proxy-thanos"
Jul 07 16:49:23.530585 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.530585    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerName="kube-rbac-proxy-thanos"
Jul 07 16:49:23.530784 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.530603    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerName="prometheus"
Jul 07 16:49:23.530784 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.530608    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerName="prometheus"
Jul 07 16:49:23.530784 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.530615    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerName="kube-rbac-proxy-web"
Jul 07 16:49:23.530784 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.530620    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerName="kube-rbac-proxy-web"
Jul 07 16:49:23.530784 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.530626    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerName="thanos-sidecar"
Jul 07 16:49:23.530784 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.530630    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerName="thanos-sidecar"
Jul 07 16:49:23.530784 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.530677    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerName="kube-rbac-proxy-thanos"
Jul 07 16:49:23.530784 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.530686    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerName="kube-rbac-proxy-web"
Jul 07 16:49:23.530784 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.530692    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerName="prometheus"
Jul 07 16:49:23.530784 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.530697    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerName="thanos-sidecar"
Jul 07 16:49:23.530784 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.530702    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerName="kube-rbac-proxy"
Jul 07 16:49:23.530784 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.530707    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" containerName="config-reloader"
Jul 07 16:49:23.534341 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.534327    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.537506 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.537475    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 07 16:49:23.537506 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.537488    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 07 16:49:23.537506 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.537501    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-248q1c10rg6lp\""
Jul 07 16:49:23.537885 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.537515    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 07 16:49:23.537885 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.537749    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 07 16:49:23.538047 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.538025    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 07 16:49:23.538354 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.538336    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 07 16:49:23.538462 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.538428    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 07 16:49:23.538569 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.538549    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 07 16:49:23.538698 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.538589    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 07 16:49:23.538839 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.538823    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 07 16:49:23.538897 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.538880    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 07 16:49:23.538971 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.538958    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5w2mx\""
Jul 07 16:49:23.539012 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.538976    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 07 16:49:23.542446 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.542425    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 07 16:49:23.546404 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.546384    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 07 16:49:23.557012 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.556988    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 16:49:23.588266 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.588234    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09b5c233-4ae7-4e77-95f1-d195d5bafb4b" path="/var/lib/kubelet/pods/09b5c233-4ae7-4e77-95f1-d195d5bafb4b/volumes"
Jul 07 16:49:23.604657 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.604618    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a237ad78-f074-4073-8732-7f7e4f8e8477-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.604827 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.604673    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a237ad78-f074-4073-8732-7f7e4f8e8477-web-config\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.604827 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.604700    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a237ad78-f074-4073-8732-7f7e4f8e8477-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.604827 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.604748    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a237ad78-f074-4073-8732-7f7e4f8e8477-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.604827 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.604775    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a237ad78-f074-4073-8732-7f7e4f8e8477-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.605043 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.604949    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a237ad78-f074-4073-8732-7f7e4f8e8477-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.605043 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.604995    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a237ad78-f074-4073-8732-7f7e4f8e8477-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.605170 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.605076    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a237ad78-f074-4073-8732-7f7e4f8e8477-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.605170 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.605109    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a237ad78-f074-4073-8732-7f7e4f8e8477-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.605170 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.605155    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a237ad78-f074-4073-8732-7f7e4f8e8477-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.605332 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.605188    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a237ad78-f074-4073-8732-7f7e4f8e8477-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.605332 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.605212    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a237ad78-f074-4073-8732-7f7e4f8e8477-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.605332 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.605235    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp58l\" (UniqueName: \"kubernetes.io/projected/a237ad78-f074-4073-8732-7f7e4f8e8477-kube-api-access-qp58l\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.605332 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.605272    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a237ad78-f074-4073-8732-7f7e4f8e8477-config\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.605332 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.605296    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a237ad78-f074-4073-8732-7f7e4f8e8477-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.605332 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.605316    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a237ad78-f074-4073-8732-7f7e4f8e8477-config-out\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.605618 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.605345    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a237ad78-f074-4073-8732-7f7e4f8e8477-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.605618 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.605386    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a237ad78-f074-4073-8732-7f7e4f8e8477-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.605618 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.605409    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a237ad78-f074-4073-8732-7f7e4f8e8477-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.605618 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.605438    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a237ad78-f074-4073-8732-7f7e4f8e8477-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.706865 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.706764    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a237ad78-f074-4073-8732-7f7e4f8e8477-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.706865 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.706815    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a237ad78-f074-4073-8732-7f7e4f8e8477-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.706865 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.706853    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a237ad78-f074-4073-8732-7f7e4f8e8477-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.707180 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.706881    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a237ad78-f074-4073-8732-7f7e4f8e8477-web-config\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.707180 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.706909    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a237ad78-f074-4073-8732-7f7e4f8e8477-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.707180 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.706946    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a237ad78-f074-4073-8732-7f7e4f8e8477-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.707180 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.706972    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a237ad78-f074-4073-8732-7f7e4f8e8477-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.707180 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.706999    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a237ad78-f074-4073-8732-7f7e4f8e8477-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.707180 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.707027    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a237ad78-f074-4073-8732-7f7e4f8e8477-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.707180 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.707064    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a237ad78-f074-4073-8732-7f7e4f8e8477-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.707180 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.707094    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a237ad78-f074-4073-8732-7f7e4f8e8477-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.707180 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.707126    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a237ad78-f074-4073-8732-7f7e4f8e8477-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.707180 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.707182    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a237ad78-f074-4073-8732-7f7e4f8e8477-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.707757 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.707206    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a237ad78-f074-4073-8732-7f7e4f8e8477-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.707757 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.707228    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp58l\" (UniqueName: \"kubernetes.io/projected/a237ad78-f074-4073-8732-7f7e4f8e8477-kube-api-access-qp58l\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.707757 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.707264    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a237ad78-f074-4073-8732-7f7e4f8e8477-config\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.707757 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.707287    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a237ad78-f074-4073-8732-7f7e4f8e8477-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.707757 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.707312    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a237ad78-f074-4073-8732-7f7e4f8e8477-config-out\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.707757 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.707347    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a237ad78-f074-4073-8732-7f7e4f8e8477-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.707757 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.707377    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a237ad78-f074-4073-8732-7f7e4f8e8477-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.708404 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.708380    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a237ad78-f074-4073-8732-7f7e4f8e8477-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.708446 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.708391    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a237ad78-f074-4073-8732-7f7e4f8e8477-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.708495 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.708468    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a237ad78-f074-4073-8732-7f7e4f8e8477-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.708670 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.708644    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a237ad78-f074-4073-8732-7f7e4f8e8477-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.711172 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.709604    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a237ad78-f074-4073-8732-7f7e4f8e8477-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.711172 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.711009    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a237ad78-f074-4073-8732-7f7e4f8e8477-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.712727 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.711346    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a237ad78-f074-4073-8732-7f7e4f8e8477-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.712727 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.711407    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a237ad78-f074-4073-8732-7f7e4f8e8477-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.712727 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.711478    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a237ad78-f074-4073-8732-7f7e4f8e8477-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.712727 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.712010    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a237ad78-f074-4073-8732-7f7e4f8e8477-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.712727 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.712222    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a237ad78-f074-4073-8732-7f7e4f8e8477-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.713036 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.712971    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a237ad78-f074-4073-8732-7f7e4f8e8477-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.713448 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.713429    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a237ad78-f074-4073-8732-7f7e4f8e8477-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.713878 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.713817    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a237ad78-f074-4073-8732-7f7e4f8e8477-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.713982 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.713951    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a237ad78-f074-4073-8732-7f7e4f8e8477-config-out\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.714663 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.714641    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a237ad78-f074-4073-8732-7f7e4f8e8477-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.715019 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.715000    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a237ad78-f074-4073-8732-7f7e4f8e8477-web-config\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.715084 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.715052    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a237ad78-f074-4073-8732-7f7e4f8e8477-config\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.715447 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.715428    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a237ad78-f074-4073-8732-7f7e4f8e8477-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.720796 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.720776    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp58l\" (UniqueName: \"kubernetes.io/projected/a237ad78-f074-4073-8732-7f7e4f8e8477-kube-api-access-qp58l\") pod \"prometheus-k8s-0\" (UID: \"a237ad78-f074-4073-8732-7f7e4f8e8477\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:23.844725 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:23.844684    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:24.014298 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:24.014246    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 16:49:24.423885 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:49:24.423822    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda237ad78_f074_4073_8732_7f7e4f8e8477.slice/crio-72b6b79103791074ac86df3e12182557edc217c34541bc5df5c4e14f01980fdd WatchSource:0}: Error finding container 72b6b79103791074ac86df3e12182557edc217c34541bc5df5c4e14f01980fdd: Status 404 returned error can't find the container with id 72b6b79103791074ac86df3e12182557edc217c34541bc5df5c4e14f01980fdd
Jul 07 16:49:24.461388 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:24.461362    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a237ad78-f074-4073-8732-7f7e4f8e8477","Type":"ContainerStarted","Data":"72b6b79103791074ac86df3e12182557edc217c34541bc5df5c4e14f01980fdd"}
Jul 07 16:49:25.468130 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:25.468093    2583 generic.go:356] "Generic (PLEG): container finished" podID="a237ad78-f074-4073-8732-7f7e4f8e8477" containerID="aaf43e0a1ceb643dee077003a4e0969d1f04702391597b4e64aaf1d22317dff5" exitCode=0
Jul 07 16:49:25.468601 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:25.468186    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a237ad78-f074-4073-8732-7f7e4f8e8477","Type":"ContainerDied","Data":"aaf43e0a1ceb643dee077003a4e0969d1f04702391597b4e64aaf1d22317dff5"}
Jul 07 16:49:25.470359 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:25.470336    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l" event={"ID":"d2201672-aa8c-4b3b-9c55-49ea72d212db","Type":"ContainerStarted","Data":"3687bdf36afb6d3213ffe24c66b4cdd37d2cac9181ce978b41d03a2d957bdb17"}
Jul 07 16:49:25.470505 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:25.470389    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l" event={"ID":"d2201672-aa8c-4b3b-9c55-49ea72d212db","Type":"ContainerStarted","Data":"c9a91f1e9ab86ec9592945a05b9fe6f665d40e1ac98c64b708a9d5f0c75a59f9"}
Jul 07 16:49:25.470505 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:25.470421    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l" event={"ID":"d2201672-aa8c-4b3b-9c55-49ea72d212db","Type":"ContainerStarted","Data":"9a7d51fd0fafca395abef4f749ffa56888c997019f593a738898e33bd92fa832"}
Jul 07 16:49:25.532932 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:25.532888    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7f766df6b9-f899l" podStartSLOduration=1.6137586069999998 podStartE2EDuration="3.532871401s" totalImagesPullingTime="1.919112794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:49:25.530282472 +0000 UTC m=+158.543292291" watchObservedRunningTime="2026-07-07 16:49:25.532871401 +0000 UTC m=+158.545881219"
Jul 07 16:49:26.479425 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:26.479391    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a237ad78-f074-4073-8732-7f7e4f8e8477","Type":"ContainerStarted","Data":"2efc66cbc254eb453022b0246c207bbcd855f83a3fd88483ad8acf1434629bd6"}
Jul 07 16:49:26.479851 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:26.479461    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a237ad78-f074-4073-8732-7f7e4f8e8477","Type":"ContainerStarted","Data":"40f78bcd859ffc85ab77136f175d5f5b16a1a21280bbbf4d1810c2724ac6d1cf"}
Jul 07 16:49:26.479851 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:26.479491    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a237ad78-f074-4073-8732-7f7e4f8e8477","Type":"ContainerStarted","Data":"a491585a4647bbc5cb23f5a31b56e6fbd9624b0725d054ed3b821ea15d5c9596"}
Jul 07 16:49:26.479851 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:26.479517    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a237ad78-f074-4073-8732-7f7e4f8e8477","Type":"ContainerStarted","Data":"664959d0215fb85692e30943b23725d1c2092c5238e37a44b9d4df83b3169208"}
Jul 07 16:49:26.479851 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:26.479541    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a237ad78-f074-4073-8732-7f7e4f8e8477","Type":"ContainerStarted","Data":"231b2c452c00df9ded89eac66334021b638724ea2e73ed562fd69c80fbeb6220"}
Jul 07 16:49:26.479851 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:26.479565    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a237ad78-f074-4073-8732-7f7e4f8e8477","Type":"ContainerStarted","Data":"d1a78724acc57ef0908f7521eba48c517a14ff32cbbae2a144e0a8a55b92eb15"}
Jul 07 16:49:26.516741 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:26.516020    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.494265621 podStartE2EDuration="3.515996443s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.730822ms" isStatefulPod=false podCreationTimestamp="2026-07-07 16:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:49:26.513983162 +0000 UTC m=+159.526992982" watchObservedRunningTime="2026-07-07 16:49:26.515996443 +0000 UTC m=+159.529006262"
Jul 07 16:49:28.845437 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:49:28.845399    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:23.845067 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:50:23.845023    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:23.861438 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:50:23.861415    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:50:24.731121 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:50:24.731094    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:51:47.406466 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:51:47.406437    2583 kubelet.go:1648] "Image garbage collection succeeded"
Jul 07 16:54:35.213198 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:54:35.213162    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-85768989b-4w4z7"]
Jul 07 16:54:35.220406 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:54:35.220374    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85768989b-4w4z7"
Jul 07 16:54:35.224663 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:54:35.224632    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-qhwl4\""
Jul 07 16:54:35.225640 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:54:35.225605    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 07 16:54:35.225640 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:54:35.225613    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 07 16:54:35.225815 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:54:35.225644    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 07 16:54:35.232208 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:54:35.232183    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85768989b-4w4z7"]
Jul 07 16:54:35.233577 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:54:35.233556    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znhjm\" (UniqueName: \"kubernetes.io/projected/20019ea8-d789-4a01-8a3b-499d532b7b2e-kube-api-access-znhjm\") pod \"kserve-controller-manager-85768989b-4w4z7\" (UID: \"20019ea8-d789-4a01-8a3b-499d532b7b2e\") " pod="kserve/kserve-controller-manager-85768989b-4w4z7"
Jul 07 16:54:35.233667 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:54:35.233601    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20019ea8-d789-4a01-8a3b-499d532b7b2e-cert\") pod \"kserve-controller-manager-85768989b-4w4z7\" (UID: \"20019ea8-d789-4a01-8a3b-499d532b7b2e\") " pod="kserve/kserve-controller-manager-85768989b-4w4z7"
Jul 07 16:54:35.334885 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:54:35.334847    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20019ea8-d789-4a01-8a3b-499d532b7b2e-cert\") pod \"kserve-controller-manager-85768989b-4w4z7\" (UID: \"20019ea8-d789-4a01-8a3b-499d532b7b2e\") " pod="kserve/kserve-controller-manager-85768989b-4w4z7"
Jul 07 16:54:35.335095 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:54:35.334962    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znhjm\" (UniqueName: \"kubernetes.io/projected/20019ea8-d789-4a01-8a3b-499d532b7b2e-kube-api-access-znhjm\") pod \"kserve-controller-manager-85768989b-4w4z7\" (UID: \"20019ea8-d789-4a01-8a3b-499d532b7b2e\") " pod="kserve/kserve-controller-manager-85768989b-4w4z7"
Jul 07 16:54:35.337437 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:54:35.337412    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20019ea8-d789-4a01-8a3b-499d532b7b2e-cert\") pod \"kserve-controller-manager-85768989b-4w4z7\" (UID: \"20019ea8-d789-4a01-8a3b-499d532b7b2e\") " pod="kserve/kserve-controller-manager-85768989b-4w4z7"
Jul 07 16:54:35.346576 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:54:35.346549    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znhjm\" (UniqueName: \"kubernetes.io/projected/20019ea8-d789-4a01-8a3b-499d532b7b2e-kube-api-access-znhjm\") pod \"kserve-controller-manager-85768989b-4w4z7\" (UID: \"20019ea8-d789-4a01-8a3b-499d532b7b2e\") " pod="kserve/kserve-controller-manager-85768989b-4w4z7"
Jul 07 16:54:35.531744 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:54:35.531654    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85768989b-4w4z7"
Jul 07 16:54:35.676718 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:54:35.676671    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85768989b-4w4z7"]
Jul 07 16:54:35.679613 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:54:35.679584    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20019ea8_d789_4a01_8a3b_499d532b7b2e.slice/crio-12296fec4d1f00a5e192a4eaec9c6a459032fbf20db2053b09d57826c3c71033 WatchSource:0}: Error finding container 12296fec4d1f00a5e192a4eaec9c6a459032fbf20db2053b09d57826c3c71033: Status 404 returned error can't find the container with id 12296fec4d1f00a5e192a4eaec9c6a459032fbf20db2053b09d57826c3c71033
Jul 07 16:54:35.680935 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:54:35.680917    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 16:54:35.702560 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:54:35.702532    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85768989b-4w4z7" event={"ID":"20019ea8-d789-4a01-8a3b-499d532b7b2e","Type":"ContainerStarted","Data":"12296fec4d1f00a5e192a4eaec9c6a459032fbf20db2053b09d57826c3c71033"}
Jul 07 16:54:38.719508 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:54:38.719474    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85768989b-4w4z7" event={"ID":"20019ea8-d789-4a01-8a3b-499d532b7b2e","Type":"ContainerStarted","Data":"d8cb1587a9c4a844ed776bd436f53fc28eff9bf7109adc48c31e354c95393a91"}
Jul 07 16:54:38.719963 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:54:38.719546    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-85768989b-4w4z7"
Jul 07 16:54:38.744433 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:54:38.744375    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-85768989b-4w4z7" podStartSLOduration=1.459636546 podStartE2EDuration="3.744358787s" totalImagesPullingTime="2.284722241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:54:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:54:38.744213072 +0000 UTC m=+471.757222892" watchObservedRunningTime="2026-07-07 16:54:38.744358787 +0000 UTC m=+471.757368606"
Jul 07 16:55:09.728900 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:09.728876    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-85768989b-4w4z7"
Jul 07 16:55:10.618086 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:10.618049    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-85768989b-4w4z7"]
Jul 07 16:55:10.618333 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:10.618286    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-85768989b-4w4z7" podUID="20019ea8-d789-4a01-8a3b-499d532b7b2e" containerName="manager" containerID="cri-o://d8cb1587a9c4a844ed776bd436f53fc28eff9bf7109adc48c31e354c95393a91" gracePeriod=10
Jul 07 16:55:10.646319 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:10.646288    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-85768989b-rq7r5"]
Jul 07 16:55:10.649659 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:10.649642    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85768989b-rq7r5"
Jul 07 16:55:10.660778 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:10.660754    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85768989b-rq7r5"]
Jul 07 16:55:10.761556 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:10.761521    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84046f9b-fba7-4c61-abf9-f991d16bfb74-cert\") pod \"kserve-controller-manager-85768989b-rq7r5\" (UID: \"84046f9b-fba7-4c61-abf9-f991d16bfb74\") " pod="kserve/kserve-controller-manager-85768989b-rq7r5"
Jul 07 16:55:10.767168 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:10.761783    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-887r6\" (UniqueName: \"kubernetes.io/projected/84046f9b-fba7-4c61-abf9-f991d16bfb74-kube-api-access-887r6\") pod \"kserve-controller-manager-85768989b-rq7r5\" (UID: \"84046f9b-fba7-4c61-abf9-f991d16bfb74\") " pod="kserve/kserve-controller-manager-85768989b-rq7r5"
Jul 07 16:55:10.849903 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:10.849864    2583 generic.go:356] "Generic (PLEG): container finished" podID="20019ea8-d789-4a01-8a3b-499d532b7b2e" containerID="d8cb1587a9c4a844ed776bd436f53fc28eff9bf7109adc48c31e354c95393a91" exitCode=0
Jul 07 16:55:10.850070 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:10.849924    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85768989b-4w4z7" event={"ID":"20019ea8-d789-4a01-8a3b-499d532b7b2e","Type":"ContainerDied","Data":"d8cb1587a9c4a844ed776bd436f53fc28eff9bf7109adc48c31e354c95393a91"}
Jul 07 16:55:10.863015 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:10.862985    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887r6\" (UniqueName: \"kubernetes.io/projected/84046f9b-fba7-4c61-abf9-f991d16bfb74-kube-api-access-887r6\") pod \"kserve-controller-manager-85768989b-rq7r5\" (UID: \"84046f9b-fba7-4c61-abf9-f991d16bfb74\") " pod="kserve/kserve-controller-manager-85768989b-rq7r5"
Jul 07 16:55:10.863160 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:10.863094    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84046f9b-fba7-4c61-abf9-f991d16bfb74-cert\") pod \"kserve-controller-manager-85768989b-rq7r5\" (UID: \"84046f9b-fba7-4c61-abf9-f991d16bfb74\") " pod="kserve/kserve-controller-manager-85768989b-rq7r5"
Jul 07 16:55:10.865683 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:10.865661    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84046f9b-fba7-4c61-abf9-f991d16bfb74-cert\") pod \"kserve-controller-manager-85768989b-rq7r5\" (UID: \"84046f9b-fba7-4c61-abf9-f991d16bfb74\") " pod="kserve/kserve-controller-manager-85768989b-rq7r5"
Jul 07 16:55:10.873270 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:10.873215    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-887r6\" (UniqueName: \"kubernetes.io/projected/84046f9b-fba7-4c61-abf9-f991d16bfb74-kube-api-access-887r6\") pod \"kserve-controller-manager-85768989b-rq7r5\" (UID: \"84046f9b-fba7-4c61-abf9-f991d16bfb74\") " pod="kserve/kserve-controller-manager-85768989b-rq7r5"
Jul 07 16:55:10.887756 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:10.887737    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85768989b-4w4z7"
Jul 07 16:55:10.995181 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:10.995128    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85768989b-rq7r5"
Jul 07 16:55:11.065766 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:11.065731    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20019ea8-d789-4a01-8a3b-499d532b7b2e-cert\") pod \"20019ea8-d789-4a01-8a3b-499d532b7b2e\" (UID: \"20019ea8-d789-4a01-8a3b-499d532b7b2e\") "
Jul 07 16:55:11.065932 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:11.065800    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znhjm\" (UniqueName: \"kubernetes.io/projected/20019ea8-d789-4a01-8a3b-499d532b7b2e-kube-api-access-znhjm\") pod \"20019ea8-d789-4a01-8a3b-499d532b7b2e\" (UID: \"20019ea8-d789-4a01-8a3b-499d532b7b2e\") "
Jul 07 16:55:11.067913 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:11.067882    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20019ea8-d789-4a01-8a3b-499d532b7b2e-kube-api-access-znhjm" (OuterVolumeSpecName: "kube-api-access-znhjm") pod "20019ea8-d789-4a01-8a3b-499d532b7b2e" (UID: "20019ea8-d789-4a01-8a3b-499d532b7b2e"). InnerVolumeSpecName "kube-api-access-znhjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:55:11.068005 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:11.067932    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20019ea8-d789-4a01-8a3b-499d532b7b2e-cert" (OuterVolumeSpecName: "cert") pod "20019ea8-d789-4a01-8a3b-499d532b7b2e" (UID: "20019ea8-d789-4a01-8a3b-499d532b7b2e"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:55:11.127238 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:11.127212    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-85768989b-rq7r5"]
Jul 07 16:55:11.129308 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:55:11.129280    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84046f9b_fba7_4c61_abf9_f991d16bfb74.slice/crio-82c59be48b41861eaa184bc7d405fd78bfb45c23a8a6b0fe798cd28c0b224673 WatchSource:0}: Error finding container 82c59be48b41861eaa184bc7d405fd78bfb45c23a8a6b0fe798cd28c0b224673: Status 404 returned error can't find the container with id 82c59be48b41861eaa184bc7d405fd78bfb45c23a8a6b0fe798cd28c0b224673
Jul 07 16:55:11.166898 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:11.166866    2583 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20019ea8-d789-4a01-8a3b-499d532b7b2e-cert\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:55:11.166898 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:11.166893    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znhjm\" (UniqueName: \"kubernetes.io/projected/20019ea8-d789-4a01-8a3b-499d532b7b2e-kube-api-access-znhjm\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:55:11.855591 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:11.855492    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85768989b-rq7r5" event={"ID":"84046f9b-fba7-4c61-abf9-f991d16bfb74","Type":"ContainerStarted","Data":"0593006cf91a0005f97c7cba2bff833158a03ee8e0656c020806959be1014219"}
Jul 07 16:55:11.855591 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:11.855564    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-85768989b-rq7r5"
Jul 07 16:55:11.855591 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:11.855586    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85768989b-rq7r5" event={"ID":"84046f9b-fba7-4c61-abf9-f991d16bfb74","Type":"ContainerStarted","Data":"82c59be48b41861eaa184bc7d405fd78bfb45c23a8a6b0fe798cd28c0b224673"}
Jul 07 16:55:11.856934 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:11.856905    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-85768989b-4w4z7" event={"ID":"20019ea8-d789-4a01-8a3b-499d532b7b2e","Type":"ContainerDied","Data":"12296fec4d1f00a5e192a4eaec9c6a459032fbf20db2053b09d57826c3c71033"}
Jul 07 16:55:11.857039 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:11.856961    2583 scope.go:117] "RemoveContainer" containerID="d8cb1587a9c4a844ed776bd436f53fc28eff9bf7109adc48c31e354c95393a91"
Jul 07 16:55:11.857039 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:11.856913    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-85768989b-4w4z7"
Jul 07 16:55:11.875941 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:11.875887    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-85768989b-rq7r5" podStartSLOduration=1.440919675 podStartE2EDuration="1.875867269s" totalImagesPullingTime="434.947594ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:55:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:55:11.875161999 +0000 UTC m=+504.888171816" watchObservedRunningTime="2026-07-07 16:55:11.875867269 +0000 UTC m=+504.888877089"
Jul 07 16:55:11.894955 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:11.894912    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-85768989b-4w4z7"]
Jul 07 16:55:11.899756 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:11.899723    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-85768989b-4w4z7"]
Jul 07 16:55:13.586770 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:13.586725    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20019ea8-d789-4a01-8a3b-499d532b7b2e" path="/var/lib/kubelet/pods/20019ea8-d789-4a01-8a3b-499d532b7b2e/volumes"
Jul 07 16:55:42.868552 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:55:42.868472    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-85768989b-rq7r5"
Jul 07 16:56:00.786914 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:56:00.786878    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-6slsd"]
Jul 07 16:56:00.787417 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:56:00.787233    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20019ea8-d789-4a01-8a3b-499d532b7b2e" containerName="manager"
Jul 07 16:56:00.787417 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:56:00.787245    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="20019ea8-d789-4a01-8a3b-499d532b7b2e" containerName="manager"
Jul 07 16:56:00.787417 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:56:00.787293    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="20019ea8-d789-4a01-8a3b-499d532b7b2e" containerName="manager"
Jul 07 16:56:00.791444 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:56:00.791425    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6slsd"
Jul 07 16:56:00.794392 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:56:00.794373    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-87sf2\""
Jul 07 16:56:00.794858 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:56:00.794843    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 07 16:56:00.800794 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:56:00.800765    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-6slsd"]
Jul 07 16:56:00.807188 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:56:00.807168    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfdjf\" (UniqueName: \"kubernetes.io/projected/d02288a1-c415-4487-9751-4b2b55a45360-kube-api-access-tfdjf\") pod \"s3-init-6slsd\" (UID: \"d02288a1-c415-4487-9751-4b2b55a45360\") " pod="kserve/s3-init-6slsd"
Jul 07 16:56:00.807288 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:56:00.807205    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d02288a1-c415-4487-9751-4b2b55a45360-model-cache\") pod \"s3-init-6slsd\" (UID: \"d02288a1-c415-4487-9751-4b2b55a45360\") " pod="kserve/s3-init-6slsd"
Jul 07 16:56:00.907950 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:56:00.907914    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfdjf\" (UniqueName: \"kubernetes.io/projected/d02288a1-c415-4487-9751-4b2b55a45360-kube-api-access-tfdjf\") pod \"s3-init-6slsd\" (UID: \"d02288a1-c415-4487-9751-4b2b55a45360\") " pod="kserve/s3-init-6slsd"
Jul 07 16:56:00.908171 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:56:00.907966    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d02288a1-c415-4487-9751-4b2b55a45360-model-cache\") pod \"s3-init-6slsd\" (UID: \"d02288a1-c415-4487-9751-4b2b55a45360\") " pod="kserve/s3-init-6slsd"
Jul 07 16:56:00.908342 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:56:00.908321    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d02288a1-c415-4487-9751-4b2b55a45360-model-cache\") pod \"s3-init-6slsd\" (UID: \"d02288a1-c415-4487-9751-4b2b55a45360\") " pod="kserve/s3-init-6slsd"
Jul 07 16:56:00.920210 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:56:00.920187    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfdjf\" (UniqueName: \"kubernetes.io/projected/d02288a1-c415-4487-9751-4b2b55a45360-kube-api-access-tfdjf\") pod \"s3-init-6slsd\" (UID: \"d02288a1-c415-4487-9751-4b2b55a45360\") " pod="kserve/s3-init-6slsd"
Jul 07 16:56:01.109054 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:56:01.108966    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6slsd"
Jul 07 16:56:01.258754 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:56:01.258722    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-6slsd"]
Jul 07 16:56:01.260216 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:56:01.260186    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd02288a1_c415_4487_9751_4b2b55a45360.slice/crio-f9fc90122308044455b332f2c3ab01b16502ac19cc3f05e89a3e28203bec741e WatchSource:0}: Error finding container f9fc90122308044455b332f2c3ab01b16502ac19cc3f05e89a3e28203bec741e: Status 404 returned error can't find the container with id f9fc90122308044455b332f2c3ab01b16502ac19cc3f05e89a3e28203bec741e
Jul 07 16:56:02.056370 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:56:02.056328    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6slsd" event={"ID":"d02288a1-c415-4487-9751-4b2b55a45360","Type":"ContainerStarted","Data":"f9fc90122308044455b332f2c3ab01b16502ac19cc3f05e89a3e28203bec741e"}
Jul 07 16:56:06.078735 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:56:06.078696    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6slsd" event={"ID":"d02288a1-c415-4487-9751-4b2b55a45360","Type":"ContainerStarted","Data":"c58930098393e1b024645e7fbb6d70fd92e2c2b958ff2ef0ced0023f721d8c29"}
Jul 07 16:56:08.088362 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:56:08.088327    2583 generic.go:356] "Generic (PLEG): container finished" podID="d02288a1-c415-4487-9751-4b2b55a45360" containerID="c58930098393e1b024645e7fbb6d70fd92e2c2b958ff2ef0ced0023f721d8c29" exitCode=0
Jul 07 16:56:08.088885 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:56:08.088406    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6slsd" event={"ID":"d02288a1-c415-4487-9751-4b2b55a45360","Type":"ContainerDied","Data":"c58930098393e1b024645e7fbb6d70fd92e2c2b958ff2ef0ced0023f721d8c29"}
Jul 07 16:56:13.117509 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:56:13.117414    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6slsd" event={"ID":"d02288a1-c415-4487-9751-4b2b55a45360","Type":"ContainerStarted","Data":"2f88c9674744d07e849594cae2ef00fa7cfa936b1ba4bd9ae1fd4d354e8e0e3e"}
Jul 07 16:56:13.143332 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:56:13.143275    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-6slsd" podStartSLOduration=2.442490948 podStartE2EDuration="13.143257564s" totalImagesPullingTime="8.753620501s" totalInitContainerRuntime="1.947146115s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:56:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:56:13.14139629 +0000 UTC m=+566.154406132" watchObservedRunningTime="2026-07-07 16:56:13.143257564 +0000 UTC m=+566.156267382"
Jul 07 16:57:18.391201 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:18.391165    2583 generic.go:356] "Generic (PLEG): container finished" podID="d02288a1-c415-4487-9751-4b2b55a45360" containerID="2f88c9674744d07e849594cae2ef00fa7cfa936b1ba4bd9ae1fd4d354e8e0e3e" exitCode=0
Jul 07 16:57:18.391603 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:18.391239    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6slsd" event={"ID":"d02288a1-c415-4487-9751-4b2b55a45360","Type":"ContainerDied","Data":"2f88c9674744d07e849594cae2ef00fa7cfa936b1ba4bd9ae1fd4d354e8e0e3e"}
Jul 07 16:57:19.529911 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:19.529882    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6slsd"
Jul 07 16:57:19.702056 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:19.702026    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d02288a1-c415-4487-9751-4b2b55a45360-model-cache\") pod \"d02288a1-c415-4487-9751-4b2b55a45360\" (UID: \"d02288a1-c415-4487-9751-4b2b55a45360\") "
Jul 07 16:57:19.702252 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:19.702074    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfdjf\" (UniqueName: \"kubernetes.io/projected/d02288a1-c415-4487-9751-4b2b55a45360-kube-api-access-tfdjf\") pod \"d02288a1-c415-4487-9751-4b2b55a45360\" (UID: \"d02288a1-c415-4487-9751-4b2b55a45360\") "
Jul 07 16:57:19.704505 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:19.704466    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d02288a1-c415-4487-9751-4b2b55a45360-kube-api-access-tfdjf" (OuterVolumeSpecName: "kube-api-access-tfdjf") pod "d02288a1-c415-4487-9751-4b2b55a45360" (UID: "d02288a1-c415-4487-9751-4b2b55a45360"). InnerVolumeSpecName "kube-api-access-tfdjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:57:19.721287 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:19.721236    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d02288a1-c415-4487-9751-4b2b55a45360-model-cache" (OuterVolumeSpecName: "model-cache") pod "d02288a1-c415-4487-9751-4b2b55a45360" (UID: "d02288a1-c415-4487-9751-4b2b55a45360"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:57:19.803106 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:19.803066    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d02288a1-c415-4487-9751-4b2b55a45360-model-cache\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:57:19.803106 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:19.803097    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfdjf\" (UniqueName: \"kubernetes.io/projected/d02288a1-c415-4487-9751-4b2b55a45360-kube-api-access-tfdjf\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:57:20.400899 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:20.400875    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6slsd"
Jul 07 16:57:20.401092 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:20.400869    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6slsd" event={"ID":"d02288a1-c415-4487-9751-4b2b55a45360","Type":"ContainerDied","Data":"f9fc90122308044455b332f2c3ab01b16502ac19cc3f05e89a3e28203bec741e"}
Jul 07 16:57:20.401092 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:20.400987    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9fc90122308044455b332f2c3ab01b16502ac19cc3f05e89a3e28203bec741e"
Jul 07 16:57:32.904807 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:32.904770    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"]
Jul 07 16:57:32.905259 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:32.905102    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d02288a1-c415-4487-9751-4b2b55a45360" containerName="s3-init"
Jul 07 16:57:32.905259 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:32.905113    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d02288a1-c415-4487-9751-4b2b55a45360" containerName="s3-init"
Jul 07 16:57:32.905259 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:32.905154    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d02288a1-c415-4487-9751-4b2b55a45360" containerName="download-hf-model"
Jul 07 16:57:32.905259 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:32.905160    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d02288a1-c415-4487-9751-4b2b55a45360" containerName="download-hf-model"
Jul 07 16:57:32.905259 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:32.905221    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d02288a1-c415-4487-9751-4b2b55a45360" containerName="s3-init"
Jul 07 16:57:32.907598 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:32.907578    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:57:32.913978 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:32.913945    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-59e75-kube-rbac-proxy-sar-config\""
Jul 07 16:57:32.914109 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:32.914016    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 16:57:32.914415 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:32.914213    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-msz67\""
Jul 07 16:57:32.914415 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:32.914250    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-59e75-predictor-serving-cert\""
Jul 07 16:57:32.914415 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:32.914289    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 16:57:32.914415 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:32.914292    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 16:57:32.914965 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:32.914715    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 16:57:32.918440 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:32.918419    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a83d6124-d922-4c18-b643-15dfd4c8ee4b-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66\" (UID: \"a83d6124-d922-4c18-b643-15dfd4c8ee4b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:57:32.918537 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:32.918460    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a83d6124-d922-4c18-b643-15dfd4c8ee4b-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66\" (UID: \"a83d6124-d922-4c18-b643-15dfd4c8ee4b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:57:32.918537 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:32.918500    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a83d6124-d922-4c18-b643-15dfd4c8ee4b-proxy-tls\") pod \"isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66\" (UID: \"a83d6124-d922-4c18-b643-15dfd4c8ee4b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:57:32.918537 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:32.918523    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a83d6124-d922-4c18-b643-15dfd4c8ee4b-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66\" (UID: \"a83d6124-d922-4c18-b643-15dfd4c8ee4b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:57:32.918644 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:32.918549    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vfz9\" (UniqueName: \"kubernetes.io/projected/a83d6124-d922-4c18-b643-15dfd4c8ee4b-kube-api-access-7vfz9\") pod \"isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66\" (UID: \"a83d6124-d922-4c18-b643-15dfd4c8ee4b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:57:32.929088 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:32.929063    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"]
Jul 07 16:57:33.019564 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:33.019522    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vfz9\" (UniqueName: \"kubernetes.io/projected/a83d6124-d922-4c18-b643-15dfd4c8ee4b-kube-api-access-7vfz9\") pod \"isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66\" (UID: \"a83d6124-d922-4c18-b643-15dfd4c8ee4b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:57:33.019730 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:33.019573    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a83d6124-d922-4c18-b643-15dfd4c8ee4b-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66\" (UID: \"a83d6124-d922-4c18-b643-15dfd4c8ee4b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:57:33.019730 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:33.019622    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a83d6124-d922-4c18-b643-15dfd4c8ee4b-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66\" (UID: \"a83d6124-d922-4c18-b643-15dfd4c8ee4b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:57:33.019730 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:33.019671    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a83d6124-d922-4c18-b643-15dfd4c8ee4b-proxy-tls\") pod \"isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66\" (UID: \"a83d6124-d922-4c18-b643-15dfd4c8ee4b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:57:33.019730 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:33.019702    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a83d6124-d922-4c18-b643-15dfd4c8ee4b-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66\" (UID: \"a83d6124-d922-4c18-b643-15dfd4c8ee4b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:57:33.019889 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:57:33.019790    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-serving-cert: secret "isvc-raw-sklearn-batcher-59e75-predictor-serving-cert" not found
Jul 07 16:57:33.019924 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:57:33.019890    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a83d6124-d922-4c18-b643-15dfd4c8ee4b-proxy-tls podName:a83d6124-d922-4c18-b643-15dfd4c8ee4b nodeName:}" failed. No retries permitted until 2026-07-07 16:57:33.519869162 +0000 UTC m=+646.532878959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a83d6124-d922-4c18-b643-15dfd4c8ee4b-proxy-tls") pod "isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" (UID: "a83d6124-d922-4c18-b643-15dfd4c8ee4b") : secret "isvc-raw-sklearn-batcher-59e75-predictor-serving-cert" not found
Jul 07 16:57:33.020209 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:33.020192    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a83d6124-d922-4c18-b643-15dfd4c8ee4b-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66\" (UID: \"a83d6124-d922-4c18-b643-15dfd4c8ee4b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:57:33.020366 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:33.020349    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a83d6124-d922-4c18-b643-15dfd4c8ee4b-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66\" (UID: \"a83d6124-d922-4c18-b643-15dfd4c8ee4b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:57:33.020405 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:33.020362    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a83d6124-d922-4c18-b643-15dfd4c8ee4b-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66\" (UID: \"a83d6124-d922-4c18-b643-15dfd4c8ee4b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:57:33.030545 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:33.030513    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vfz9\" (UniqueName: \"kubernetes.io/projected/a83d6124-d922-4c18-b643-15dfd4c8ee4b-kube-api-access-7vfz9\") pod \"isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66\" (UID: \"a83d6124-d922-4c18-b643-15dfd4c8ee4b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:57:33.522974 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:33.522933    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a83d6124-d922-4c18-b643-15dfd4c8ee4b-proxy-tls\") pod \"isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66\" (UID: \"a83d6124-d922-4c18-b643-15dfd4c8ee4b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:57:33.525471 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:33.525445    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a83d6124-d922-4c18-b643-15dfd4c8ee4b-proxy-tls\") pod \"isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66\" (UID: \"a83d6124-d922-4c18-b643-15dfd4c8ee4b\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:57:33.819190 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:33.819078    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:57:33.962265 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:33.962233    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"]
Jul 07 16:57:33.964486 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:57:33.964447    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83d6124_d922_4c18_b643_15dfd4c8ee4b.slice/crio-fe6c18f47c24a7d2fe96cedc5a609eddc3d2e5b0e6fc3b90d51bd3dc56610fba WatchSource:0}: Error finding container fe6c18f47c24a7d2fe96cedc5a609eddc3d2e5b0e6fc3b90d51bd3dc56610fba: Status 404 returned error can't find the container with id fe6c18f47c24a7d2fe96cedc5a609eddc3d2e5b0e6fc3b90d51bd3dc56610fba
Jul 07 16:57:34.458944 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:34.458905    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" event={"ID":"a83d6124-d922-4c18-b643-15dfd4c8ee4b","Type":"ContainerStarted","Data":"dd3e18c2ac24620ce0e8d769510bd77a290215de89796cf1fa6789f033468d7b"}
Jul 07 16:57:34.459123 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:34.458992    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" event={"ID":"a83d6124-d922-4c18-b643-15dfd4c8ee4b","Type":"ContainerStarted","Data":"fe6c18f47c24a7d2fe96cedc5a609eddc3d2e5b0e6fc3b90d51bd3dc56610fba"}
Jul 07 16:57:38.476362 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:38.476330    2583 generic.go:356] "Generic (PLEG): container finished" podID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerID="dd3e18c2ac24620ce0e8d769510bd77a290215de89796cf1fa6789f033468d7b" exitCode=0
Jul 07 16:57:38.476852 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:38.476407    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" event={"ID":"a83d6124-d922-4c18-b643-15dfd4c8ee4b","Type":"ContainerDied","Data":"dd3e18c2ac24620ce0e8d769510bd77a290215de89796cf1fa6789f033468d7b"}
Jul 07 16:57:53.554695 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:53.554659    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" event={"ID":"a83d6124-d922-4c18-b643-15dfd4c8ee4b","Type":"ContainerStarted","Data":"5e87b3ea408047d7bae09c794a085bbc7477f835682f76df7f2df7632f2a8b0a"}
Jul 07 16:57:56.570904 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:56.570860    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" event={"ID":"a83d6124-d922-4c18-b643-15dfd4c8ee4b","Type":"ContainerStarted","Data":"dd7e7341ec004abaae9cd0888ad5f1e12da5acdf98dcbcfcc297e35008380443"}
Jul 07 16:57:58.581871 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:58.581772    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" event={"ID":"a83d6124-d922-4c18-b643-15dfd4c8ee4b","Type":"ContainerStarted","Data":"fb3c97b023670d5f3bb0efceb39bd42a6c88884ecc6ce7701c0880a13508a808"}
Jul 07 16:57:58.582346 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:58.582006    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:57:58.582346 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:58.582168    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:57:58.583529 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:58.583500    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 16:57:59.585900 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:59.585857    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 16:57:59.586351 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:59.585873    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:57:59.586969 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:57:59.586944    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:58:00.589531 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:58:00.589492    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 16:58:00.590025 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:58:00.589993    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:58:00.593381 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:58:00.593362    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:58:00.624258 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:58:00.624208    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podStartSLOduration=5.163353377 podStartE2EDuration="28.624191344s" totalImagesPullingTime="19.634346392s" totalInitContainerRuntime="3.826491575s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:57:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:57:58.613660682 +0000 UTC m=+671.626670500" watchObservedRunningTime="2026-07-07 16:58:00.624191344 +0000 UTC m=+673.637201162"
Jul 07 16:58:01.593632 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:58:01.593588    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 16:58:01.594061 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:58:01.593886    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:58:11.594111 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:58:11.594052    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 16:58:11.594631 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:58:11.594481    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:58:21.593750 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:58:21.593700    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 16:58:21.594331 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:58:21.594205    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:58:31.594307 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:58:31.594256    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 16:58:31.594823 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:58:31.594739    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:58:41.594269 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:58:41.594171    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 16:58:41.594794 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:58:41.594695    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:58:51.593624 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:58:51.593569    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 16:58:51.594029 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:58:51.594002    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:59:01.594264 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:01.594220    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 16:59:01.594819 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:01.594794    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:59:11.594315 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:11.594280    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:59:11.594801 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:11.594769    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:59:18.204739 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.204711    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"]
Jul 07 16:59:18.205306 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.204970    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="kserve-container" containerID="cri-o://5e87b3ea408047d7bae09c794a085bbc7477f835682f76df7f2df7632f2a8b0a" gracePeriod=30
Jul 07 16:59:18.205306 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.204983    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="agent" containerID="cri-o://fb3c97b023670d5f3bb0efceb39bd42a6c88884ecc6ce7701c0880a13508a808" gracePeriod=30
Jul 07 16:59:18.205306 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.205027    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="kube-rbac-proxy" containerID="cri-o://dd7e7341ec004abaae9cd0888ad5f1e12da5acdf98dcbcfcc297e35008380443" gracePeriod=30
Jul 07 16:59:18.229264 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.229236    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"]
Jul 07 16:59:18.232055 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.232033    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"
Jul 07 16:59:18.243682 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.243655    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-f934a-predictor-serving-cert\""
Jul 07 16:59:18.245033 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.245015    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-f934a-kube-rbac-proxy-sar-config\""
Jul 07 16:59:18.252389 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.252359    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"]
Jul 07 16:59:18.360870 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.360838    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-262k9\" (UniqueName: \"kubernetes.io/projected/3c168786-3492-473d-97ba-6044c874eb11-kube-api-access-262k9\") pod \"isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j\" (UID: \"3c168786-3492-473d-97ba-6044c874eb11\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"
Jul 07 16:59:18.361025 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.360906    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c168786-3492-473d-97ba-6044c874eb11-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j\" (UID: \"3c168786-3492-473d-97ba-6044c874eb11\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"
Jul 07 16:59:18.361025 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.360956    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c168786-3492-473d-97ba-6044c874eb11-proxy-tls\") pod \"isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j\" (UID: \"3c168786-3492-473d-97ba-6044c874eb11\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"
Jul 07 16:59:18.361025 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.360973    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c168786-3492-473d-97ba-6044c874eb11-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j\" (UID: \"3c168786-3492-473d-97ba-6044c874eb11\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"
Jul 07 16:59:18.361184 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.361061    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c168786-3492-473d-97ba-6044c874eb11-cabundle-cert\") pod \"isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j\" (UID: \"3c168786-3492-473d-97ba-6044c874eb11\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"
Jul 07 16:59:18.405346 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.405312    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"]
Jul 07 16:59:18.408284 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.408263    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"
Jul 07 16:59:18.411235 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.411218    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-f934a-predictor-serving-cert\""
Jul 07 16:59:18.412586 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.412572    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-f934a-kube-rbac-proxy-sar-config\""
Jul 07 16:59:18.427739 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.427708    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"]
Jul 07 16:59:18.461810 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.461723    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c168786-3492-473d-97ba-6044c874eb11-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j\" (UID: \"3c168786-3492-473d-97ba-6044c874eb11\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"
Jul 07 16:59:18.461810 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.461769    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4266173-964e-4c0a-98d1-ac9a5379820b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d\" (UID: \"c4266173-964e-4c0a-98d1-ac9a5379820b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"
Jul 07 16:59:18.461810 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.461790    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c168786-3492-473d-97ba-6044c874eb11-proxy-tls\") pod \"isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j\" (UID: \"3c168786-3492-473d-97ba-6044c874eb11\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"
Jul 07 16:59:18.461810 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.461810    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c168786-3492-473d-97ba-6044c874eb11-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j\" (UID: \"3c168786-3492-473d-97ba-6044c874eb11\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"
Jul 07 16:59:18.462216 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.461881    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4266173-964e-4c0a-98d1-ac9a5379820b-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d\" (UID: \"c4266173-964e-4c0a-98d1-ac9a5379820b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"
Jul 07 16:59:18.462216 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:59:18.461916    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-serving-cert: secret "isvc-sklearn-graph-raw-f934a-predictor-serving-cert" not found
Jul 07 16:59:18.462216 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.461923    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4266173-964e-4c0a-98d1-ac9a5379820b-cabundle-cert\") pod \"isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d\" (UID: \"c4266173-964e-4c0a-98d1-ac9a5379820b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"
Jul 07 16:59:18.462216 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.461958    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c168786-3492-473d-97ba-6044c874eb11-cabundle-cert\") pod \"isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j\" (UID: \"3c168786-3492-473d-97ba-6044c874eb11\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"
Jul 07 16:59:18.462216 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:59:18.461983    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c168786-3492-473d-97ba-6044c874eb11-proxy-tls podName:3c168786-3492-473d-97ba-6044c874eb11 nodeName:}" failed. No retries permitted until 2026-07-07 16:59:18.961961095 +0000 UTC m=+751.974970908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3c168786-3492-473d-97ba-6044c874eb11-proxy-tls") pod "isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j" (UID: "3c168786-3492-473d-97ba-6044c874eb11") : secret "isvc-sklearn-graph-raw-f934a-predictor-serving-cert" not found
Jul 07 16:59:18.462216 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.462077    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f24kl\" (UniqueName: \"kubernetes.io/projected/c4266173-964e-4c0a-98d1-ac9a5379820b-kube-api-access-f24kl\") pod \"isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d\" (UID: \"c4266173-964e-4c0a-98d1-ac9a5379820b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"
Jul 07 16:59:18.462216 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.462113    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-262k9\" (UniqueName: \"kubernetes.io/projected/3c168786-3492-473d-97ba-6044c874eb11-kube-api-access-262k9\") pod \"isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j\" (UID: \"3c168786-3492-473d-97ba-6044c874eb11\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"
Jul 07 16:59:18.462216 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.462218    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4266173-964e-4c0a-98d1-ac9a5379820b-proxy-tls\") pod \"isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d\" (UID: \"c4266173-964e-4c0a-98d1-ac9a5379820b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"
Jul 07 16:59:18.462636 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.462222    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c168786-3492-473d-97ba-6044c874eb11-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j\" (UID: \"3c168786-3492-473d-97ba-6044c874eb11\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"
Jul 07 16:59:18.462636 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.462560    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c168786-3492-473d-97ba-6044c874eb11-cabundle-cert\") pod \"isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j\" (UID: \"3c168786-3492-473d-97ba-6044c874eb11\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"
Jul 07 16:59:18.462732 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.462672    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c168786-3492-473d-97ba-6044c874eb11-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j\" (UID: \"3c168786-3492-473d-97ba-6044c874eb11\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"
Jul 07 16:59:18.476967 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.476937    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-262k9\" (UniqueName: \"kubernetes.io/projected/3c168786-3492-473d-97ba-6044c874eb11-kube-api-access-262k9\") pod \"isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j\" (UID: \"3c168786-3492-473d-97ba-6044c874eb11\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"
Jul 07 16:59:18.563371 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.563327    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4266173-964e-4c0a-98d1-ac9a5379820b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d\" (UID: \"c4266173-964e-4c0a-98d1-ac9a5379820b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"
Jul 07 16:59:18.563567 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.563405    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4266173-964e-4c0a-98d1-ac9a5379820b-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d\" (UID: \"c4266173-964e-4c0a-98d1-ac9a5379820b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"
Jul 07 16:59:18.563567 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.563436    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4266173-964e-4c0a-98d1-ac9a5379820b-cabundle-cert\") pod \"isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d\" (UID: \"c4266173-964e-4c0a-98d1-ac9a5379820b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"
Jul 07 16:59:18.563567 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.563489    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f24kl\" (UniqueName: \"kubernetes.io/projected/c4266173-964e-4c0a-98d1-ac9a5379820b-kube-api-access-f24kl\") pod \"isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d\" (UID: \"c4266173-964e-4c0a-98d1-ac9a5379820b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"
Jul 07 16:59:18.563567 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.563541    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4266173-964e-4c0a-98d1-ac9a5379820b-proxy-tls\") pod \"isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d\" (UID: \"c4266173-964e-4c0a-98d1-ac9a5379820b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"
Jul 07 16:59:18.563932 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.563904    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4266173-964e-4c0a-98d1-ac9a5379820b-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d\" (UID: \"c4266173-964e-4c0a-98d1-ac9a5379820b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"
Jul 07 16:59:18.564335 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.564311    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4266173-964e-4c0a-98d1-ac9a5379820b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d\" (UID: \"c4266173-964e-4c0a-98d1-ac9a5379820b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"
Jul 07 16:59:18.564438 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.564377    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4266173-964e-4c0a-98d1-ac9a5379820b-cabundle-cert\") pod \"isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d\" (UID: \"c4266173-964e-4c0a-98d1-ac9a5379820b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"
Jul 07 16:59:18.566267 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.566246    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4266173-964e-4c0a-98d1-ac9a5379820b-proxy-tls\") pod \"isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d\" (UID: \"c4266173-964e-4c0a-98d1-ac9a5379820b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"
Jul 07 16:59:18.575190 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.575131    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f24kl\" (UniqueName: \"kubernetes.io/projected/c4266173-964e-4c0a-98d1-ac9a5379820b-kube-api-access-f24kl\") pod \"isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d\" (UID: \"c4266173-964e-4c0a-98d1-ac9a5379820b\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"
Jul 07 16:59:18.718239 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.718125    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"
Jul 07 16:59:18.860113 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.860089    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"]
Jul 07 16:59:18.862648 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:59:18.862616    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4266173_964e_4c0a_98d1_ac9a5379820b.slice/crio-ca7105dc9061322f441a2a90bd741ddc5fa5d0d97301ebf5ea6d84e01fc7bb3f WatchSource:0}: Error finding container ca7105dc9061322f441a2a90bd741ddc5fa5d0d97301ebf5ea6d84e01fc7bb3f: Status 404 returned error can't find the container with id ca7105dc9061322f441a2a90bd741ddc5fa5d0d97301ebf5ea6d84e01fc7bb3f
Jul 07 16:59:18.922405 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.922370    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d" event={"ID":"c4266173-964e-4c0a-98d1-ac9a5379820b","Type":"ContainerStarted","Data":"39a07001253f382add635e813ba7e15069cd26abb66a69c3275394329af4cadf"}
Jul 07 16:59:18.922558 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.922440    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d" event={"ID":"c4266173-964e-4c0a-98d1-ac9a5379820b","Type":"ContainerStarted","Data":"ca7105dc9061322f441a2a90bd741ddc5fa5d0d97301ebf5ea6d84e01fc7bb3f"}
Jul 07 16:59:18.924850 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.924823    2583 generic.go:356] "Generic (PLEG): container finished" podID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerID="dd7e7341ec004abaae9cd0888ad5f1e12da5acdf98dcbcfcc297e35008380443" exitCode=2
Jul 07 16:59:18.924964 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.924898    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" event={"ID":"a83d6124-d922-4c18-b643-15dfd4c8ee4b","Type":"ContainerDied","Data":"dd7e7341ec004abaae9cd0888ad5f1e12da5acdf98dcbcfcc297e35008380443"}
Jul 07 16:59:18.966846 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.966811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c168786-3492-473d-97ba-6044c874eb11-proxy-tls\") pod \"isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j\" (UID: \"3c168786-3492-473d-97ba-6044c874eb11\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"
Jul 07 16:59:18.969405 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:18.969326    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c168786-3492-473d-97ba-6044c874eb11-proxy-tls\") pod \"isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j\" (UID: \"3c168786-3492-473d-97ba-6044c874eb11\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"
Jul 07 16:59:19.141339 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:19.141296    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"
Jul 07 16:59:19.293267 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:19.293240    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"]
Jul 07 16:59:19.295257 ip-10-0-141-189 kubenswrapper[2583]: W0707 16:59:19.295235    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c168786_3492_473d_97ba_6044c874eb11.slice/crio-045084f284c2acda78b6fe9a4d5fddbea4bd3d3ad03f72b454bd8d4b5bc2088b WatchSource:0}: Error finding container 045084f284c2acda78b6fe9a4d5fddbea4bd3d3ad03f72b454bd8d4b5bc2088b: Status 404 returned error can't find the container with id 045084f284c2acda78b6fe9a4d5fddbea4bd3d3ad03f72b454bd8d4b5bc2088b
Jul 07 16:59:19.931712 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:19.931674    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j" event={"ID":"3c168786-3492-473d-97ba-6044c874eb11","Type":"ContainerStarted","Data":"b19a21afa4ad792182cc596ea4211677efb664074dc9dfae25a9c65cad0ccce8"}
Jul 07 16:59:19.931890 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:19.931744    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j" event={"ID":"3c168786-3492-473d-97ba-6044c874eb11","Type":"ContainerStarted","Data":"045084f284c2acda78b6fe9a4d5fddbea4bd3d3ad03f72b454bd8d4b5bc2088b"}
Jul 07 16:59:20.590514 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:20.590473    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 07 16:59:21.594348 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:21.594311    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 16:59:21.595195 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:21.595160    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:59:22.946280 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:22.946250    2583 generic.go:356] "Generic (PLEG): container finished" podID="c4266173-964e-4c0a-98d1-ac9a5379820b" containerID="39a07001253f382add635e813ba7e15069cd26abb66a69c3275394329af4cadf" exitCode=0
Jul 07 16:59:22.946629 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:22.946326    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d" event={"ID":"c4266173-964e-4c0a-98d1-ac9a5379820b","Type":"ContainerDied","Data":"39a07001253f382add635e813ba7e15069cd26abb66a69c3275394329af4cadf"}
Jul 07 16:59:23.956430 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:23.956391    2583 generic.go:356] "Generic (PLEG): container finished" podID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerID="5e87b3ea408047d7bae09c794a085bbc7477f835682f76df7f2df7632f2a8b0a" exitCode=0
Jul 07 16:59:23.956865 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:23.956432    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" event={"ID":"a83d6124-d922-4c18-b643-15dfd4c8ee4b","Type":"ContainerDied","Data":"5e87b3ea408047d7bae09c794a085bbc7477f835682f76df7f2df7632f2a8b0a"}
Jul 07 16:59:23.958517 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:23.958490    2583 generic.go:356] "Generic (PLEG): container finished" podID="3c168786-3492-473d-97ba-6044c874eb11" containerID="b19a21afa4ad792182cc596ea4211677efb664074dc9dfae25a9c65cad0ccce8" exitCode=0
Jul 07 16:59:23.958639 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:23.958529    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j" event={"ID":"3c168786-3492-473d-97ba-6044c874eb11","Type":"ContainerDied","Data":"b19a21afa4ad792182cc596ea4211677efb664074dc9dfae25a9c65cad0ccce8"}
Jul 07 16:59:25.590375 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:25.590332    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 07 16:59:25.969676 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:25.969640    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j" event={"ID":"3c168786-3492-473d-97ba-6044c874eb11","Type":"ContainerStarted","Data":"5d6e1f7eef72dcd4c0be70600f3fd4e0984a78a6e909396ce22a94191e785019"}
Jul 07 16:59:25.969815 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:25.969713    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j" event={"ID":"3c168786-3492-473d-97ba-6044c874eb11","Type":"ContainerStarted","Data":"de482e8ed156ca7cafa321c5201cb8f6d56d90d7e0f0e1ec582169b7ac78dbbf"}
Jul 07 16:59:25.969938 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:25.969913    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"
Jul 07 16:59:26.010344 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:26.010294    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j" podStartSLOduration=4.032423472 podStartE2EDuration="8.010279099s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.977855627s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:59:26.008611245 +0000 UTC m=+759.021621063" watchObservedRunningTime="2026-07-07 16:59:26.010279099 +0000 UTC m=+759.023288917"
Jul 07 16:59:26.974195 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:26.974165    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"
Jul 07 16:59:26.975350 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:26.975325    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j" podUID="3c168786-3492-473d-97ba-6044c874eb11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 16:59:27.978060 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:27.978024    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j" podUID="3c168786-3492-473d-97ba-6044c874eb11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 16:59:30.589838 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:30.589802    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 07 16:59:30.590392 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:30.589956    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:59:31.594364 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:31.594313    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 16:59:31.595984 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:31.595949    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:59:32.983220 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:32.983123    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"
Jul 07 16:59:32.983876 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:32.983778    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j" podUID="3c168786-3492-473d-97ba-6044c874eb11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 16:59:35.590489 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:35.590452    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 07 16:59:40.590264 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:40.590211    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 07 16:59:41.594126 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:41.594086    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 16:59:41.594599 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:41.594295    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:59:41.595824 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:41.595789    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:59:41.595962 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:41.595948    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:59:42.984393 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:42.984357    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j" podUID="3c168786-3492-473d-97ba-6044c874eb11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 16:59:45.065253 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:45.065212    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d" event={"ID":"c4266173-964e-4c0a-98d1-ac9a5379820b","Type":"ContainerStarted","Data":"e7b68bfdaa1ca04f4421b6cda0e31b5f3260e3ee9aaa0be7a20df2cc38f8a2f9"}
Jul 07 16:59:45.065667 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:45.065311    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d" event={"ID":"c4266173-964e-4c0a-98d1-ac9a5379820b","Type":"ContainerStarted","Data":"4e1b3536c6bcbb8bc4e03b0df893b72560d12cb5428f6880b25386caacf310f3"}
Jul 07 16:59:45.065667 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:45.065514    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"
Jul 07 16:59:45.091213 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:45.091154    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d" podStartSLOduration=1.633720836 podStartE2EDuration="27.091117809s" totalImagesPullingTime="21.485466343s" totalInitContainerRuntime="3.97193063s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:59:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:59:45.090622768 +0000 UTC m=+778.103632600" watchObservedRunningTime="2026-07-07 16:59:45.091117809 +0000 UTC m=+778.104127629"
Jul 07 16:59:45.589877 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:45.589840    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 07 16:59:46.069888 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:46.069855    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"
Jul 07 16:59:46.071298 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:46.071269    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d" podUID="c4266173-964e-4c0a-98d1-ac9a5379820b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 16:59:47.074477 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:47.074433    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d" podUID="c4266173-964e-4c0a-98d1-ac9a5379820b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 16:59:48.890400 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:48.890370    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:59:48.949089 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:48.949061    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a83d6124-d922-4c18-b643-15dfd4c8ee4b-proxy-tls\") pod \"a83d6124-d922-4c18-b643-15dfd4c8ee4b\" (UID: \"a83d6124-d922-4c18-b643-15dfd4c8ee4b\") "
Jul 07 16:59:48.949089 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:48.949101    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a83d6124-d922-4c18-b643-15dfd4c8ee4b-kube-rbac-proxy-sar-config\") pod \"a83d6124-d922-4c18-b643-15dfd4c8ee4b\" (UID: \"a83d6124-d922-4c18-b643-15dfd4c8ee4b\") "
Jul 07 16:59:48.949357 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:48.949202    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vfz9\" (UniqueName: \"kubernetes.io/projected/a83d6124-d922-4c18-b643-15dfd4c8ee4b-kube-api-access-7vfz9\") pod \"a83d6124-d922-4c18-b643-15dfd4c8ee4b\" (UID: \"a83d6124-d922-4c18-b643-15dfd4c8ee4b\") "
Jul 07 16:59:48.949357 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:48.949236    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a83d6124-d922-4c18-b643-15dfd4c8ee4b-cabundle-cert\") pod \"a83d6124-d922-4c18-b643-15dfd4c8ee4b\" (UID: \"a83d6124-d922-4c18-b643-15dfd4c8ee4b\") "
Jul 07 16:59:48.949357 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:48.949273    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a83d6124-d922-4c18-b643-15dfd4c8ee4b-kserve-provision-location\") pod \"a83d6124-d922-4c18-b643-15dfd4c8ee4b\" (UID: \"a83d6124-d922-4c18-b643-15dfd4c8ee4b\") "
Jul 07 16:59:48.949625 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:48.949595    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83d6124-d922-4c18-b643-15dfd4c8ee4b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a83d6124-d922-4c18-b643-15dfd4c8ee4b" (UID: "a83d6124-d922-4c18-b643-15dfd4c8ee4b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:59:48.949706 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:48.949636    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83d6124-d922-4c18-b643-15dfd4c8ee4b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a83d6124-d922-4c18-b643-15dfd4c8ee4b" (UID: "a83d6124-d922-4c18-b643-15dfd4c8ee4b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:59:48.949706 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:48.949673    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83d6124-d922-4c18-b643-15dfd4c8ee4b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a83d6124-d922-4c18-b643-15dfd4c8ee4b" (UID: "a83d6124-d922-4c18-b643-15dfd4c8ee4b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:59:48.951514 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:48.951487    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83d6124-d922-4c18-b643-15dfd4c8ee4b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a83d6124-d922-4c18-b643-15dfd4c8ee4b" (UID: "a83d6124-d922-4c18-b643-15dfd4c8ee4b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:59:48.951628 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:48.951548    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83d6124-d922-4c18-b643-15dfd4c8ee4b-kube-api-access-7vfz9" (OuterVolumeSpecName: "kube-api-access-7vfz9") pod "a83d6124-d922-4c18-b643-15dfd4c8ee4b" (UID: "a83d6124-d922-4c18-b643-15dfd4c8ee4b"). InnerVolumeSpecName "kube-api-access-7vfz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:59:49.050276 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:49.050182    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vfz9\" (UniqueName: \"kubernetes.io/projected/a83d6124-d922-4c18-b643-15dfd4c8ee4b-kube-api-access-7vfz9\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:59:49.050276 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:49.050215    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a83d6124-d922-4c18-b643-15dfd4c8ee4b-cabundle-cert\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:59:49.050276 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:49.050229    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a83d6124-d922-4c18-b643-15dfd4c8ee4b-kserve-provision-location\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:59:49.050276 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:49.050243    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a83d6124-d922-4c18-b643-15dfd4c8ee4b-proxy-tls\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:59:49.050276 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:49.050255    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a83d6124-d922-4c18-b643-15dfd4c8ee4b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 16:59:49.084827 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:49.084791    2583 generic.go:356] "Generic (PLEG): container finished" podID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerID="fb3c97b023670d5f3bb0efceb39bd42a6c88884ecc6ce7701c0880a13508a808" exitCode=0
Jul 07 16:59:49.084964 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:49.084855    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"
Jul 07 16:59:49.084964 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:49.084864    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" event={"ID":"a83d6124-d922-4c18-b643-15dfd4c8ee4b","Type":"ContainerDied","Data":"fb3c97b023670d5f3bb0efceb39bd42a6c88884ecc6ce7701c0880a13508a808"}
Jul 07 16:59:49.084964 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:49.084934    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66" event={"ID":"a83d6124-d922-4c18-b643-15dfd4c8ee4b","Type":"ContainerDied","Data":"fe6c18f47c24a7d2fe96cedc5a609eddc3d2e5b0e6fc3b90d51bd3dc56610fba"}
Jul 07 16:59:49.085081 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:49.084968    2583 scope.go:117] "RemoveContainer" containerID="fb3c97b023670d5f3bb0efceb39bd42a6c88884ecc6ce7701c0880a13508a808"
Jul 07 16:59:49.094124 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:49.094105    2583 scope.go:117] "RemoveContainer" containerID="dd7e7341ec004abaae9cd0888ad5f1e12da5acdf98dcbcfcc297e35008380443"
Jul 07 16:59:49.101682 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:49.101663    2583 scope.go:117] "RemoveContainer" containerID="5e87b3ea408047d7bae09c794a085bbc7477f835682f76df7f2df7632f2a8b0a"
Jul 07 16:59:49.109069 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:49.109052    2583 scope.go:117] "RemoveContainer" containerID="dd3e18c2ac24620ce0e8d769510bd77a290215de89796cf1fa6789f033468d7b"
Jul 07 16:59:49.117845 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:49.117751    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"]
Jul 07 16:59:49.117912 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:49.117892    2583 scope.go:117] "RemoveContainer" containerID="fb3c97b023670d5f3bb0efceb39bd42a6c88884ecc6ce7701c0880a13508a808"
Jul 07 16:59:49.118290 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:59:49.118270    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb3c97b023670d5f3bb0efceb39bd42a6c88884ecc6ce7701c0880a13508a808\": container with ID starting with fb3c97b023670d5f3bb0efceb39bd42a6c88884ecc6ce7701c0880a13508a808 not found: ID does not exist" containerID="fb3c97b023670d5f3bb0efceb39bd42a6c88884ecc6ce7701c0880a13508a808"
Jul 07 16:59:49.118356 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:49.118300    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb3c97b023670d5f3bb0efceb39bd42a6c88884ecc6ce7701c0880a13508a808"} err="failed to get container status \"fb3c97b023670d5f3bb0efceb39bd42a6c88884ecc6ce7701c0880a13508a808\": rpc error: code = NotFound desc = could not find container \"fb3c97b023670d5f3bb0efceb39bd42a6c88884ecc6ce7701c0880a13508a808\": container with ID starting with fb3c97b023670d5f3bb0efceb39bd42a6c88884ecc6ce7701c0880a13508a808 not found: ID does not exist"
Jul 07 16:59:49.118356 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:49.118334    2583 scope.go:117] "RemoveContainer" containerID="dd7e7341ec004abaae9cd0888ad5f1e12da5acdf98dcbcfcc297e35008380443"
Jul 07 16:59:49.118669 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:59:49.118649    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd7e7341ec004abaae9cd0888ad5f1e12da5acdf98dcbcfcc297e35008380443\": container with ID starting with dd7e7341ec004abaae9cd0888ad5f1e12da5acdf98dcbcfcc297e35008380443 not found: ID does not exist" containerID="dd7e7341ec004abaae9cd0888ad5f1e12da5acdf98dcbcfcc297e35008380443"
Jul 07 16:59:49.118754 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:49.118672    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd7e7341ec004abaae9cd0888ad5f1e12da5acdf98dcbcfcc297e35008380443"} err="failed to get container status \"dd7e7341ec004abaae9cd0888ad5f1e12da5acdf98dcbcfcc297e35008380443\": rpc error: code = NotFound desc = could not find container \"dd7e7341ec004abaae9cd0888ad5f1e12da5acdf98dcbcfcc297e35008380443\": container with ID starting with dd7e7341ec004abaae9cd0888ad5f1e12da5acdf98dcbcfcc297e35008380443 not found: ID does not exist"
Jul 07 16:59:49.118754 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:49.118689    2583 scope.go:117] "RemoveContainer" containerID="5e87b3ea408047d7bae09c794a085bbc7477f835682f76df7f2df7632f2a8b0a"
Jul 07 16:59:49.119031 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:59:49.119004    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e87b3ea408047d7bae09c794a085bbc7477f835682f76df7f2df7632f2a8b0a\": container with ID starting with 5e87b3ea408047d7bae09c794a085bbc7477f835682f76df7f2df7632f2a8b0a not found: ID does not exist" containerID="5e87b3ea408047d7bae09c794a085bbc7477f835682f76df7f2df7632f2a8b0a"
Jul 07 16:59:49.119119 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:49.119035    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e87b3ea408047d7bae09c794a085bbc7477f835682f76df7f2df7632f2a8b0a"} err="failed to get container status \"5e87b3ea408047d7bae09c794a085bbc7477f835682f76df7f2df7632f2a8b0a\": rpc error: code = NotFound desc = could not find container \"5e87b3ea408047d7bae09c794a085bbc7477f835682f76df7f2df7632f2a8b0a\": container with ID starting with 5e87b3ea408047d7bae09c794a085bbc7477f835682f76df7f2df7632f2a8b0a not found: ID does not exist"
Jul 07 16:59:49.119119 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:49.119056    2583 scope.go:117] "RemoveContainer" containerID="dd3e18c2ac24620ce0e8d769510bd77a290215de89796cf1fa6789f033468d7b"
Jul 07 16:59:49.119943 ip-10-0-141-189 kubenswrapper[2583]: E0707 16:59:49.119918    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd3e18c2ac24620ce0e8d769510bd77a290215de89796cf1fa6789f033468d7b\": container with ID starting with dd3e18c2ac24620ce0e8d769510bd77a290215de89796cf1fa6789f033468d7b not found: ID does not exist" containerID="dd3e18c2ac24620ce0e8d769510bd77a290215de89796cf1fa6789f033468d7b"
Jul 07 16:59:49.120047 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:49.119950    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd3e18c2ac24620ce0e8d769510bd77a290215de89796cf1fa6789f033468d7b"} err="failed to get container status \"dd3e18c2ac24620ce0e8d769510bd77a290215de89796cf1fa6789f033468d7b\": rpc error: code = NotFound desc = could not find container \"dd3e18c2ac24620ce0e8d769510bd77a290215de89796cf1fa6789f033468d7b\": container with ID starting with dd3e18c2ac24620ce0e8d769510bd77a290215de89796cf1fa6789f033468d7b not found: ID does not exist"
Jul 07 16:59:49.121795 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:49.121777    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-59e75-predictor-5cc5ddbf96-7bl66"]
Jul 07 16:59:49.591715 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:49.587560    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" path="/var/lib/kubelet/pods/a83d6124-d922-4c18-b643-15dfd4c8ee4b/volumes"
Jul 07 16:59:52.078921 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:52.078894    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"
Jul 07 16:59:52.079451 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:52.079424    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d" podUID="c4266173-964e-4c0a-98d1-ac9a5379820b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 16:59:52.983895 ip-10-0-141-189 kubenswrapper[2583]: I0707 16:59:52.983846    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j" podUID="3c168786-3492-473d-97ba-6044c874eb11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 17:00:02.080042 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:00:02.080001    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d" podUID="c4266173-964e-4c0a-98d1-ac9a5379820b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 17:00:02.983653 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:00:02.983611    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j" podUID="3c168786-3492-473d-97ba-6044c874eb11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 17:00:12.079444 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:00:12.079362    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d" podUID="c4266173-964e-4c0a-98d1-ac9a5379820b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 17:00:12.983878 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:00:12.983840    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j" podUID="3c168786-3492-473d-97ba-6044c874eb11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 17:00:22.079727 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:00:22.079683    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d" podUID="c4266173-964e-4c0a-98d1-ac9a5379820b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 17:00:22.984195 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:00:22.984161    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j" podUID="3c168786-3492-473d-97ba-6044c874eb11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 17:00:32.079485 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:00:32.079439    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d" podUID="c4266173-964e-4c0a-98d1-ac9a5379820b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 17:00:32.983981 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:00:32.983935    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j" podUID="3c168786-3492-473d-97ba-6044c874eb11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 17:00:42.079721 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:00:42.079681    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d" podUID="c4266173-964e-4c0a-98d1-ac9a5379820b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 17:00:42.985216 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:00:42.985185    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"
Jul 07 17:00:52.080314 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:00:52.080283    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"
Jul 07 17:01:08.575852 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.575816    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"]
Jul 07 17:01:08.576354 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.576191    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="agent"
Jul 07 17:01:08.576354 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.576203    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="agent"
Jul 07 17:01:08.576354 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.576220    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="kserve-container"
Jul 07 17:01:08.576354 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.576226    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="kserve-container"
Jul 07 17:01:08.576354 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.576233    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="kube-rbac-proxy"
Jul 07 17:01:08.576354 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.576238    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="kube-rbac-proxy"
Jul 07 17:01:08.576354 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.576246    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="storage-initializer"
Jul 07 17:01:08.576354 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.576252    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="storage-initializer"
Jul 07 17:01:08.576354 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.576320    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="agent"
Jul 07 17:01:08.576354 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.576329    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="kserve-container"
Jul 07 17:01:08.576354 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.576337    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a83d6124-d922-4c18-b643-15dfd4c8ee4b" containerName="kube-rbac-proxy"
Jul 07 17:01:08.579544 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.579526    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"
Jul 07 17:01:08.582362 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.582333    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-4815b-predictor-serving-cert\""
Jul 07 17:01:08.582517 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.582333    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-4815b-kube-rbac-proxy-sar-config\""
Jul 07 17:01:08.595391 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.595363    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"]
Jul 07 17:01:08.683437 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.683405    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"]
Jul 07 17:01:08.683703 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.683652    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j" podUID="3c168786-3492-473d-97ba-6044c874eb11" containerName="kserve-container" containerID="cri-o://de482e8ed156ca7cafa321c5201cb8f6d56d90d7e0f0e1ec582169b7ac78dbbf" gracePeriod=30
Jul 07 17:01:08.683835 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.683699    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j" podUID="3c168786-3492-473d-97ba-6044c874eb11" containerName="kube-rbac-proxy" containerID="cri-o://5d6e1f7eef72dcd4c0be70600f3fd4e0984a78a6e909396ce22a94191e785019" gracePeriod=30
Jul 07 17:01:08.717317 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.717285    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"]
Jul 07 17:01:08.720295 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.720264    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"
Jul 07 17:01:08.722835 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.722813    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-4815b-predictor-serving-cert\""
Jul 07 17:01:08.722835 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.722825    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-4815b-kube-rbac-proxy-sar-config\""
Jul 07 17:01:08.736397 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.736368    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"]
Jul 07 17:01:08.750596 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.750222    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb\" (UID: \"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"
Jul 07 17:01:08.750596 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.750309    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx6bn\" (UniqueName: \"kubernetes.io/projected/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-kube-api-access-bx6bn\") pod \"isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb\" (UID: \"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"
Jul 07 17:01:08.750596 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.750419    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb\" (UID: \"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"
Jul 07 17:01:08.750596 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.750496    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb\" (UID: \"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"
Jul 07 17:01:08.750596 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.750522    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb\" (UID: \"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"
Jul 07 17:01:08.852033 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.851946    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae9a3d02-0f8c-48a6-81fc-329470cada36-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7\" (UID: \"ae9a3d02-0f8c-48a6-81fc-329470cada36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"
Jul 07 17:01:08.852033 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.851998    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bljfg\" (UniqueName: \"kubernetes.io/projected/ae9a3d02-0f8c-48a6-81fc-329470cada36-kube-api-access-bljfg\") pod \"isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7\" (UID: \"ae9a3d02-0f8c-48a6-81fc-329470cada36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"
Jul 07 17:01:08.852238 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.852048    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae9a3d02-0f8c-48a6-81fc-329470cada36-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7\" (UID: \"ae9a3d02-0f8c-48a6-81fc-329470cada36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"
Jul 07 17:01:08.852238 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.852084    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb\" (UID: \"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"
Jul 07 17:01:08.852347 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.852199    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb\" (UID: \"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"
Jul 07 17:01:08.852710 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.852623    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb\" (UID: \"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"
Jul 07 17:01:08.852943 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.852861    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb\" (UID: \"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"
Jul 07 17:01:08.852943 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.852886    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb\" (UID: \"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"
Jul 07 17:01:08.852943 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.852898    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb\" (UID: \"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"
Jul 07 17:01:08.853178 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.852972    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae9a3d02-0f8c-48a6-81fc-329470cada36-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7\" (UID: \"ae9a3d02-0f8c-48a6-81fc-329470cada36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"
Jul 07 17:01:08.853178 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.853016    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx6bn\" (UniqueName: \"kubernetes.io/projected/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-kube-api-access-bx6bn\") pod \"isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb\" (UID: \"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"
Jul 07 17:01:08.853178 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.853052    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae9a3d02-0f8c-48a6-81fc-329470cada36-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7\" (UID: \"ae9a3d02-0f8c-48a6-81fc-329470cada36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"
Jul 07 17:01:08.853605 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.853578    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb\" (UID: \"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"
Jul 07 17:01:08.855423 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.855400    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb\" (UID: \"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"
Jul 07 17:01:08.876110 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.876083    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx6bn\" (UniqueName: \"kubernetes.io/projected/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-kube-api-access-bx6bn\") pod \"isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb\" (UID: \"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"
Jul 07 17:01:08.889956 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.889932    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"
Jul 07 17:01:08.912733 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.912702    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"]
Jul 07 17:01:08.912970 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.912947    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d" podUID="c4266173-964e-4c0a-98d1-ac9a5379820b" containerName="kserve-container" containerID="cri-o://4e1b3536c6bcbb8bc4e03b0df893b72560d12cb5428f6880b25386caacf310f3" gracePeriod=30
Jul 07 17:01:08.913102 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.913055    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d" podUID="c4266173-964e-4c0a-98d1-ac9a5379820b" containerName="kube-rbac-proxy" containerID="cri-o://e7b68bfdaa1ca04f4421b6cda0e31b5f3260e3ee9aaa0be7a20df2cc38f8a2f9" gracePeriod=30
Jul 07 17:01:08.954721 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.954672    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae9a3d02-0f8c-48a6-81fc-329470cada36-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7\" (UID: \"ae9a3d02-0f8c-48a6-81fc-329470cada36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"
Jul 07 17:01:08.954861 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.954756    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae9a3d02-0f8c-48a6-81fc-329470cada36-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7\" (UID: \"ae9a3d02-0f8c-48a6-81fc-329470cada36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"
Jul 07 17:01:08.954861 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.954796    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae9a3d02-0f8c-48a6-81fc-329470cada36-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7\" (UID: \"ae9a3d02-0f8c-48a6-81fc-329470cada36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"
Jul 07 17:01:08.954861 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.954843    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bljfg\" (UniqueName: \"kubernetes.io/projected/ae9a3d02-0f8c-48a6-81fc-329470cada36-kube-api-access-bljfg\") pod \"isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7\" (UID: \"ae9a3d02-0f8c-48a6-81fc-329470cada36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"
Jul 07 17:01:08.955061 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.954899    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae9a3d02-0f8c-48a6-81fc-329470cada36-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7\" (UID: \"ae9a3d02-0f8c-48a6-81fc-329470cada36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"
Jul 07 17:01:08.955061 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:01:08.955039    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-4815b-predictor-serving-cert" not found
Jul 07 17:01:08.955198 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:01:08.955114    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae9a3d02-0f8c-48a6-81fc-329470cada36-proxy-tls podName:ae9a3d02-0f8c-48a6-81fc-329470cada36 nodeName:}" failed. No retries permitted until 2026-07-07 17:01:09.455092166 +0000 UTC m=+862.468101971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ae9a3d02-0f8c-48a6-81fc-329470cada36-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7" (UID: "ae9a3d02-0f8c-48a6-81fc-329470cada36") : secret "isvc-xgboost-graph-raw-hpa-4815b-predictor-serving-cert" not found
Jul 07 17:01:08.955401 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.955340    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae9a3d02-0f8c-48a6-81fc-329470cada36-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7\" (UID: \"ae9a3d02-0f8c-48a6-81fc-329470cada36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"
Jul 07 17:01:08.955499 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.955469    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae9a3d02-0f8c-48a6-81fc-329470cada36-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7\" (UID: \"ae9a3d02-0f8c-48a6-81fc-329470cada36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"
Jul 07 17:01:08.955723 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.955693    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae9a3d02-0f8c-48a6-81fc-329470cada36-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7\" (UID: \"ae9a3d02-0f8c-48a6-81fc-329470cada36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"
Jul 07 17:01:08.968340 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:08.968319    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bljfg\" (UniqueName: \"kubernetes.io/projected/ae9a3d02-0f8c-48a6-81fc-329470cada36-kube-api-access-bljfg\") pod \"isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7\" (UID: \"ae9a3d02-0f8c-48a6-81fc-329470cada36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"
Jul 07 17:01:09.033238 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:09.033211    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"]
Jul 07 17:01:09.035369 ip-10-0-141-189 kubenswrapper[2583]: W0707 17:01:09.035326    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b1ed7d_e9fe_49ba_93af_53c2d9ecca49.slice/crio-e82fb0ac4a246837becd8c17eb6702f1e7d47b5255c91f88483f393e40444b16 WatchSource:0}: Error finding container e82fb0ac4a246837becd8c17eb6702f1e7d47b5255c91f88483f393e40444b16: Status 404 returned error can't find the container with id e82fb0ac4a246837becd8c17eb6702f1e7d47b5255c91f88483f393e40444b16
Jul 07 17:01:09.441240 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:09.441203    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb" event={"ID":"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49","Type":"ContainerStarted","Data":"b1e306e9b9b2cf1914e2e0a1b1d2d327ddaff4438b3d499c5a12733fcc954a51"}
Jul 07 17:01:09.441451 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:09.441322    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb" event={"ID":"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49","Type":"ContainerStarted","Data":"e82fb0ac4a246837becd8c17eb6702f1e7d47b5255c91f88483f393e40444b16"}
Jul 07 17:01:09.443522 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:09.443494    2583 generic.go:356] "Generic (PLEG): container finished" podID="3c168786-3492-473d-97ba-6044c874eb11" containerID="5d6e1f7eef72dcd4c0be70600f3fd4e0984a78a6e909396ce22a94191e785019" exitCode=2
Jul 07 17:01:09.443641 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:09.443565    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j" event={"ID":"3c168786-3492-473d-97ba-6044c874eb11","Type":"ContainerDied","Data":"5d6e1f7eef72dcd4c0be70600f3fd4e0984a78a6e909396ce22a94191e785019"}
Jul 07 17:01:09.445548 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:09.445519    2583 generic.go:356] "Generic (PLEG): container finished" podID="c4266173-964e-4c0a-98d1-ac9a5379820b" containerID="e7b68bfdaa1ca04f4421b6cda0e31b5f3260e3ee9aaa0be7a20df2cc38f8a2f9" exitCode=2
Jul 07 17:01:09.445673 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:09.445580    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d" event={"ID":"c4266173-964e-4c0a-98d1-ac9a5379820b","Type":"ContainerDied","Data":"e7b68bfdaa1ca04f4421b6cda0e31b5f3260e3ee9aaa0be7a20df2cc38f8a2f9"}
Jul 07 17:01:09.459166 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:09.459131    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae9a3d02-0f8c-48a6-81fc-329470cada36-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7\" (UID: \"ae9a3d02-0f8c-48a6-81fc-329470cada36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"
Jul 07 17:01:09.461606 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:09.461578    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae9a3d02-0f8c-48a6-81fc-329470cada36-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7\" (UID: \"ae9a3d02-0f8c-48a6-81fc-329470cada36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"
Jul 07 17:01:09.631765 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:09.631725    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"
Jul 07 17:01:09.787700 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:09.787411    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"]
Jul 07 17:01:09.790203 ip-10-0-141-189 kubenswrapper[2583]: W0707 17:01:09.790177    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9a3d02_0f8c_48a6_81fc_329470cada36.slice/crio-da5c963c492711b028e74fb5c75e8978376c942e54826e1df6bbe28cec8a20d9 WatchSource:0}: Error finding container da5c963c492711b028e74fb5c75e8978376c942e54826e1df6bbe28cec8a20d9: Status 404 returned error can't find the container with id da5c963c492711b028e74fb5c75e8978376c942e54826e1df6bbe28cec8a20d9
Jul 07 17:01:10.452423 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:10.452384    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7" event={"ID":"ae9a3d02-0f8c-48a6-81fc-329470cada36","Type":"ContainerStarted","Data":"37e7cf6a58677708f4577c4e037790518ebfd6f718cac74a39e01b0dd0fa958d"}
Jul 07 17:01:10.452619 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:10.452448    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7" event={"ID":"ae9a3d02-0f8c-48a6-81fc-329470cada36","Type":"ContainerStarted","Data":"da5c963c492711b028e74fb5c75e8978376c942e54826e1df6bbe28cec8a20d9"}
Jul 07 17:01:12.075707 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:12.075650    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d" podUID="c4266173-964e-4c0a-98d1-ac9a5379820b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 07 17:01:12.079488 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:12.079461    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d" podUID="c4266173-964e-4c0a-98d1-ac9a5379820b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 17:01:12.762930 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:12.762906    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"
Jul 07 17:01:12.892236 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:12.892121    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4266173-964e-4c0a-98d1-ac9a5379820b-kube-rbac-proxy-sar-config\") pod \"c4266173-964e-4c0a-98d1-ac9a5379820b\" (UID: \"c4266173-964e-4c0a-98d1-ac9a5379820b\") "
Jul 07 17:01:12.892236 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:12.892188    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f24kl\" (UniqueName: \"kubernetes.io/projected/c4266173-964e-4c0a-98d1-ac9a5379820b-kube-api-access-f24kl\") pod \"c4266173-964e-4c0a-98d1-ac9a5379820b\" (UID: \"c4266173-964e-4c0a-98d1-ac9a5379820b\") "
Jul 07 17:01:12.892236 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:12.892215    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4266173-964e-4c0a-98d1-ac9a5379820b-cabundle-cert\") pod \"c4266173-964e-4c0a-98d1-ac9a5379820b\" (UID: \"c4266173-964e-4c0a-98d1-ac9a5379820b\") "
Jul 07 17:01:12.892517 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:12.892245    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4266173-964e-4c0a-98d1-ac9a5379820b-proxy-tls\") pod \"c4266173-964e-4c0a-98d1-ac9a5379820b\" (UID: \"c4266173-964e-4c0a-98d1-ac9a5379820b\") "
Jul 07 17:01:12.892517 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:12.892300    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4266173-964e-4c0a-98d1-ac9a5379820b-kserve-provision-location\") pod \"c4266173-964e-4c0a-98d1-ac9a5379820b\" (UID: \"c4266173-964e-4c0a-98d1-ac9a5379820b\") "
Jul 07 17:01:12.892626 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:12.892505    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4266173-964e-4c0a-98d1-ac9a5379820b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c4266173-964e-4c0a-98d1-ac9a5379820b" (UID: "c4266173-964e-4c0a-98d1-ac9a5379820b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:01:12.892669 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:12.892655    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4266173-964e-4c0a-98d1-ac9a5379820b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c4266173-964e-4c0a-98d1-ac9a5379820b" (UID: "c4266173-964e-4c0a-98d1-ac9a5379820b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:01:12.892743 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:12.892708    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4266173-964e-4c0a-98d1-ac9a5379820b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c4266173-964e-4c0a-98d1-ac9a5379820b" (UID: "c4266173-964e-4c0a-98d1-ac9a5379820b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:01:12.894415 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:12.894393    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4266173-964e-4c0a-98d1-ac9a5379820b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c4266173-964e-4c0a-98d1-ac9a5379820b" (UID: "c4266173-964e-4c0a-98d1-ac9a5379820b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:01:12.894495 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:12.894422    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4266173-964e-4c0a-98d1-ac9a5379820b-kube-api-access-f24kl" (OuterVolumeSpecName: "kube-api-access-f24kl") pod "c4266173-964e-4c0a-98d1-ac9a5379820b" (UID: "c4266173-964e-4c0a-98d1-ac9a5379820b"). InnerVolumeSpecName "kube-api-access-f24kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:01:12.978877 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:12.978834    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j" podUID="3c168786-3492-473d-97ba-6044c874eb11" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 07 17:01:12.984234 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:12.984194    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j" podUID="3c168786-3492-473d-97ba-6044c874eb11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 17:01:12.993523 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:12.993502    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4266173-964e-4c0a-98d1-ac9a5379820b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:01:12.993523 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:12.993524    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f24kl\" (UniqueName: \"kubernetes.io/projected/c4266173-964e-4c0a-98d1-ac9a5379820b-kube-api-access-f24kl\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:01:12.993634 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:12.993534    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4266173-964e-4c0a-98d1-ac9a5379820b-cabundle-cert\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:01:12.993634 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:12.993544    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4266173-964e-4c0a-98d1-ac9a5379820b-proxy-tls\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:01:12.993634 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:12.993551    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4266173-964e-4c0a-98d1-ac9a5379820b-kserve-provision-location\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:01:13.467465 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.467435    2583 generic.go:356] "Generic (PLEG): container finished" podID="c4266173-964e-4c0a-98d1-ac9a5379820b" containerID="4e1b3536c6bcbb8bc4e03b0df893b72560d12cb5428f6880b25386caacf310f3" exitCode=0
Jul 07 17:01:13.467922 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.467505    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"
Jul 07 17:01:13.467922 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.467526    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d" event={"ID":"c4266173-964e-4c0a-98d1-ac9a5379820b","Type":"ContainerDied","Data":"4e1b3536c6bcbb8bc4e03b0df893b72560d12cb5428f6880b25386caacf310f3"}
Jul 07 17:01:13.467922 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.467590    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d" event={"ID":"c4266173-964e-4c0a-98d1-ac9a5379820b","Type":"ContainerDied","Data":"ca7105dc9061322f441a2a90bd741ddc5fa5d0d97301ebf5ea6d84e01fc7bb3f"}
Jul 07 17:01:13.467922 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.467649    2583 scope.go:117] "RemoveContainer" containerID="e7b68bfdaa1ca04f4421b6cda0e31b5f3260e3ee9aaa0be7a20df2cc38f8a2f9"
Jul 07 17:01:13.469162 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.469125    2583 generic.go:356] "Generic (PLEG): container finished" podID="35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" containerID="b1e306e9b9b2cf1914e2e0a1b1d2d327ddaff4438b3d499c5a12733fcc954a51" exitCode=0
Jul 07 17:01:13.469288 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.469167    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb" event={"ID":"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49","Type":"ContainerDied","Data":"b1e306e9b9b2cf1914e2e0a1b1d2d327ddaff4438b3d499c5a12733fcc954a51"}
Jul 07 17:01:13.496232 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.496201    2583 scope.go:117] "RemoveContainer" containerID="4e1b3536c6bcbb8bc4e03b0df893b72560d12cb5428f6880b25386caacf310f3"
Jul 07 17:01:13.506762 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.506666    2583 scope.go:117] "RemoveContainer" containerID="39a07001253f382add635e813ba7e15069cd26abb66a69c3275394329af4cadf"
Jul 07 17:01:13.519295 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.519268    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"]
Jul 07 17:01:13.523570 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.523527    2583 scope.go:117] "RemoveContainer" containerID="e7b68bfdaa1ca04f4421b6cda0e31b5f3260e3ee9aaa0be7a20df2cc38f8a2f9"
Jul 07 17:01:13.524484 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:01:13.524049    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7b68bfdaa1ca04f4421b6cda0e31b5f3260e3ee9aaa0be7a20df2cc38f8a2f9\": container with ID starting with e7b68bfdaa1ca04f4421b6cda0e31b5f3260e3ee9aaa0be7a20df2cc38f8a2f9 not found: ID does not exist" containerID="e7b68bfdaa1ca04f4421b6cda0e31b5f3260e3ee9aaa0be7a20df2cc38f8a2f9"
Jul 07 17:01:13.524484 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.524093    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b68bfdaa1ca04f4421b6cda0e31b5f3260e3ee9aaa0be7a20df2cc38f8a2f9"} err="failed to get container status \"e7b68bfdaa1ca04f4421b6cda0e31b5f3260e3ee9aaa0be7a20df2cc38f8a2f9\": rpc error: code = NotFound desc = could not find container \"e7b68bfdaa1ca04f4421b6cda0e31b5f3260e3ee9aaa0be7a20df2cc38f8a2f9\": container with ID starting with e7b68bfdaa1ca04f4421b6cda0e31b5f3260e3ee9aaa0be7a20df2cc38f8a2f9 not found: ID does not exist"
Jul 07 17:01:13.524888 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.524864    2583 scope.go:117] "RemoveContainer" containerID="4e1b3536c6bcbb8bc4e03b0df893b72560d12cb5428f6880b25386caacf310f3"
Jul 07 17:01:13.525329 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:01:13.525301    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e1b3536c6bcbb8bc4e03b0df893b72560d12cb5428f6880b25386caacf310f3\": container with ID starting with 4e1b3536c6bcbb8bc4e03b0df893b72560d12cb5428f6880b25386caacf310f3 not found: ID does not exist" containerID="4e1b3536c6bcbb8bc4e03b0df893b72560d12cb5428f6880b25386caacf310f3"
Jul 07 17:01:13.525481 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.525338    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e1b3536c6bcbb8bc4e03b0df893b72560d12cb5428f6880b25386caacf310f3"} err="failed to get container status \"4e1b3536c6bcbb8bc4e03b0df893b72560d12cb5428f6880b25386caacf310f3\": rpc error: code = NotFound desc = could not find container \"4e1b3536c6bcbb8bc4e03b0df893b72560d12cb5428f6880b25386caacf310f3\": container with ID starting with 4e1b3536c6bcbb8bc4e03b0df893b72560d12cb5428f6880b25386caacf310f3 not found: ID does not exist"
Jul 07 17:01:13.525481 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.525361    2583 scope.go:117] "RemoveContainer" containerID="39a07001253f382add635e813ba7e15069cd26abb66a69c3275394329af4cadf"
Jul 07 17:01:13.525894 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:01:13.525864    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39a07001253f382add635e813ba7e15069cd26abb66a69c3275394329af4cadf\": container with ID starting with 39a07001253f382add635e813ba7e15069cd26abb66a69c3275394329af4cadf not found: ID does not exist" containerID="39a07001253f382add635e813ba7e15069cd26abb66a69c3275394329af4cadf"
Jul 07 17:01:13.525987 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.525899    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39a07001253f382add635e813ba7e15069cd26abb66a69c3275394329af4cadf"} err="failed to get container status \"39a07001253f382add635e813ba7e15069cd26abb66a69c3275394329af4cadf\": rpc error: code = NotFound desc = could not find container \"39a07001253f382add635e813ba7e15069cd26abb66a69c3275394329af4cadf\": container with ID starting with 39a07001253f382add635e813ba7e15069cd26abb66a69c3275394329af4cadf not found: ID does not exist"
Jul 07 17:01:13.528293 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.528236    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-f934a-predictor-6d7f49bfdf-2z54d"]
Jul 07 17:01:13.587785 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.587758    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4266173-964e-4c0a-98d1-ac9a5379820b" path="/var/lib/kubelet/pods/c4266173-964e-4c0a-98d1-ac9a5379820b/volumes"
Jul 07 17:01:13.645617 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.645594    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"
Jul 07 17:01:13.799107 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.799076    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-262k9\" (UniqueName: \"kubernetes.io/projected/3c168786-3492-473d-97ba-6044c874eb11-kube-api-access-262k9\") pod \"3c168786-3492-473d-97ba-6044c874eb11\" (UID: \"3c168786-3492-473d-97ba-6044c874eb11\") "
Jul 07 17:01:13.799107 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.799112    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c168786-3492-473d-97ba-6044c874eb11-cabundle-cert\") pod \"3c168786-3492-473d-97ba-6044c874eb11\" (UID: \"3c168786-3492-473d-97ba-6044c874eb11\") "
Jul 07 17:01:13.799398 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.799198    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c168786-3492-473d-97ba-6044c874eb11-kube-rbac-proxy-sar-config\") pod \"3c168786-3492-473d-97ba-6044c874eb11\" (UID: \"3c168786-3492-473d-97ba-6044c874eb11\") "
Jul 07 17:01:13.799398 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.799227    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c168786-3492-473d-97ba-6044c874eb11-proxy-tls\") pod \"3c168786-3492-473d-97ba-6044c874eb11\" (UID: \"3c168786-3492-473d-97ba-6044c874eb11\") "
Jul 07 17:01:13.799398 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.799273    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c168786-3492-473d-97ba-6044c874eb11-kserve-provision-location\") pod \"3c168786-3492-473d-97ba-6044c874eb11\" (UID: \"3c168786-3492-473d-97ba-6044c874eb11\") "
Jul 07 17:01:13.799578 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.799551    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c168786-3492-473d-97ba-6044c874eb11-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3c168786-3492-473d-97ba-6044c874eb11" (UID: "3c168786-3492-473d-97ba-6044c874eb11"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:01:13.799704 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.799674    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c168786-3492-473d-97ba-6044c874eb11-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3c168786-3492-473d-97ba-6044c874eb11" (UID: "3c168786-3492-473d-97ba-6044c874eb11"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:01:13.799704 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.799680    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c168786-3492-473d-97ba-6044c874eb11-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3c168786-3492-473d-97ba-6044c874eb11" (UID: "3c168786-3492-473d-97ba-6044c874eb11"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:01:13.801464 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.801438    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c168786-3492-473d-97ba-6044c874eb11-kube-api-access-262k9" (OuterVolumeSpecName: "kube-api-access-262k9") pod "3c168786-3492-473d-97ba-6044c874eb11" (UID: "3c168786-3492-473d-97ba-6044c874eb11"). InnerVolumeSpecName "kube-api-access-262k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:01:13.801541 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.801513    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c168786-3492-473d-97ba-6044c874eb11-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3c168786-3492-473d-97ba-6044c874eb11" (UID: "3c168786-3492-473d-97ba-6044c874eb11"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:01:13.900306 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.900276    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c168786-3492-473d-97ba-6044c874eb11-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:01:13.900306 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.900302    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c168786-3492-473d-97ba-6044c874eb11-proxy-tls\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:01:13.900306 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.900311    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c168786-3492-473d-97ba-6044c874eb11-kserve-provision-location\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:01:13.900525 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.900320    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-262k9\" (UniqueName: \"kubernetes.io/projected/3c168786-3492-473d-97ba-6044c874eb11-kube-api-access-262k9\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:01:13.900525 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:13.900328    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c168786-3492-473d-97ba-6044c874eb11-cabundle-cert\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:01:14.475174 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:14.475118    2583 generic.go:356] "Generic (PLEG): container finished" podID="3c168786-3492-473d-97ba-6044c874eb11" containerID="de482e8ed156ca7cafa321c5201cb8f6d56d90d7e0f0e1ec582169b7ac78dbbf" exitCode=0
Jul 07 17:01:14.475707 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:14.475170    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j" event={"ID":"3c168786-3492-473d-97ba-6044c874eb11","Type":"ContainerDied","Data":"de482e8ed156ca7cafa321c5201cb8f6d56d90d7e0f0e1ec582169b7ac78dbbf"}
Jul 07 17:01:14.475707 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:14.475203    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"
Jul 07 17:01:14.475707 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:14.475259    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j" event={"ID":"3c168786-3492-473d-97ba-6044c874eb11","Type":"ContainerDied","Data":"045084f284c2acda78b6fe9a4d5fddbea4bd3d3ad03f72b454bd8d4b5bc2088b"}
Jul 07 17:01:14.475707 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:14.475295    2583 scope.go:117] "RemoveContainer" containerID="5d6e1f7eef72dcd4c0be70600f3fd4e0984a78a6e909396ce22a94191e785019"
Jul 07 17:01:14.477930 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:14.477905    2583 generic.go:356] "Generic (PLEG): container finished" podID="ae9a3d02-0f8c-48a6-81fc-329470cada36" containerID="37e7cf6a58677708f4577c4e037790518ebfd6f718cac74a39e01b0dd0fa958d" exitCode=0
Jul 07 17:01:14.478031 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:14.477973    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7" event={"ID":"ae9a3d02-0f8c-48a6-81fc-329470cada36","Type":"ContainerDied","Data":"37e7cf6a58677708f4577c4e037790518ebfd6f718cac74a39e01b0dd0fa958d"}
Jul 07 17:01:14.480553 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:14.480520    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb" event={"ID":"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49","Type":"ContainerStarted","Data":"037205f77419d2c4d7952f69eefd488b8912b076fd424b720148b9d3421458cf"}
Jul 07 17:01:14.480657 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:14.480591    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb" event={"ID":"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49","Type":"ContainerStarted","Data":"74166dea16f26c4c32702ba938d4b9ba291c01d0a70d3cd3523c62f91f7829f1"}
Jul 07 17:01:14.483409 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:14.483387    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"
Jul 07 17:01:14.483534 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:14.483444    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"
Jul 07 17:01:14.484601 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:14.484577    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb" podUID="35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 17:01:14.484928 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:14.484911    2583 scope.go:117] "RemoveContainer" containerID="de482e8ed156ca7cafa321c5201cb8f6d56d90d7e0f0e1ec582169b7ac78dbbf"
Jul 07 17:01:14.493239 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:14.493221    2583 scope.go:117] "RemoveContainer" containerID="b19a21afa4ad792182cc596ea4211677efb664074dc9dfae25a9c65cad0ccce8"
Jul 07 17:01:14.502222 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:14.502200    2583 scope.go:117] "RemoveContainer" containerID="5d6e1f7eef72dcd4c0be70600f3fd4e0984a78a6e909396ce22a94191e785019"
Jul 07 17:01:14.502477 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:01:14.502459    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d6e1f7eef72dcd4c0be70600f3fd4e0984a78a6e909396ce22a94191e785019\": container with ID starting with 5d6e1f7eef72dcd4c0be70600f3fd4e0984a78a6e909396ce22a94191e785019 not found: ID does not exist" containerID="5d6e1f7eef72dcd4c0be70600f3fd4e0984a78a6e909396ce22a94191e785019"
Jul 07 17:01:14.502518 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:14.502486    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d6e1f7eef72dcd4c0be70600f3fd4e0984a78a6e909396ce22a94191e785019"} err="failed to get container status \"5d6e1f7eef72dcd4c0be70600f3fd4e0984a78a6e909396ce22a94191e785019\": rpc error: code = NotFound desc = could not find container \"5d6e1f7eef72dcd4c0be70600f3fd4e0984a78a6e909396ce22a94191e785019\": container with ID starting with 5d6e1f7eef72dcd4c0be70600f3fd4e0984a78a6e909396ce22a94191e785019 not found: ID does not exist"
Jul 07 17:01:14.502518 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:14.502503    2583 scope.go:117] "RemoveContainer" containerID="de482e8ed156ca7cafa321c5201cb8f6d56d90d7e0f0e1ec582169b7ac78dbbf"
Jul 07 17:01:14.502726 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:01:14.502708    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de482e8ed156ca7cafa321c5201cb8f6d56d90d7e0f0e1ec582169b7ac78dbbf\": container with ID starting with de482e8ed156ca7cafa321c5201cb8f6d56d90d7e0f0e1ec582169b7ac78dbbf not found: ID does not exist" containerID="de482e8ed156ca7cafa321c5201cb8f6d56d90d7e0f0e1ec582169b7ac78dbbf"
Jul 07 17:01:14.502762 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:14.502734    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de482e8ed156ca7cafa321c5201cb8f6d56d90d7e0f0e1ec582169b7ac78dbbf"} err="failed to get container status \"de482e8ed156ca7cafa321c5201cb8f6d56d90d7e0f0e1ec582169b7ac78dbbf\": rpc error: code = NotFound desc = could not find container \"de482e8ed156ca7cafa321c5201cb8f6d56d90d7e0f0e1ec582169b7ac78dbbf\": container with ID starting with de482e8ed156ca7cafa321c5201cb8f6d56d90d7e0f0e1ec582169b7ac78dbbf not found: ID does not exist"
Jul 07 17:01:14.502762 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:14.502749    2583 scope.go:117] "RemoveContainer" containerID="b19a21afa4ad792182cc596ea4211677efb664074dc9dfae25a9c65cad0ccce8"
Jul 07 17:01:14.502973 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:01:14.502957    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b19a21afa4ad792182cc596ea4211677efb664074dc9dfae25a9c65cad0ccce8\": container with ID starting with b19a21afa4ad792182cc596ea4211677efb664074dc9dfae25a9c65cad0ccce8 not found: ID does not exist" containerID="b19a21afa4ad792182cc596ea4211677efb664074dc9dfae25a9c65cad0ccce8"
Jul 07 17:01:14.503009 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:14.502979    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b19a21afa4ad792182cc596ea4211677efb664074dc9dfae25a9c65cad0ccce8"} err="failed to get container status \"b19a21afa4ad792182cc596ea4211677efb664074dc9dfae25a9c65cad0ccce8\": rpc error: code = NotFound desc = could not find container \"b19a21afa4ad792182cc596ea4211677efb664074dc9dfae25a9c65cad0ccce8\": container with ID starting with b19a21afa4ad792182cc596ea4211677efb664074dc9dfae25a9c65cad0ccce8 not found: ID does not exist"
Jul 07 17:01:14.524518 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:14.524491    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"]
Jul 07 17:01:14.532162 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:14.532118    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-f934a-predictor-869cdf6f8d-7m99j"]
Jul 07 17:01:14.556973 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:14.556917    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb" podStartSLOduration=2.884126718 podStartE2EDuration="6.556900129s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.672773411s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:01:14.554949334 +0000 UTC m=+867.567959168" watchObservedRunningTime="2026-07-07 17:01:14.556900129 +0000 UTC m=+867.569909949"
Jul 07 17:01:15.488329 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:15.488295    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7" event={"ID":"ae9a3d02-0f8c-48a6-81fc-329470cada36","Type":"ContainerStarted","Data":"2e276a36cea7aca97f3ffc0fd0bcbaa87b904395b4b4dcfc532b3f63e8a90c17"}
Jul 07 17:01:15.488802 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:15.488352    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7" event={"ID":"ae9a3d02-0f8c-48a6-81fc-329470cada36","Type":"ContainerStarted","Data":"9af043e816382dc433539a7f52d5c167ccdfaf816a6ffaff40502cfeb00184dd"}
Jul 07 17:01:15.488802 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:15.488616    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb" podUID="35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 17:01:15.488913 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:15.488817    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"
Jul 07 17:01:15.488960 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:15.488932    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"
Jul 07 17:01:15.490270 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:15.490248    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7" podUID="ae9a3d02-0f8c-48a6-81fc-329470cada36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 17:01:15.513648 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:15.513589    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7" podStartSLOduration=3.8288711749999997 podStartE2EDuration="7.513571848s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.684700673s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:01:15.511854689 +0000 UTC m=+868.524864509" watchObservedRunningTime="2026-07-07 17:01:15.513571848 +0000 UTC m=+868.526581669"
Jul 07 17:01:15.587529 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:15.587495    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c168786-3492-473d-97ba-6044c874eb11" path="/var/lib/kubelet/pods/3c168786-3492-473d-97ba-6044c874eb11/volumes"
Jul 07 17:01:16.493742 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:16.493705    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7" podUID="ae9a3d02-0f8c-48a6-81fc-329470cada36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 17:01:20.493316 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:20.493290    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"
Jul 07 17:01:20.493874 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:20.493842    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb" podUID="35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 17:01:21.498420 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:21.498393    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"
Jul 07 17:01:21.498976 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:21.498948    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7" podUID="ae9a3d02-0f8c-48a6-81fc-329470cada36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 17:01:30.494011 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:30.493972    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb" podUID="35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 17:01:31.499571 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:31.499527    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7" podUID="ae9a3d02-0f8c-48a6-81fc-329470cada36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 17:01:40.494739 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:40.494648    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb" podUID="35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 17:01:41.499550 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:41.499500    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7" podUID="ae9a3d02-0f8c-48a6-81fc-329470cada36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 17:01:50.494027 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:50.493983    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb" podUID="35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 17:01:51.498927 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:01:51.498885    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7" podUID="ae9a3d02-0f8c-48a6-81fc-329470cada36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 17:02:00.494703 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:00.494658    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb" podUID="35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 17:02:01.499289 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:01.499251    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7" podUID="ae9a3d02-0f8c-48a6-81fc-329470cada36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 17:02:10.494679 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:10.494641    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb" podUID="35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 17:02:11.499718 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:11.499678    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7" podUID="ae9a3d02-0f8c-48a6-81fc-329470cada36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 17:02:20.493927 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:20.493884    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb" podUID="35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 17:02:21.500325 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:21.500294    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"
Jul 07 17:02:30.494953 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:30.494918    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"
Jul 07 17:02:58.918881 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:58.918847    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg"]
Jul 07 17:02:58.919359 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:58.919282    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c168786-3492-473d-97ba-6044c874eb11" containerName="kserve-container"
Jul 07 17:02:58.919359 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:58.919295    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c168786-3492-473d-97ba-6044c874eb11" containerName="kserve-container"
Jul 07 17:02:58.919359 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:58.919303    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c168786-3492-473d-97ba-6044c874eb11" containerName="kube-rbac-proxy"
Jul 07 17:02:58.919359 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:58.919312    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c168786-3492-473d-97ba-6044c874eb11" containerName="kube-rbac-proxy"
Jul 07 17:02:58.919359 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:58.919331    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4266173-964e-4c0a-98d1-ac9a5379820b" containerName="storage-initializer"
Jul 07 17:02:58.919359 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:58.919336    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4266173-964e-4c0a-98d1-ac9a5379820b" containerName="storage-initializer"
Jul 07 17:02:58.919359 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:58.919349    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c168786-3492-473d-97ba-6044c874eb11" containerName="storage-initializer"
Jul 07 17:02:58.919359 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:58.919354    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c168786-3492-473d-97ba-6044c874eb11" containerName="storage-initializer"
Jul 07 17:02:58.919359 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:58.919359    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4266173-964e-4c0a-98d1-ac9a5379820b" containerName="kserve-container"
Jul 07 17:02:58.919359 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:58.919364    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4266173-964e-4c0a-98d1-ac9a5379820b" containerName="kserve-container"
Jul 07 17:02:58.919915 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:58.919379    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4266173-964e-4c0a-98d1-ac9a5379820b" containerName="kube-rbac-proxy"
Jul 07 17:02:58.919915 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:58.919385    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4266173-964e-4c0a-98d1-ac9a5379820b" containerName="kube-rbac-proxy"
Jul 07 17:02:58.919915 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:58.919443    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4266173-964e-4c0a-98d1-ac9a5379820b" containerName="kserve-container"
Jul 07 17:02:58.919915 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:58.919451    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c168786-3492-473d-97ba-6044c874eb11" containerName="kserve-container"
Jul 07 17:02:58.919915 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:58.919457    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c168786-3492-473d-97ba-6044c874eb11" containerName="kube-rbac-proxy"
Jul 07 17:02:58.919915 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:58.919466    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4266173-964e-4c0a-98d1-ac9a5379820b" containerName="kube-rbac-proxy"
Jul 07 17:02:58.921568 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:58.921549    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg"
Jul 07 17:02:58.924250 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:58.924226    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-41dba-predictor-serving-cert\""
Jul 07 17:02:58.924372 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:58.924232    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-41dba-kube-rbac-proxy-sar-config\""
Jul 07 17:02:58.936513 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:58.936481    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg"]
Jul 07 17:02:58.979761 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:58.979729    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"]
Jul 07 17:02:58.980000 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:58.979975    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb" podUID="35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" containerName="kserve-container" containerID="cri-o://74166dea16f26c4c32702ba938d4b9ba291c01d0a70d3cd3523c62f91f7829f1" gracePeriod=30
Jul 07 17:02:58.980081 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:58.980001    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb" podUID="35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" containerName="kube-rbac-proxy" containerID="cri-o://037205f77419d2c4d7952f69eefd488b8912b076fd424b720148b9d3421458cf" gracePeriod=30
Jul 07 17:02:59.031151 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:59.031102    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"]
Jul 07 17:02:59.031394 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:59.031371    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7" podUID="ae9a3d02-0f8c-48a6-81fc-329470cada36" containerName="kserve-container" containerID="cri-o://9af043e816382dc433539a7f52d5c167ccdfaf816a6ffaff40502cfeb00184dd" gracePeriod=30
Jul 07 17:02:59.031464 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:59.031407    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7" podUID="ae9a3d02-0f8c-48a6-81fc-329470cada36" containerName="kube-rbac-proxy" containerID="cri-o://2e276a36cea7aca97f3ffc0fd0bcbaa87b904395b4b4dcfc532b3f63e8a90c17" gracePeriod=30
Jul 07 17:02:59.043910 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:59.043887    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8t8d\" (UniqueName: \"kubernetes.io/projected/bfcd374c-ea10-46d5-8242-9cb049f8867f-kube-api-access-d8t8d\") pod \"message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg\" (UID: \"bfcd374c-ea10-46d5-8242-9cb049f8867f\") " pod="kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg"
Jul 07 17:02:59.043990 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:59.043934    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfcd374c-ea10-46d5-8242-9cb049f8867f-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg\" (UID: \"bfcd374c-ea10-46d5-8242-9cb049f8867f\") " pod="kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg"
Jul 07 17:02:59.044032 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:59.043993    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfcd374c-ea10-46d5-8242-9cb049f8867f-proxy-tls\") pod \"message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg\" (UID: \"bfcd374c-ea10-46d5-8242-9cb049f8867f\") " pod="kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg"
Jul 07 17:02:59.145537 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:59.145499    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8t8d\" (UniqueName: \"kubernetes.io/projected/bfcd374c-ea10-46d5-8242-9cb049f8867f-kube-api-access-d8t8d\") pod \"message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg\" (UID: \"bfcd374c-ea10-46d5-8242-9cb049f8867f\") " pod="kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg"
Jul 07 17:02:59.145695 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:59.145577    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfcd374c-ea10-46d5-8242-9cb049f8867f-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg\" (UID: \"bfcd374c-ea10-46d5-8242-9cb049f8867f\") " pod="kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg"
Jul 07 17:02:59.145695 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:59.145620    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfcd374c-ea10-46d5-8242-9cb049f8867f-proxy-tls\") pod \"message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg\" (UID: \"bfcd374c-ea10-46d5-8242-9cb049f8867f\") " pod="kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg"
Jul 07 17:02:59.145873 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:02:59.145778    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-serving-cert: secret "message-dumper-raw-41dba-predictor-serving-cert" not found
Jul 07 17:02:59.145873 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:02:59.145865    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfcd374c-ea10-46d5-8242-9cb049f8867f-proxy-tls podName:bfcd374c-ea10-46d5-8242-9cb049f8867f nodeName:}" failed. No retries permitted until 2026-07-07 17:02:59.645841632 +0000 UTC m=+972.658851451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bfcd374c-ea10-46d5-8242-9cb049f8867f-proxy-tls") pod "message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg" (UID: "bfcd374c-ea10-46d5-8242-9cb049f8867f") : secret "message-dumper-raw-41dba-predictor-serving-cert" not found
Jul 07 17:02:59.146307 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:59.146287    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfcd374c-ea10-46d5-8242-9cb049f8867f-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg\" (UID: \"bfcd374c-ea10-46d5-8242-9cb049f8867f\") " pod="kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg"
Jul 07 17:02:59.158343 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:59.158320    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8t8d\" (UniqueName: \"kubernetes.io/projected/bfcd374c-ea10-46d5-8242-9cb049f8867f-kube-api-access-d8t8d\") pod \"message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg\" (UID: \"bfcd374c-ea10-46d5-8242-9cb049f8867f\") " pod="kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg"
Jul 07 17:02:59.650078 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:59.650039    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfcd374c-ea10-46d5-8242-9cb049f8867f-proxy-tls\") pod \"message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg\" (UID: \"bfcd374c-ea10-46d5-8242-9cb049f8867f\") " pod="kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg"
Jul 07 17:02:59.652749 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:59.652723    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfcd374c-ea10-46d5-8242-9cb049f8867f-proxy-tls\") pod \"message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg\" (UID: \"bfcd374c-ea10-46d5-8242-9cb049f8867f\") " pod="kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg"
Jul 07 17:02:59.832487 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:59.832427    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg"
Jul 07 17:02:59.953904 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:59.953873    2583 generic.go:356] "Generic (PLEG): container finished" podID="ae9a3d02-0f8c-48a6-81fc-329470cada36" containerID="2e276a36cea7aca97f3ffc0fd0bcbaa87b904395b4b4dcfc532b3f63e8a90c17" exitCode=2
Jul 07 17:02:59.954358 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:59.953949    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7" event={"ID":"ae9a3d02-0f8c-48a6-81fc-329470cada36","Type":"ContainerDied","Data":"2e276a36cea7aca97f3ffc0fd0bcbaa87b904395b4b4dcfc532b3f63e8a90c17"}
Jul 07 17:02:59.955922 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:59.955904    2583 generic.go:356] "Generic (PLEG): container finished" podID="35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" containerID="037205f77419d2c4d7952f69eefd488b8912b076fd424b720148b9d3421458cf" exitCode=2
Jul 07 17:02:59.956003 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:59.955955    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb" event={"ID":"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49","Type":"ContainerDied","Data":"037205f77419d2c4d7952f69eefd488b8912b076fd424b720148b9d3421458cf"}
Jul 07 17:02:59.981125 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:59.981101    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg"]
Jul 07 17:02:59.982787 ip-10-0-141-189 kubenswrapper[2583]: W0707 17:02:59.982763    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfcd374c_ea10_46d5_8242_9cb049f8867f.slice/crio-9f1fcb8c44bc31eee1b9244989e5fd25088b0de26f8ff7996c5ce9f5ea938070 WatchSource:0}: Error finding container 9f1fcb8c44bc31eee1b9244989e5fd25088b0de26f8ff7996c5ce9f5ea938070: Status 404 returned error can't find the container with id 9f1fcb8c44bc31eee1b9244989e5fd25088b0de26f8ff7996c5ce9f5ea938070
Jul 07 17:02:59.984724 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:02:59.984708    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 17:03:00.489485 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:00.489442    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb" podUID="35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 07 17:03:00.493905 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:00.493877    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb" podUID="35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 17:03:00.962979 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:00.962946    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg" event={"ID":"bfcd374c-ea10-46d5-8242-9cb049f8867f","Type":"ContainerStarted","Data":"9f1fcb8c44bc31eee1b9244989e5fd25088b0de26f8ff7996c5ce9f5ea938070"}
Jul 07 17:03:01.494188 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:01.494082    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7" podUID="ae9a3d02-0f8c-48a6-81fc-329470cada36" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 07 17:03:01.499463 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:01.499439    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7" podUID="ae9a3d02-0f8c-48a6-81fc-329470cada36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 17:03:01.969336 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:01.969302    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg" event={"ID":"bfcd374c-ea10-46d5-8242-9cb049f8867f","Type":"ContainerStarted","Data":"30d9c4134804b2a192b6001141aa542b2b2a24b8c3c3accadbb9eb39b0fe1259"}
Jul 07 17:03:01.969725 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:01.969368    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg" event={"ID":"bfcd374c-ea10-46d5-8242-9cb049f8867f","Type":"ContainerStarted","Data":"6c69f5e829d46ef560dd7f3526853c52abb1605e3b1c37e6dcbd284243d1b8f8"}
Jul 07 17:03:01.969725 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:01.969394    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg"
Jul 07 17:03:01.969725 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:01.969414    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg"
Jul 07 17:03:01.971114 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:01.971094    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg"
Jul 07 17:03:02.023557 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:02.023507    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg" podStartSLOduration=2.835386712 podStartE2EDuration="4.023490321s" totalImagesPullingTime="1.188103609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:02:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:03:01.994932306 +0000 UTC m=+975.007942126" watchObservedRunningTime="2026-07-07 17:03:02.023490321 +0000 UTC m=+975.036500139"
Jul 07 17:03:02.975642 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:02.975609    2583 generic.go:356] "Generic (PLEG): container finished" podID="ae9a3d02-0f8c-48a6-81fc-329470cada36" containerID="9af043e816382dc433539a7f52d5c167ccdfaf816a6ffaff40502cfeb00184dd" exitCode=0
Jul 07 17:03:02.976109 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:02.975686    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7" event={"ID":"ae9a3d02-0f8c-48a6-81fc-329470cada36","Type":"ContainerDied","Data":"9af043e816382dc433539a7f52d5c167ccdfaf816a6ffaff40502cfeb00184dd"}
Jul 07 17:03:02.988766 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:02.988748    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"
Jul 07 17:03:03.082675 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:03.082590    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae9a3d02-0f8c-48a6-81fc-329470cada36-kserve-provision-location\") pod \"ae9a3d02-0f8c-48a6-81fc-329470cada36\" (UID: \"ae9a3d02-0f8c-48a6-81fc-329470cada36\") "
Jul 07 17:03:03.082675 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:03.082649    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae9a3d02-0f8c-48a6-81fc-329470cada36-proxy-tls\") pod \"ae9a3d02-0f8c-48a6-81fc-329470cada36\" (UID: \"ae9a3d02-0f8c-48a6-81fc-329470cada36\") "
Jul 07 17:03:03.082891 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:03.082679    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae9a3d02-0f8c-48a6-81fc-329470cada36-kube-rbac-proxy-sar-config\") pod \"ae9a3d02-0f8c-48a6-81fc-329470cada36\" (UID: \"ae9a3d02-0f8c-48a6-81fc-329470cada36\") "
Jul 07 17:03:03.082891 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:03.082750    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae9a3d02-0f8c-48a6-81fc-329470cada36-cabundle-cert\") pod \"ae9a3d02-0f8c-48a6-81fc-329470cada36\" (UID: \"ae9a3d02-0f8c-48a6-81fc-329470cada36\") "
Jul 07 17:03:03.082891 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:03.082787    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bljfg\" (UniqueName: \"kubernetes.io/projected/ae9a3d02-0f8c-48a6-81fc-329470cada36-kube-api-access-bljfg\") pod \"ae9a3d02-0f8c-48a6-81fc-329470cada36\" (UID: \"ae9a3d02-0f8c-48a6-81fc-329470cada36\") "
Jul 07 17:03:03.083217 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:03.083170    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae9a3d02-0f8c-48a6-81fc-329470cada36-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ae9a3d02-0f8c-48a6-81fc-329470cada36" (UID: "ae9a3d02-0f8c-48a6-81fc-329470cada36"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:03:03.083217 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:03.083183    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae9a3d02-0f8c-48a6-81fc-329470cada36-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ae9a3d02-0f8c-48a6-81fc-329470cada36" (UID: "ae9a3d02-0f8c-48a6-81fc-329470cada36"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:03:03.083361 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:03.083285    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9a3d02-0f8c-48a6-81fc-329470cada36-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae9a3d02-0f8c-48a6-81fc-329470cada36" (UID: "ae9a3d02-0f8c-48a6-81fc-329470cada36"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:03:03.084954 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:03.084934    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae9a3d02-0f8c-48a6-81fc-329470cada36-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ae9a3d02-0f8c-48a6-81fc-329470cada36" (UID: "ae9a3d02-0f8c-48a6-81fc-329470cada36"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:03:03.085052 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:03.085036    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae9a3d02-0f8c-48a6-81fc-329470cada36-kube-api-access-bljfg" (OuterVolumeSpecName: "kube-api-access-bljfg") pod "ae9a3d02-0f8c-48a6-81fc-329470cada36" (UID: "ae9a3d02-0f8c-48a6-81fc-329470cada36"). InnerVolumeSpecName "kube-api-access-bljfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:03:03.183457 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:03.183421    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae9a3d02-0f8c-48a6-81fc-329470cada36-kserve-provision-location\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:03:03.183457 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:03.183453    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae9a3d02-0f8c-48a6-81fc-329470cada36-proxy-tls\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:03:03.183651 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:03.183468    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae9a3d02-0f8c-48a6-81fc-329470cada36-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:03:03.183651 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:03.183477    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae9a3d02-0f8c-48a6-81fc-329470cada36-cabundle-cert\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:03:03.183651 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:03.183485    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bljfg\" (UniqueName: \"kubernetes.io/projected/ae9a3d02-0f8c-48a6-81fc-329470cada36-kube-api-access-bljfg\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:03:03.983294 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:03.983261    2583 generic.go:356] "Generic (PLEG): container finished" podID="35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" containerID="74166dea16f26c4c32702ba938d4b9ba291c01d0a70d3cd3523c62f91f7829f1" exitCode=0
Jul 07 17:03:03.983761 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:03.983336    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb" event={"ID":"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49","Type":"ContainerDied","Data":"74166dea16f26c4c32702ba938d4b9ba291c01d0a70d3cd3523c62f91f7829f1"}
Jul 07 17:03:03.985480 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:03.985452    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7" event={"ID":"ae9a3d02-0f8c-48a6-81fc-329470cada36","Type":"ContainerDied","Data":"da5c963c492711b028e74fb5c75e8978376c942e54826e1df6bbe28cec8a20d9"}
Jul 07 17:03:03.985597 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:03.985518    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"
Jul 07 17:03:03.985597 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:03.985522    2583 scope.go:117] "RemoveContainer" containerID="2e276a36cea7aca97f3ffc0fd0bcbaa87b904395b4b4dcfc532b3f63e8a90c17"
Jul 07 17:03:03.994928 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:03.994897    2583 scope.go:117] "RemoveContainer" containerID="9af043e816382dc433539a7f52d5c167ccdfaf816a6ffaff40502cfeb00184dd"
Jul 07 17:03:04.002983 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:04.002964    2583 scope.go:117] "RemoveContainer" containerID="37e7cf6a58677708f4577c4e037790518ebfd6f718cac74a39e01b0dd0fa958d"
Jul 07 17:03:04.012981 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:04.012957    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"]
Jul 07 17:03:04.014768 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:04.014745    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-4815b-predictor-64d779f8b9-8hrh7"]
Jul 07 17:03:04.034859 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:04.034835    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"
Jul 07 17:03:04.091406 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:04.091375    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-proxy-tls\") pod \"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49\" (UID: \"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49\") "
Jul 07 17:03:04.091608 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:04.091435    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-kserve-provision-location\") pod \"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49\" (UID: \"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49\") "
Jul 07 17:03:04.091608 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:04.091492    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-cabundle-cert\") pod \"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49\" (UID: \"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49\") "
Jul 07 17:03:04.091608 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:04.091549    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bx6bn\" (UniqueName: \"kubernetes.io/projected/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-kube-api-access-bx6bn\") pod \"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49\" (UID: \"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49\") "
Jul 07 17:03:04.091608 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:04.091582    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-kube-rbac-proxy-sar-config\") pod \"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49\" (UID: \"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49\") "
Jul 07 17:03:04.091915 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:04.091871    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" (UID: "35b1ed7d-e9fe-49ba-93af-53c2d9ecca49"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:03:04.091915 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:04.091892    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" (UID: "35b1ed7d-e9fe-49ba-93af-53c2d9ecca49"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:03:04.092112 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:04.091974    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" (UID: "35b1ed7d-e9fe-49ba-93af-53c2d9ecca49"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:03:04.093792 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:04.093770    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-kube-api-access-bx6bn" (OuterVolumeSpecName: "kube-api-access-bx6bn") pod "35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" (UID: "35b1ed7d-e9fe-49ba-93af-53c2d9ecca49"). InnerVolumeSpecName "kube-api-access-bx6bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:03:04.093886 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:04.093864    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" (UID: "35b1ed7d-e9fe-49ba-93af-53c2d9ecca49"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:03:04.193239 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:04.193207    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-proxy-tls\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:03:04.193239 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:04.193234    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-kserve-provision-location\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:03:04.193239 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:04.193244    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-cabundle-cert\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:03:04.193467 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:04.193253    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx6bn\" (UniqueName: \"kubernetes.io/projected/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-kube-api-access-bx6bn\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:03:04.193467 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:04.193261    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:03:04.993656 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:04.993620    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb" event={"ID":"35b1ed7d-e9fe-49ba-93af-53c2d9ecca49","Type":"ContainerDied","Data":"e82fb0ac4a246837becd8c17eb6702f1e7d47b5255c91f88483f393e40444b16"}
Jul 07 17:03:04.993656 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:04.993645    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"
Jul 07 17:03:04.994251 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:04.993693    2583 scope.go:117] "RemoveContainer" containerID="037205f77419d2c4d7952f69eefd488b8912b076fd424b720148b9d3421458cf"
Jul 07 17:03:05.003436 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:05.003418    2583 scope.go:117] "RemoveContainer" containerID="74166dea16f26c4c32702ba938d4b9ba291c01d0a70d3cd3523c62f91f7829f1"
Jul 07 17:03:05.010989 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:05.010973    2583 scope.go:117] "RemoveContainer" containerID="b1e306e9b9b2cf1914e2e0a1b1d2d327ddaff4438b3d499c5a12733fcc954a51"
Jul 07 17:03:05.019390 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:05.019366    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"]
Jul 07 17:03:05.024771 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:05.024748    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-4815b-predictor-5cc558bdd-vl6jb"]
Jul 07 17:03:05.586686 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:05.586653    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" path="/var/lib/kubelet/pods/35b1ed7d-e9fe-49ba-93af-53c2d9ecca49/volumes"
Jul 07 17:03:05.587166 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:05.587153    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae9a3d02-0f8c-48a6-81fc-329470cada36" path="/var/lib/kubelet/pods/ae9a3d02-0f8c-48a6-81fc-329470cada36/volumes"
Jul 07 17:03:08.990285 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:08.990210    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg"
Jul 07 17:03:18.946063 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:18.946029    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"]
Jul 07 17:03:18.946552 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:18.946387    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" containerName="kserve-container"
Jul 07 17:03:18.946552 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:18.946398    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" containerName="kserve-container"
Jul 07 17:03:18.946552 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:18.946409    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae9a3d02-0f8c-48a6-81fc-329470cada36" containerName="kserve-container"
Jul 07 17:03:18.946552 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:18.946414    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae9a3d02-0f8c-48a6-81fc-329470cada36" containerName="kserve-container"
Jul 07 17:03:18.946552 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:18.946421    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" containerName="storage-initializer"
Jul 07 17:03:18.946552 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:18.946426    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" containerName="storage-initializer"
Jul 07 17:03:18.946552 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:18.946438    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae9a3d02-0f8c-48a6-81fc-329470cada36" containerName="kube-rbac-proxy"
Jul 07 17:03:18.946552 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:18.946444    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae9a3d02-0f8c-48a6-81fc-329470cada36" containerName="kube-rbac-proxy"
Jul 07 17:03:18.946552 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:18.946453    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae9a3d02-0f8c-48a6-81fc-329470cada36" containerName="storage-initializer"
Jul 07 17:03:18.946552 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:18.946458    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae9a3d02-0f8c-48a6-81fc-329470cada36" containerName="storage-initializer"
Jul 07 17:03:18.946552 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:18.946465    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" containerName="kube-rbac-proxy"
Jul 07 17:03:18.946552 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:18.946470    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" containerName="kube-rbac-proxy"
Jul 07 17:03:18.946552 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:18.946548    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" containerName="kube-rbac-proxy"
Jul 07 17:03:18.946552 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:18.946559    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae9a3d02-0f8c-48a6-81fc-329470cada36" containerName="kube-rbac-proxy"
Jul 07 17:03:18.946974 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:18.946569    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="35b1ed7d-e9fe-49ba-93af-53c2d9ecca49" containerName="kserve-container"
Jul 07 17:03:18.946974 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:18.946574    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae9a3d02-0f8c-48a6-81fc-329470cada36" containerName="kserve-container"
Jul 07 17:03:18.949122 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:18.949101    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:03:18.952028 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:18.952005    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-41dba-predictor-serving-cert\""
Jul 07 17:03:18.952120 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:18.952010    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 17:03:18.952120 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:18.952095    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-41dba-kube-rbac-proxy-sar-config\""
Jul 07 17:03:18.952314 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:18.952296    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 17:03:18.966807 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:18.966784    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"]
Jul 07 17:03:19.129312 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:19.129275    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43ead570-570e-4b3d-97e4-d019938a2076-cabundle-cert\") pod \"isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm\" (UID: \"43ead570-570e-4b3d-97e4-d019938a2076\") " pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:03:19.129312 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:19.129315    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29hpp\" (UniqueName: \"kubernetes.io/projected/43ead570-570e-4b3d-97e4-d019938a2076-kube-api-access-29hpp\") pod \"isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm\" (UID: \"43ead570-570e-4b3d-97e4-d019938a2076\") " pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:03:19.129546 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:19.129333    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43ead570-570e-4b3d-97e4-d019938a2076-kserve-provision-location\") pod \"isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm\" (UID: \"43ead570-570e-4b3d-97e4-d019938a2076\") " pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:03:19.129546 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:19.129359    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43ead570-570e-4b3d-97e4-d019938a2076-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm\" (UID: \"43ead570-570e-4b3d-97e4-d019938a2076\") " pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:03:19.129546 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:19.129412    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43ead570-570e-4b3d-97e4-d019938a2076-proxy-tls\") pod \"isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm\" (UID: \"43ead570-570e-4b3d-97e4-d019938a2076\") " pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:03:19.230769 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:19.230674    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43ead570-570e-4b3d-97e4-d019938a2076-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm\" (UID: \"43ead570-570e-4b3d-97e4-d019938a2076\") " pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:03:19.230769 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:19.230729    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43ead570-570e-4b3d-97e4-d019938a2076-proxy-tls\") pod \"isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm\" (UID: \"43ead570-570e-4b3d-97e4-d019938a2076\") " pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:03:19.231009 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:19.230829    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43ead570-570e-4b3d-97e4-d019938a2076-cabundle-cert\") pod \"isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm\" (UID: \"43ead570-570e-4b3d-97e4-d019938a2076\") " pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:03:19.231009 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:19.230858    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29hpp\" (UniqueName: \"kubernetes.io/projected/43ead570-570e-4b3d-97e4-d019938a2076-kube-api-access-29hpp\") pod \"isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm\" (UID: \"43ead570-570e-4b3d-97e4-d019938a2076\") " pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:03:19.231009 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:19.230881    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43ead570-570e-4b3d-97e4-d019938a2076-kserve-provision-location\") pod \"isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm\" (UID: \"43ead570-570e-4b3d-97e4-d019938a2076\") " pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:03:19.231009 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:03:19.230933    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-serving-cert: secret "isvc-logger-raw-41dba-predictor-serving-cert" not found
Jul 07 17:03:19.231251 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:03:19.231027    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43ead570-570e-4b3d-97e4-d019938a2076-proxy-tls podName:43ead570-570e-4b3d-97e4-d019938a2076 nodeName:}" failed. No retries permitted until 2026-07-07 17:03:19.731005644 +0000 UTC m=+992.744015472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/43ead570-570e-4b3d-97e4-d019938a2076-proxy-tls") pod "isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" (UID: "43ead570-570e-4b3d-97e4-d019938a2076") : secret "isvc-logger-raw-41dba-predictor-serving-cert" not found
Jul 07 17:03:19.231251 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:19.231237    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43ead570-570e-4b3d-97e4-d019938a2076-kserve-provision-location\") pod \"isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm\" (UID: \"43ead570-570e-4b3d-97e4-d019938a2076\") " pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:03:19.231511 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:19.231493    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43ead570-570e-4b3d-97e4-d019938a2076-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm\" (UID: \"43ead570-570e-4b3d-97e4-d019938a2076\") " pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:03:19.231552 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:19.231498    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43ead570-570e-4b3d-97e4-d019938a2076-cabundle-cert\") pod \"isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm\" (UID: \"43ead570-570e-4b3d-97e4-d019938a2076\") " pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:03:19.244645 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:19.244622    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29hpp\" (UniqueName: \"kubernetes.io/projected/43ead570-570e-4b3d-97e4-d019938a2076-kube-api-access-29hpp\") pod \"isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm\" (UID: \"43ead570-570e-4b3d-97e4-d019938a2076\") " pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:03:19.736058 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:19.736024    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43ead570-570e-4b3d-97e4-d019938a2076-proxy-tls\") pod \"isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm\" (UID: \"43ead570-570e-4b3d-97e4-d019938a2076\") " pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:03:19.738385 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:19.738362    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43ead570-570e-4b3d-97e4-d019938a2076-proxy-tls\") pod \"isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm\" (UID: \"43ead570-570e-4b3d-97e4-d019938a2076\") " pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:03:19.859741 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:19.859705    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:03:20.003566 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:20.003544    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"]
Jul 07 17:03:20.005377 ip-10-0-141-189 kubenswrapper[2583]: W0707 17:03:20.005350    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ead570_570e_4b3d_97e4_d019938a2076.slice/crio-a742d6116a04c52f84fe8f9cb78673b7a0c2b953a96831c79f74626bf98965fd WatchSource:0}: Error finding container a742d6116a04c52f84fe8f9cb78673b7a0c2b953a96831c79f74626bf98965fd: Status 404 returned error can't find the container with id a742d6116a04c52f84fe8f9cb78673b7a0c2b953a96831c79f74626bf98965fd
Jul 07 17:03:20.062387 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:20.062358    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" event={"ID":"43ead570-570e-4b3d-97e4-d019938a2076","Type":"ContainerStarted","Data":"9f325ca31d44e4ef81a6ce551ff94b9b2578605be62f5134f2494c8200e62b6d"}
Jul 07 17:03:20.062529 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:20.062432    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" event={"ID":"43ead570-570e-4b3d-97e4-d019938a2076","Type":"ContainerStarted","Data":"a742d6116a04c52f84fe8f9cb78673b7a0c2b953a96831c79f74626bf98965fd"}
Jul 07 17:03:24.082094 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:24.082056    2583 generic.go:356] "Generic (PLEG): container finished" podID="43ead570-570e-4b3d-97e4-d019938a2076" containerID="9f325ca31d44e4ef81a6ce551ff94b9b2578605be62f5134f2494c8200e62b6d" exitCode=0
Jul 07 17:03:24.082506 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:24.082147    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" event={"ID":"43ead570-570e-4b3d-97e4-d019938a2076","Type":"ContainerDied","Data":"9f325ca31d44e4ef81a6ce551ff94b9b2578605be62f5134f2494c8200e62b6d"}
Jul 07 17:03:25.089152 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:25.089099    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" event={"ID":"43ead570-570e-4b3d-97e4-d019938a2076","Type":"ContainerStarted","Data":"83d49a47efe867203f83a80c560220b539be918a557bb6f472f1c7c8fcda86d4"}
Jul 07 17:03:25.089590 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:25.089194    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" event={"ID":"43ead570-570e-4b3d-97e4-d019938a2076","Type":"ContainerStarted","Data":"210d7e067c0a6bfc55685bb1167b6ba7675a5e6d78918242041d6d1a2abd1f32"}
Jul 07 17:03:25.089590 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:25.089224    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" event={"ID":"43ead570-570e-4b3d-97e4-d019938a2076","Type":"ContainerStarted","Data":"8217d86c3626c2bad6bc2aa9e6f91fcaff3a08c0961a3722ad4c43693bdda8e5"}
Jul 07 17:03:25.089590 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:25.089542    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:03:25.089728 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:25.089611    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:03:25.089728 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:25.089645    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:03:25.090982 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:25.090899    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 17:03:25.091547 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:25.091526    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:03:25.114926 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:25.114874    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podStartSLOduration=3.210078441 podStartE2EDuration="7.11485817s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.904779729s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:03:25.113459813 +0000 UTC m=+998.126469632" watchObservedRunningTime="2026-07-07 17:03:25.11485817 +0000 UTC m=+998.127867991"
Jul 07 17:03:26.094783 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:26.094740    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 17:03:26.095208 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:26.095120    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:03:31.099210 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:31.099180    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:03:31.099753 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:31.099721    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 17:03:31.099984 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:31.099959    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:03:41.099843 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:41.099803    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 17:03:41.100380 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:41.100354    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:03:51.100444 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:51.100388    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 17:03:51.100920 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:03:51.100834    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:04:01.099916 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:01.099873    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 17:04:01.100386 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:01.100340    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:04:11.099627 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:11.099587    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 17:04:11.100052 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:11.100034    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:04:21.100263 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:21.100223    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 17:04:21.100783 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:21.100756    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:04:31.099643 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:31.099597    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 17:04:31.100219 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:31.099945    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:04:41.100390 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:41.100313    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:04:41.100767 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:41.100402    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:04:53.932192 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:53.932158    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg_bfcd374c-ea10-46d5-8242-9cb049f8867f/kserve-container/0.log"
Jul 07 17:04:54.373834 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.373801    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"]
Jul 07 17:04:54.377579 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.377562    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"
Jul 07 17:04:54.380589 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.380564    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-dd75c-kube-rbac-proxy-sar-config\""
Jul 07 17:04:54.380720 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.380593    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-dd75c-predictor-serving-cert\""
Jul 07 17:04:54.387646 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.387624    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"]
Jul 07 17:04:54.387881 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.387856    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="kserve-container" containerID="cri-o://8217d86c3626c2bad6bc2aa9e6f91fcaff3a08c0961a3722ad4c43693bdda8e5" gracePeriod=30
Jul 07 17:04:54.387929 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.387870    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="agent" containerID="cri-o://83d49a47efe867203f83a80c560220b539be918a557bb6f472f1c7c8fcda86d4" gracePeriod=30
Jul 07 17:04:54.387929 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.387888    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="kube-rbac-proxy" containerID="cri-o://210d7e067c0a6bfc55685bb1167b6ba7675a5e6d78918242041d6d1a2abd1f32" gracePeriod=30
Jul 07 17:04:54.393934 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.393824    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"]
Jul 07 17:04:54.459723 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.459690    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg"]
Jul 07 17:04:54.459985 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.459958    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg" podUID="bfcd374c-ea10-46d5-8242-9cb049f8867f" containerName="kserve-container" containerID="cri-o://6c69f5e829d46ef560dd7f3526853c52abb1605e3b1c37e6dcbd284243d1b8f8" gracePeriod=30
Jul 07 17:04:54.460098 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.459979    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg" podUID="bfcd374c-ea10-46d5-8242-9cb049f8867f" containerName="kube-rbac-proxy" containerID="cri-o://30d9c4134804b2a192b6001141aa542b2b2a24b8c3c3accadbb9eb39b0fe1259" gracePeriod=30
Jul 07 17:04:54.479242 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.479215    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b52f9c73-60cf-4897-a8c1-2ae824819a23-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc\" (UID: \"b52f9c73-60cf-4897-a8c1-2ae824819a23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"
Jul 07 17:04:54.479347 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.479265    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b52f9c73-60cf-4897-a8c1-2ae824819a23-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc\" (UID: \"b52f9c73-60cf-4897-a8c1-2ae824819a23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"
Jul 07 17:04:54.479347 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.479302    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b52f9c73-60cf-4897-a8c1-2ae824819a23-cabundle-cert\") pod \"isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc\" (UID: \"b52f9c73-60cf-4897-a8c1-2ae824819a23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"
Jul 07 17:04:54.479430 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.479400    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pkhb\" (UniqueName: \"kubernetes.io/projected/b52f9c73-60cf-4897-a8c1-2ae824819a23-kube-api-access-6pkhb\") pod \"isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc\" (UID: \"b52f9c73-60cf-4897-a8c1-2ae824819a23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"
Jul 07 17:04:54.479430 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.479419    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b52f9c73-60cf-4897-a8c1-2ae824819a23-proxy-tls\") pod \"isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc\" (UID: \"b52f9c73-60cf-4897-a8c1-2ae824819a23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"
Jul 07 17:04:54.580966 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.580935    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b52f9c73-60cf-4897-a8c1-2ae824819a23-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc\" (UID: \"b52f9c73-60cf-4897-a8c1-2ae824819a23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"
Jul 07 17:04:54.581055 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.581012    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b52f9c73-60cf-4897-a8c1-2ae824819a23-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc\" (UID: \"b52f9c73-60cf-4897-a8c1-2ae824819a23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"
Jul 07 17:04:54.581055 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.581046    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b52f9c73-60cf-4897-a8c1-2ae824819a23-cabundle-cert\") pod \"isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc\" (UID: \"b52f9c73-60cf-4897-a8c1-2ae824819a23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"
Jul 07 17:04:54.581586 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.581227    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pkhb\" (UniqueName: \"kubernetes.io/projected/b52f9c73-60cf-4897-a8c1-2ae824819a23-kube-api-access-6pkhb\") pod \"isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc\" (UID: \"b52f9c73-60cf-4897-a8c1-2ae824819a23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"
Jul 07 17:04:54.581586 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.581261    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b52f9c73-60cf-4897-a8c1-2ae824819a23-proxy-tls\") pod \"isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc\" (UID: \"b52f9c73-60cf-4897-a8c1-2ae824819a23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"
Jul 07 17:04:54.581774 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.581626    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b52f9c73-60cf-4897-a8c1-2ae824819a23-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc\" (UID: \"b52f9c73-60cf-4897-a8c1-2ae824819a23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"
Jul 07 17:04:54.581774 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.581636    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b52f9c73-60cf-4897-a8c1-2ae824819a23-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc\" (UID: \"b52f9c73-60cf-4897-a8c1-2ae824819a23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"
Jul 07 17:04:54.581878 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.581824    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b52f9c73-60cf-4897-a8c1-2ae824819a23-cabundle-cert\") pod \"isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc\" (UID: \"b52f9c73-60cf-4897-a8c1-2ae824819a23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"
Jul 07 17:04:54.583852 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.583829    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b52f9c73-60cf-4897-a8c1-2ae824819a23-proxy-tls\") pod \"isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc\" (UID: \"b52f9c73-60cf-4897-a8c1-2ae824819a23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"
Jul 07 17:04:54.593023 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.592979    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pkhb\" (UniqueName: \"kubernetes.io/projected/b52f9c73-60cf-4897-a8c1-2ae824819a23-kube-api-access-6pkhb\") pod \"isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc\" (UID: \"b52f9c73-60cf-4897-a8c1-2ae824819a23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"
Jul 07 17:04:54.687836 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.687801    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"
Jul 07 17:04:54.706431 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.706406    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg"
Jul 07 17:04:54.842212 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.842190    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"]
Jul 07 17:04:54.844299 ip-10-0-141-189 kubenswrapper[2583]: W0707 17:04:54.844271    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb52f9c73_60cf_4897_a8c1_2ae824819a23.slice/crio-ad1145e0466b52ba4897521a2121665bf4f502c4eff8d7173879b141c2433f5a WatchSource:0}: Error finding container ad1145e0466b52ba4897521a2121665bf4f502c4eff8d7173879b141c2433f5a: Status 404 returned error can't find the container with id ad1145e0466b52ba4897521a2121665bf4f502c4eff8d7173879b141c2433f5a
Jul 07 17:04:54.884065 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.883997    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfcd374c-ea10-46d5-8242-9cb049f8867f-proxy-tls\") pod \"bfcd374c-ea10-46d5-8242-9cb049f8867f\" (UID: \"bfcd374c-ea10-46d5-8242-9cb049f8867f\") "
Jul 07 17:04:54.884065 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.884045    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8t8d\" (UniqueName: \"kubernetes.io/projected/bfcd374c-ea10-46d5-8242-9cb049f8867f-kube-api-access-d8t8d\") pod \"bfcd374c-ea10-46d5-8242-9cb049f8867f\" (UID: \"bfcd374c-ea10-46d5-8242-9cb049f8867f\") "
Jul 07 17:04:54.884266 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.884080    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfcd374c-ea10-46d5-8242-9cb049f8867f-kube-rbac-proxy-sar-config\") pod \"bfcd374c-ea10-46d5-8242-9cb049f8867f\" (UID: \"bfcd374c-ea10-46d5-8242-9cb049f8867f\") "
Jul 07 17:04:54.884738 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.884709    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfcd374c-ea10-46d5-8242-9cb049f8867f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bfcd374c-ea10-46d5-8242-9cb049f8867f" (UID: "bfcd374c-ea10-46d5-8242-9cb049f8867f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:04:54.886234 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.886209    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfcd374c-ea10-46d5-8242-9cb049f8867f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bfcd374c-ea10-46d5-8242-9cb049f8867f" (UID: "bfcd374c-ea10-46d5-8242-9cb049f8867f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:04:54.886460 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.886438    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfcd374c-ea10-46d5-8242-9cb049f8867f-kube-api-access-d8t8d" (OuterVolumeSpecName: "kube-api-access-d8t8d") pod "bfcd374c-ea10-46d5-8242-9cb049f8867f" (UID: "bfcd374c-ea10-46d5-8242-9cb049f8867f"). InnerVolumeSpecName "kube-api-access-d8t8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:04:54.985175 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.985116    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfcd374c-ea10-46d5-8242-9cb049f8867f-proxy-tls\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:04:54.985175 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.985172    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8t8d\" (UniqueName: \"kubernetes.io/projected/bfcd374c-ea10-46d5-8242-9cb049f8867f-kube-api-access-d8t8d\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:04:54.985175 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:54.985183    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfcd374c-ea10-46d5-8242-9cb049f8867f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:04:55.482472 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:55.482439    2583 generic.go:356] "Generic (PLEG): container finished" podID="43ead570-570e-4b3d-97e4-d019938a2076" containerID="210d7e067c0a6bfc55685bb1167b6ba7675a5e6d78918242041d6d1a2abd1f32" exitCode=2
Jul 07 17:04:55.482668 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:55.482518    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" event={"ID":"43ead570-570e-4b3d-97e4-d019938a2076","Type":"ContainerDied","Data":"210d7e067c0a6bfc55685bb1167b6ba7675a5e6d78918242041d6d1a2abd1f32"}
Jul 07 17:04:55.484032 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:55.484004    2583 generic.go:356] "Generic (PLEG): container finished" podID="bfcd374c-ea10-46d5-8242-9cb049f8867f" containerID="30d9c4134804b2a192b6001141aa542b2b2a24b8c3c3accadbb9eb39b0fe1259" exitCode=2
Jul 07 17:04:55.484032 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:55.484028    2583 generic.go:356] "Generic (PLEG): container finished" podID="bfcd374c-ea10-46d5-8242-9cb049f8867f" containerID="6c69f5e829d46ef560dd7f3526853c52abb1605e3b1c37e6dcbd284243d1b8f8" exitCode=2
Jul 07 17:04:55.484250 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:55.484056    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg"
Jul 07 17:04:55.484250 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:55.484051    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg" event={"ID":"bfcd374c-ea10-46d5-8242-9cb049f8867f","Type":"ContainerDied","Data":"30d9c4134804b2a192b6001141aa542b2b2a24b8c3c3accadbb9eb39b0fe1259"}
Jul 07 17:04:55.484250 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:55.484201    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg" event={"ID":"bfcd374c-ea10-46d5-8242-9cb049f8867f","Type":"ContainerDied","Data":"6c69f5e829d46ef560dd7f3526853c52abb1605e3b1c37e6dcbd284243d1b8f8"}
Jul 07 17:04:55.484250 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:55.484232    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg" event={"ID":"bfcd374c-ea10-46d5-8242-9cb049f8867f","Type":"ContainerDied","Data":"9f1fcb8c44bc31eee1b9244989e5fd25088b0de26f8ff7996c5ce9f5ea938070"}
Jul 07 17:04:55.484466 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:55.484264    2583 scope.go:117] "RemoveContainer" containerID="30d9c4134804b2a192b6001141aa542b2b2a24b8c3c3accadbb9eb39b0fe1259"
Jul 07 17:04:55.485850 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:55.485829    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" event={"ID":"b52f9c73-60cf-4897-a8c1-2ae824819a23","Type":"ContainerStarted","Data":"1c2d7d07bdb2b31ac8472c3e78a170fdea3559c8497ef2884752b346a3a002e1"}
Jul 07 17:04:55.485948 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:55.485884    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" event={"ID":"b52f9c73-60cf-4897-a8c1-2ae824819a23","Type":"ContainerStarted","Data":"ad1145e0466b52ba4897521a2121665bf4f502c4eff8d7173879b141c2433f5a"}
Jul 07 17:04:55.493570 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:55.493556    2583 scope.go:117] "RemoveContainer" containerID="6c69f5e829d46ef560dd7f3526853c52abb1605e3b1c37e6dcbd284243d1b8f8"
Jul 07 17:04:55.501487 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:55.501471    2583 scope.go:117] "RemoveContainer" containerID="30d9c4134804b2a192b6001141aa542b2b2a24b8c3c3accadbb9eb39b0fe1259"
Jul 07 17:04:55.501771 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:04:55.501756    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30d9c4134804b2a192b6001141aa542b2b2a24b8c3c3accadbb9eb39b0fe1259\": container with ID starting with 30d9c4134804b2a192b6001141aa542b2b2a24b8c3c3accadbb9eb39b0fe1259 not found: ID does not exist" containerID="30d9c4134804b2a192b6001141aa542b2b2a24b8c3c3accadbb9eb39b0fe1259"
Jul 07 17:04:55.501816 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:55.501779    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30d9c4134804b2a192b6001141aa542b2b2a24b8c3c3accadbb9eb39b0fe1259"} err="failed to get container status \"30d9c4134804b2a192b6001141aa542b2b2a24b8c3c3accadbb9eb39b0fe1259\": rpc error: code = NotFound desc = could not find container \"30d9c4134804b2a192b6001141aa542b2b2a24b8c3c3accadbb9eb39b0fe1259\": container with ID starting with 30d9c4134804b2a192b6001141aa542b2b2a24b8c3c3accadbb9eb39b0fe1259 not found: ID does not exist"
Jul 07 17:04:55.501816 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:55.501795    2583 scope.go:117] "RemoveContainer" containerID="6c69f5e829d46ef560dd7f3526853c52abb1605e3b1c37e6dcbd284243d1b8f8"
Jul 07 17:04:55.502041 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:04:55.502023    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c69f5e829d46ef560dd7f3526853c52abb1605e3b1c37e6dcbd284243d1b8f8\": container with ID starting with 6c69f5e829d46ef560dd7f3526853c52abb1605e3b1c37e6dcbd284243d1b8f8 not found: ID does not exist" containerID="6c69f5e829d46ef560dd7f3526853c52abb1605e3b1c37e6dcbd284243d1b8f8"
Jul 07 17:04:55.502077 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:55.502044    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c69f5e829d46ef560dd7f3526853c52abb1605e3b1c37e6dcbd284243d1b8f8"} err="failed to get container status \"6c69f5e829d46ef560dd7f3526853c52abb1605e3b1c37e6dcbd284243d1b8f8\": rpc error: code = NotFound desc = could not find container \"6c69f5e829d46ef560dd7f3526853c52abb1605e3b1c37e6dcbd284243d1b8f8\": container with ID starting with 6c69f5e829d46ef560dd7f3526853c52abb1605e3b1c37e6dcbd284243d1b8f8 not found: ID does not exist"
Jul 07 17:04:55.502077 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:55.502057    2583 scope.go:117] "RemoveContainer" containerID="30d9c4134804b2a192b6001141aa542b2b2a24b8c3c3accadbb9eb39b0fe1259"
Jul 07 17:04:55.502359 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:55.502339    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30d9c4134804b2a192b6001141aa542b2b2a24b8c3c3accadbb9eb39b0fe1259"} err="failed to get container status \"30d9c4134804b2a192b6001141aa542b2b2a24b8c3c3accadbb9eb39b0fe1259\": rpc error: code = NotFound desc = could not find container \"30d9c4134804b2a192b6001141aa542b2b2a24b8c3c3accadbb9eb39b0fe1259\": container with ID starting with 30d9c4134804b2a192b6001141aa542b2b2a24b8c3c3accadbb9eb39b0fe1259 not found: ID does not exist"
Jul 07 17:04:55.502413 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:55.502360    2583 scope.go:117] "RemoveContainer" containerID="6c69f5e829d46ef560dd7f3526853c52abb1605e3b1c37e6dcbd284243d1b8f8"
Jul 07 17:04:55.502580 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:55.502563    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c69f5e829d46ef560dd7f3526853c52abb1605e3b1c37e6dcbd284243d1b8f8"} err="failed to get container status \"6c69f5e829d46ef560dd7f3526853c52abb1605e3b1c37e6dcbd284243d1b8f8\": rpc error: code = NotFound desc = could not find container \"6c69f5e829d46ef560dd7f3526853c52abb1605e3b1c37e6dcbd284243d1b8f8\": container with ID starting with 6c69f5e829d46ef560dd7f3526853c52abb1605e3b1c37e6dcbd284243d1b8f8 not found: ID does not exist"
Jul 07 17:04:55.527919 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:55.527888    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg"]
Jul 07 17:04:55.534677 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:55.534645    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-41dba-predictor-5bd99b54b4-2b8lg"]
Jul 07 17:04:55.586895 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:55.586864    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfcd374c-ea10-46d5-8242-9cb049f8867f" path="/var/lib/kubelet/pods/bfcd374c-ea10-46d5-8242-9cb049f8867f/volumes"
Jul 07 17:04:56.095248 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:56.095209    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 07 17:04:59.506432 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:59.506396    2583 generic.go:356] "Generic (PLEG): container finished" podID="43ead570-570e-4b3d-97e4-d019938a2076" containerID="8217d86c3626c2bad6bc2aa9e6f91fcaff3a08c0961a3722ad4c43693bdda8e5" exitCode=0
Jul 07 17:04:59.506878 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:59.506475    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" event={"ID":"43ead570-570e-4b3d-97e4-d019938a2076","Type":"ContainerDied","Data":"8217d86c3626c2bad6bc2aa9e6f91fcaff3a08c0961a3722ad4c43693bdda8e5"}
Jul 07 17:04:59.507980 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:59.507960    2583 generic.go:356] "Generic (PLEG): container finished" podID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerID="1c2d7d07bdb2b31ac8472c3e78a170fdea3559c8497ef2884752b346a3a002e1" exitCode=0
Jul 07 17:04:59.508074 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:04:59.508025    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" event={"ID":"b52f9c73-60cf-4897-a8c1-2ae824819a23","Type":"ContainerDied","Data":"1c2d7d07bdb2b31ac8472c3e78a170fdea3559c8497ef2884752b346a3a002e1"}
Jul 07 17:05:00.514822 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:00.514786    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" event={"ID":"b52f9c73-60cf-4897-a8c1-2ae824819a23","Type":"ContainerStarted","Data":"7c71be809aa31fa3f7b48896b4a9a8323a89926d442b664ac77701882fe2536b"}
Jul 07 17:05:00.515243 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:00.514850    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" event={"ID":"b52f9c73-60cf-4897-a8c1-2ae824819a23","Type":"ContainerStarted","Data":"3823c3e7ff89fbd6fe5cf21c7433dda4b97613bdf59fd47fbe641e9e2c1e3e42"}
Jul 07 17:05:00.515243 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:00.515085    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"
Jul 07 17:05:00.548381 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:00.548332    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" podStartSLOduration=2.722169488 podStartE2EDuration="6.548313638s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.82614415s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:05:00.548040847 +0000 UTC m=+1093.561050668" watchObservedRunningTime="2026-07-07 17:05:00.548313638 +0000 UTC m=+1093.561323517"
Jul 07 17:05:01.095200 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:01.095160    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 07 17:05:01.100510 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:01.100477    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 17:05:01.100884 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:01.100858    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:05:01.519160 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:01.519108    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"
Jul 07 17:05:01.520396 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:01.520364    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 17:05:02.523211 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:02.523171    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 17:05:06.095334 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:06.095294    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 07 17:05:06.095741 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:06.095416    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:05:07.527418 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:07.527387    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"
Jul 07 17:05:07.527876 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:07.527850    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 17:05:11.095380 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:11.095336    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 07 17:05:11.099662 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:11.099627    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 17:05:11.100016 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:11.099993    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:05:16.094851 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:16.094798    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 07 17:05:17.528688 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:17.528647    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 17:05:21.095071 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:21.095027    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 07 17:05:21.100371 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:21.100346    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 17:05:21.100494 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:21.100475    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:05:21.100798 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:21.100774    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:05:21.100883 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:21.100871    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:05:24.532751 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.532725    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:05:24.618525 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.618492    2583 generic.go:356] "Generic (PLEG): container finished" podID="43ead570-570e-4b3d-97e4-d019938a2076" containerID="83d49a47efe867203f83a80c560220b539be918a557bb6f472f1c7c8fcda86d4" exitCode=0
Jul 07 17:05:24.618686 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.618568    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" event={"ID":"43ead570-570e-4b3d-97e4-d019938a2076","Type":"ContainerDied","Data":"83d49a47efe867203f83a80c560220b539be918a557bb6f472f1c7c8fcda86d4"}
Jul 07 17:05:24.618686 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.618609    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"
Jul 07 17:05:24.618686 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.618621    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm" event={"ID":"43ead570-570e-4b3d-97e4-d019938a2076","Type":"ContainerDied","Data":"a742d6116a04c52f84fe8f9cb78673b7a0c2b953a96831c79f74626bf98965fd"}
Jul 07 17:05:24.618686 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.618662    2583 scope.go:117] "RemoveContainer" containerID="83d49a47efe867203f83a80c560220b539be918a557bb6f472f1c7c8fcda86d4"
Jul 07 17:05:24.627537 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.627519    2583 scope.go:117] "RemoveContainer" containerID="210d7e067c0a6bfc55685bb1167b6ba7675a5e6d78918242041d6d1a2abd1f32"
Jul 07 17:05:24.637326 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.637298    2583 scope.go:117] "RemoveContainer" containerID="8217d86c3626c2bad6bc2aa9e6f91fcaff3a08c0961a3722ad4c43693bdda8e5"
Jul 07 17:05:24.638158 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.638115    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29hpp\" (UniqueName: \"kubernetes.io/projected/43ead570-570e-4b3d-97e4-d019938a2076-kube-api-access-29hpp\") pod \"43ead570-570e-4b3d-97e4-d019938a2076\" (UID: \"43ead570-570e-4b3d-97e4-d019938a2076\") "
Jul 07 17:05:24.638268 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.638170    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43ead570-570e-4b3d-97e4-d019938a2076-proxy-tls\") pod \"43ead570-570e-4b3d-97e4-d019938a2076\" (UID: \"43ead570-570e-4b3d-97e4-d019938a2076\") "
Jul 07 17:05:24.638268 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.638247    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43ead570-570e-4b3d-97e4-d019938a2076-cabundle-cert\") pod \"43ead570-570e-4b3d-97e4-d019938a2076\" (UID: \"43ead570-570e-4b3d-97e4-d019938a2076\") "
Jul 07 17:05:24.638374 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.638285    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43ead570-570e-4b3d-97e4-d019938a2076-kube-rbac-proxy-sar-config\") pod \"43ead570-570e-4b3d-97e4-d019938a2076\" (UID: \"43ead570-570e-4b3d-97e4-d019938a2076\") "
Jul 07 17:05:24.638476 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.638401    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43ead570-570e-4b3d-97e4-d019938a2076-kserve-provision-location\") pod \"43ead570-570e-4b3d-97e4-d019938a2076\" (UID: \"43ead570-570e-4b3d-97e4-d019938a2076\") "
Jul 07 17:05:24.638816 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.638624    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ead570-570e-4b3d-97e4-d019938a2076-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "43ead570-570e-4b3d-97e4-d019938a2076" (UID: "43ead570-570e-4b3d-97e4-d019938a2076"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:05:24.638816 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.638740    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ead570-570e-4b3d-97e4-d019938a2076-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "43ead570-570e-4b3d-97e4-d019938a2076" (UID: "43ead570-570e-4b3d-97e4-d019938a2076"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:05:24.639016 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.638952    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ead570-570e-4b3d-97e4-d019938a2076-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "43ead570-570e-4b3d-97e4-d019938a2076" (UID: "43ead570-570e-4b3d-97e4-d019938a2076"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:05:24.640427 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.640397    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ead570-570e-4b3d-97e4-d019938a2076-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "43ead570-570e-4b3d-97e4-d019938a2076" (UID: "43ead570-570e-4b3d-97e4-d019938a2076"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:05:24.640849 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.640828    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ead570-570e-4b3d-97e4-d019938a2076-kube-api-access-29hpp" (OuterVolumeSpecName: "kube-api-access-29hpp") pod "43ead570-570e-4b3d-97e4-d019938a2076" (UID: "43ead570-570e-4b3d-97e4-d019938a2076"). InnerVolumeSpecName "kube-api-access-29hpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:05:24.655095 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.655073    2583 scope.go:117] "RemoveContainer" containerID="9f325ca31d44e4ef81a6ce551ff94b9b2578605be62f5134f2494c8200e62b6d"
Jul 07 17:05:24.662891 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.662871    2583 scope.go:117] "RemoveContainer" containerID="83d49a47efe867203f83a80c560220b539be918a557bb6f472f1c7c8fcda86d4"
Jul 07 17:05:24.663184 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:05:24.663164    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83d49a47efe867203f83a80c560220b539be918a557bb6f472f1c7c8fcda86d4\": container with ID starting with 83d49a47efe867203f83a80c560220b539be918a557bb6f472f1c7c8fcda86d4 not found: ID does not exist" containerID="83d49a47efe867203f83a80c560220b539be918a557bb6f472f1c7c8fcda86d4"
Jul 07 17:05:24.663251 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.663195    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83d49a47efe867203f83a80c560220b539be918a557bb6f472f1c7c8fcda86d4"} err="failed to get container status \"83d49a47efe867203f83a80c560220b539be918a557bb6f472f1c7c8fcda86d4\": rpc error: code = NotFound desc = could not find container \"83d49a47efe867203f83a80c560220b539be918a557bb6f472f1c7c8fcda86d4\": container with ID starting with 83d49a47efe867203f83a80c560220b539be918a557bb6f472f1c7c8fcda86d4 not found: ID does not exist"
Jul 07 17:05:24.663251 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.663213    2583 scope.go:117] "RemoveContainer" containerID="210d7e067c0a6bfc55685bb1167b6ba7675a5e6d78918242041d6d1a2abd1f32"
Jul 07 17:05:24.663425 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:05:24.663407    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"210d7e067c0a6bfc55685bb1167b6ba7675a5e6d78918242041d6d1a2abd1f32\": container with ID starting with 210d7e067c0a6bfc55685bb1167b6ba7675a5e6d78918242041d6d1a2abd1f32 not found: ID does not exist" containerID="210d7e067c0a6bfc55685bb1167b6ba7675a5e6d78918242041d6d1a2abd1f32"
Jul 07 17:05:24.663462 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.663432    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"210d7e067c0a6bfc55685bb1167b6ba7675a5e6d78918242041d6d1a2abd1f32"} err="failed to get container status \"210d7e067c0a6bfc55685bb1167b6ba7675a5e6d78918242041d6d1a2abd1f32\": rpc error: code = NotFound desc = could not find container \"210d7e067c0a6bfc55685bb1167b6ba7675a5e6d78918242041d6d1a2abd1f32\": container with ID starting with 210d7e067c0a6bfc55685bb1167b6ba7675a5e6d78918242041d6d1a2abd1f32 not found: ID does not exist"
Jul 07 17:05:24.663462 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.663448    2583 scope.go:117] "RemoveContainer" containerID="8217d86c3626c2bad6bc2aa9e6f91fcaff3a08c0961a3722ad4c43693bdda8e5"
Jul 07 17:05:24.663685 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:05:24.663661    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8217d86c3626c2bad6bc2aa9e6f91fcaff3a08c0961a3722ad4c43693bdda8e5\": container with ID starting with 8217d86c3626c2bad6bc2aa9e6f91fcaff3a08c0961a3722ad4c43693bdda8e5 not found: ID does not exist" containerID="8217d86c3626c2bad6bc2aa9e6f91fcaff3a08c0961a3722ad4c43693bdda8e5"
Jul 07 17:05:24.663933 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.663916    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8217d86c3626c2bad6bc2aa9e6f91fcaff3a08c0961a3722ad4c43693bdda8e5"} err="failed to get container status \"8217d86c3626c2bad6bc2aa9e6f91fcaff3a08c0961a3722ad4c43693bdda8e5\": rpc error: code = NotFound desc = could not find container \"8217d86c3626c2bad6bc2aa9e6f91fcaff3a08c0961a3722ad4c43693bdda8e5\": container with ID starting with 8217d86c3626c2bad6bc2aa9e6f91fcaff3a08c0961a3722ad4c43693bdda8e5 not found: ID does not exist"
Jul 07 17:05:24.663969 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.663934    2583 scope.go:117] "RemoveContainer" containerID="9f325ca31d44e4ef81a6ce551ff94b9b2578605be62f5134f2494c8200e62b6d"
Jul 07 17:05:24.664193 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:05:24.664167    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f325ca31d44e4ef81a6ce551ff94b9b2578605be62f5134f2494c8200e62b6d\": container with ID starting with 9f325ca31d44e4ef81a6ce551ff94b9b2578605be62f5134f2494c8200e62b6d not found: ID does not exist" containerID="9f325ca31d44e4ef81a6ce551ff94b9b2578605be62f5134f2494c8200e62b6d"
Jul 07 17:05:24.664287 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.664197    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f325ca31d44e4ef81a6ce551ff94b9b2578605be62f5134f2494c8200e62b6d"} err="failed to get container status \"9f325ca31d44e4ef81a6ce551ff94b9b2578605be62f5134f2494c8200e62b6d\": rpc error: code = NotFound desc = could not find container \"9f325ca31d44e4ef81a6ce551ff94b9b2578605be62f5134f2494c8200e62b6d\": container with ID starting with 9f325ca31d44e4ef81a6ce551ff94b9b2578605be62f5134f2494c8200e62b6d not found: ID does not exist"
Jul 07 17:05:24.739448 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.739416    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43ead570-570e-4b3d-97e4-d019938a2076-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:05:24.739448 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.739442    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43ead570-570e-4b3d-97e4-d019938a2076-kserve-provision-location\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:05:24.739448 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.739452    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29hpp\" (UniqueName: \"kubernetes.io/projected/43ead570-570e-4b3d-97e4-d019938a2076-kube-api-access-29hpp\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:05:24.739682 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.739461    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43ead570-570e-4b3d-97e4-d019938a2076-proxy-tls\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:05:24.739682 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.739469    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43ead570-570e-4b3d-97e4-d019938a2076-cabundle-cert\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:05:24.945367 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.945334    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"]
Jul 07 17:05:24.950829 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:24.950802    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-41dba-predictor-fd5b4b66f-8nvpm"]
Jul 07 17:05:25.587007 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:25.586973    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43ead570-570e-4b3d-97e4-d019938a2076" path="/var/lib/kubelet/pods/43ead570-570e-4b3d-97e4-d019938a2076/volumes"
Jul 07 17:05:27.527970 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:27.527933    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 17:05:37.528491 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:37.528451    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 17:05:47.527900 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:47.527857    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 17:05:57.528086 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:05:57.528041    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 17:06:07.528419 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:06:07.528332    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 17:06:17.528766 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:06:17.528714    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 17:06:24.582329 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:06:24.582282    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 17:06:34.582898 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:06:34.582850    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 17:06:44.582369 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:06:44.582327    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 17:06:54.582275 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:06:54.582228    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 17:07:04.582450 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:04.582401    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 17:07:14.583256 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:14.583212    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 17:07:24.582625 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:24.582581    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 17:07:27.585041 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:27.585001    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 17:07:37.588788 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:37.588707    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"
Jul 07 17:07:44.547701 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:44.547659    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"]
Jul 07 17:07:44.548195 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:44.547958    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kserve-container" containerID="cri-o://3823c3e7ff89fbd6fe5cf21c7433dda4b97613bdf59fd47fbe641e9e2c1e3e42" gracePeriod=30
Jul 07 17:07:44.548195 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:44.547998    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kube-rbac-proxy" containerID="cri-o://7c71be809aa31fa3f7b48896b4a9a8323a89926d442b664ac77701882fe2536b" gracePeriod=30
Jul 07 17:07:45.210828 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:45.210793    2583 generic.go:356] "Generic (PLEG): container finished" podID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerID="7c71be809aa31fa3f7b48896b4a9a8323a89926d442b664ac77701882fe2536b" exitCode=2
Jul 07 17:07:45.211004 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:45.210839    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" event={"ID":"b52f9c73-60cf-4897-a8c1-2ae824819a23","Type":"ContainerDied","Data":"7c71be809aa31fa3f7b48896b4a9a8323a89926d442b664ac77701882fe2536b"}
Jul 07 17:07:47.523585 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:47.523536    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 07 17:07:47.585051 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:47.585009    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 07 17:07:49.771899 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.771866    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"]
Jul 07 17:07:49.772434 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.772414    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfcd374c-ea10-46d5-8242-9cb049f8867f" containerName="kube-rbac-proxy"
Jul 07 17:07:49.772518 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.772437    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfcd374c-ea10-46d5-8242-9cb049f8867f" containerName="kube-rbac-proxy"
Jul 07 17:07:49.772518 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.772455    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfcd374c-ea10-46d5-8242-9cb049f8867f" containerName="kserve-container"
Jul 07 17:07:49.772518 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.772463    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfcd374c-ea10-46d5-8242-9cb049f8867f" containerName="kserve-container"
Jul 07 17:07:49.772518 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.772471    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="agent"
Jul 07 17:07:49.772518 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.772480    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="agent"
Jul 07 17:07:49.772518 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.772494    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="storage-initializer"
Jul 07 17:07:49.772518 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.772503    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="storage-initializer"
Jul 07 17:07:49.772866 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.772540    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="kserve-container"
Jul 07 17:07:49.772866 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.772548    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="kserve-container"
Jul 07 17:07:49.772866 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.772559    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="kube-rbac-proxy"
Jul 07 17:07:49.772866 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.772567    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="kube-rbac-proxy"
Jul 07 17:07:49.772866 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.772646    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="kserve-container"
Jul 07 17:07:49.772866 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.772660    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="agent"
Jul 07 17:07:49.772866 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.772670    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfcd374c-ea10-46d5-8242-9cb049f8867f" containerName="kserve-container"
Jul 07 17:07:49.772866 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.772678    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="43ead570-570e-4b3d-97e4-d019938a2076" containerName="kube-rbac-proxy"
Jul 07 17:07:49.772866 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.772689    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfcd374c-ea10-46d5-8242-9cb049f8867f" containerName="kube-rbac-proxy"
Jul 07 17:07:49.776043 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.776020    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"
Jul 07 17:07:49.779200 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.779172    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-0c9a81-predictor-serving-cert\""
Jul 07 17:07:49.779331 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.779260    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-0c9a81-kube-rbac-proxy-sar-config\""
Jul 07 17:07:49.792050 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.792024    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"]
Jul 07 17:07:49.820000 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.819968    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09cccda9-2bc2-4e97-99a9-ef8097cb931e-cabundle-cert\") pod \"isvc-primary-0c9a81-predictor-5566d97cdf-5prxc\" (UID: \"09cccda9-2bc2-4e97-99a9-ef8097cb931e\") " pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"
Jul 07 17:07:49.820000 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.820005    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09cccda9-2bc2-4e97-99a9-ef8097cb931e-proxy-tls\") pod \"isvc-primary-0c9a81-predictor-5566d97cdf-5prxc\" (UID: \"09cccda9-2bc2-4e97-99a9-ef8097cb931e\") " pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"
Jul 07 17:07:49.820270 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.820044    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wnhj\" (UniqueName: \"kubernetes.io/projected/09cccda9-2bc2-4e97-99a9-ef8097cb931e-kube-api-access-7wnhj\") pod \"isvc-primary-0c9a81-predictor-5566d97cdf-5prxc\" (UID: \"09cccda9-2bc2-4e97-99a9-ef8097cb931e\") " pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"
Jul 07 17:07:49.820270 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.820126    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09cccda9-2bc2-4e97-99a9-ef8097cb931e-kube-rbac-proxy-sar-config\") pod \"isvc-primary-0c9a81-predictor-5566d97cdf-5prxc\" (UID: \"09cccda9-2bc2-4e97-99a9-ef8097cb931e\") " pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"
Jul 07 17:07:49.820270 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.820203    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09cccda9-2bc2-4e97-99a9-ef8097cb931e-kserve-provision-location\") pod \"isvc-primary-0c9a81-predictor-5566d97cdf-5prxc\" (UID: \"09cccda9-2bc2-4e97-99a9-ef8097cb931e\") " pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"
Jul 07 17:07:49.921590 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.921554    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wnhj\" (UniqueName: \"kubernetes.io/projected/09cccda9-2bc2-4e97-99a9-ef8097cb931e-kube-api-access-7wnhj\") pod \"isvc-primary-0c9a81-predictor-5566d97cdf-5prxc\" (UID: \"09cccda9-2bc2-4e97-99a9-ef8097cb931e\") " pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"
Jul 07 17:07:49.921590 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.921591    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09cccda9-2bc2-4e97-99a9-ef8097cb931e-kube-rbac-proxy-sar-config\") pod \"isvc-primary-0c9a81-predictor-5566d97cdf-5prxc\" (UID: \"09cccda9-2bc2-4e97-99a9-ef8097cb931e\") " pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"
Jul 07 17:07:49.921845 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.921618    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09cccda9-2bc2-4e97-99a9-ef8097cb931e-kserve-provision-location\") pod \"isvc-primary-0c9a81-predictor-5566d97cdf-5prxc\" (UID: \"09cccda9-2bc2-4e97-99a9-ef8097cb931e\") " pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"
Jul 07 17:07:49.921845 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.921689    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09cccda9-2bc2-4e97-99a9-ef8097cb931e-cabundle-cert\") pod \"isvc-primary-0c9a81-predictor-5566d97cdf-5prxc\" (UID: \"09cccda9-2bc2-4e97-99a9-ef8097cb931e\") " pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"
Jul 07 17:07:49.921845 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.921709    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09cccda9-2bc2-4e97-99a9-ef8097cb931e-proxy-tls\") pod \"isvc-primary-0c9a81-predictor-5566d97cdf-5prxc\" (UID: \"09cccda9-2bc2-4e97-99a9-ef8097cb931e\") " pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"
Jul 07 17:07:49.921845 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:07:49.921817    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-serving-cert: secret "isvc-primary-0c9a81-predictor-serving-cert" not found
Jul 07 17:07:49.922084 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:07:49.921875    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09cccda9-2bc2-4e97-99a9-ef8097cb931e-proxy-tls podName:09cccda9-2bc2-4e97-99a9-ef8097cb931e nodeName:}" failed. No retries permitted until 2026-07-07 17:07:50.421859943 +0000 UTC m=+1263.434869756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/09cccda9-2bc2-4e97-99a9-ef8097cb931e-proxy-tls") pod "isvc-primary-0c9a81-predictor-5566d97cdf-5prxc" (UID: "09cccda9-2bc2-4e97-99a9-ef8097cb931e") : secret "isvc-primary-0c9a81-predictor-serving-cert" not found
Jul 07 17:07:49.922131 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.922091    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09cccda9-2bc2-4e97-99a9-ef8097cb931e-kserve-provision-location\") pod \"isvc-primary-0c9a81-predictor-5566d97cdf-5prxc\" (UID: \"09cccda9-2bc2-4e97-99a9-ef8097cb931e\") " pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"
Jul 07 17:07:49.922454 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.922419    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09cccda9-2bc2-4e97-99a9-ef8097cb931e-cabundle-cert\") pod \"isvc-primary-0c9a81-predictor-5566d97cdf-5prxc\" (UID: \"09cccda9-2bc2-4e97-99a9-ef8097cb931e\") " pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"
Jul 07 17:07:49.922609 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.922594    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09cccda9-2bc2-4e97-99a9-ef8097cb931e-kube-rbac-proxy-sar-config\") pod \"isvc-primary-0c9a81-predictor-5566d97cdf-5prxc\" (UID: \"09cccda9-2bc2-4e97-99a9-ef8097cb931e\") " pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"
Jul 07 17:07:49.936490 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:49.936462    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wnhj\" (UniqueName: \"kubernetes.io/projected/09cccda9-2bc2-4e97-99a9-ef8097cb931e-kube-api-access-7wnhj\") pod \"isvc-primary-0c9a81-predictor-5566d97cdf-5prxc\" (UID: \"09cccda9-2bc2-4e97-99a9-ef8097cb931e\") " pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"
Jul 07 17:07:50.425091 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:50.425052    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09cccda9-2bc2-4e97-99a9-ef8097cb931e-proxy-tls\") pod \"isvc-primary-0c9a81-predictor-5566d97cdf-5prxc\" (UID: \"09cccda9-2bc2-4e97-99a9-ef8097cb931e\") " pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"
Jul 07 17:07:50.427696 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:50.427651    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09cccda9-2bc2-4e97-99a9-ef8097cb931e-proxy-tls\") pod \"isvc-primary-0c9a81-predictor-5566d97cdf-5prxc\" (UID: \"09cccda9-2bc2-4e97-99a9-ef8097cb931e\") " pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"
Jul 07 17:07:50.686111 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:50.686013    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"
Jul 07 17:07:50.842075 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:50.842050    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"]
Jul 07 17:07:50.844734 ip-10-0-141-189 kubenswrapper[2583]: W0707 17:07:50.844708    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09cccda9_2bc2_4e97_99a9_ef8097cb931e.slice/crio-69d0317c89dd9ec0c89781ee1e8e6c1201fbc3331e09655ecbcbe6d920a9270c WatchSource:0}: Error finding container 69d0317c89dd9ec0c89781ee1e8e6c1201fbc3331e09655ecbcbe6d920a9270c: Status 404 returned error can't find the container with id 69d0317c89dd9ec0c89781ee1e8e6c1201fbc3331e09655ecbcbe6d920a9270c
Jul 07 17:07:51.243841 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:51.243804    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc" event={"ID":"09cccda9-2bc2-4e97-99a9-ef8097cb931e","Type":"ContainerStarted","Data":"c52f58b6599f1a9f905298ffaf860f5a953c5b47fbf6d0ce3e65fe0a352de945"}
Jul 07 17:07:51.244023 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:51.243864    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc" event={"ID":"09cccda9-2bc2-4e97-99a9-ef8097cb931e","Type":"ContainerStarted","Data":"69d0317c89dd9ec0c89781ee1e8e6c1201fbc3331e09655ecbcbe6d920a9270c"}
Jul 07 17:07:52.523534 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:52.523488    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 07 17:07:54.532534 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:54.532511    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"
Jul 07 17:07:54.662635 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:54.662552    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b52f9c73-60cf-4897-a8c1-2ae824819a23-kserve-provision-location\") pod \"b52f9c73-60cf-4897-a8c1-2ae824819a23\" (UID: \"b52f9c73-60cf-4897-a8c1-2ae824819a23\") "
Jul 07 17:07:54.662635 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:54.662598    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b52f9c73-60cf-4897-a8c1-2ae824819a23-proxy-tls\") pod \"b52f9c73-60cf-4897-a8c1-2ae824819a23\" (UID: \"b52f9c73-60cf-4897-a8c1-2ae824819a23\") "
Jul 07 17:07:54.662635 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:54.662623    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b52f9c73-60cf-4897-a8c1-2ae824819a23-cabundle-cert\") pod \"b52f9c73-60cf-4897-a8c1-2ae824819a23\" (UID: \"b52f9c73-60cf-4897-a8c1-2ae824819a23\") "
Jul 07 17:07:54.662960 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:54.662648    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b52f9c73-60cf-4897-a8c1-2ae824819a23-kube-rbac-proxy-sar-config\") pod \"b52f9c73-60cf-4897-a8c1-2ae824819a23\" (UID: \"b52f9c73-60cf-4897-a8c1-2ae824819a23\") "
Jul 07 17:07:54.662960 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:54.662729    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pkhb\" (UniqueName: \"kubernetes.io/projected/b52f9c73-60cf-4897-a8c1-2ae824819a23-kube-api-access-6pkhb\") pod \"b52f9c73-60cf-4897-a8c1-2ae824819a23\" (UID: \"b52f9c73-60cf-4897-a8c1-2ae824819a23\") "
Jul 07 17:07:54.662960 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:54.662843    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b52f9c73-60cf-4897-a8c1-2ae824819a23-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b52f9c73-60cf-4897-a8c1-2ae824819a23" (UID: "b52f9c73-60cf-4897-a8c1-2ae824819a23"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:07:54.662960 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:54.662925    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b52f9c73-60cf-4897-a8c1-2ae824819a23-kserve-provision-location\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:07:54.663169 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:54.662999    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b52f9c73-60cf-4897-a8c1-2ae824819a23-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b52f9c73-60cf-4897-a8c1-2ae824819a23" (UID: "b52f9c73-60cf-4897-a8c1-2ae824819a23"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:07:54.663169 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:54.663069    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b52f9c73-60cf-4897-a8c1-2ae824819a23-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b52f9c73-60cf-4897-a8c1-2ae824819a23" (UID: "b52f9c73-60cf-4897-a8c1-2ae824819a23"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:07:54.664895 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:54.664874    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b52f9c73-60cf-4897-a8c1-2ae824819a23-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b52f9c73-60cf-4897-a8c1-2ae824819a23" (UID: "b52f9c73-60cf-4897-a8c1-2ae824819a23"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:07:54.664973 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:54.664903    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b52f9c73-60cf-4897-a8c1-2ae824819a23-kube-api-access-6pkhb" (OuterVolumeSpecName: "kube-api-access-6pkhb") pod "b52f9c73-60cf-4897-a8c1-2ae824819a23" (UID: "b52f9c73-60cf-4897-a8c1-2ae824819a23"). InnerVolumeSpecName "kube-api-access-6pkhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:07:54.763484 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:54.763449    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b52f9c73-60cf-4897-a8c1-2ae824819a23-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:07:54.763484 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:54.763478    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pkhb\" (UniqueName: \"kubernetes.io/projected/b52f9c73-60cf-4897-a8c1-2ae824819a23-kube-api-access-6pkhb\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:07:54.763484 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:54.763487    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b52f9c73-60cf-4897-a8c1-2ae824819a23-proxy-tls\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:07:54.763715 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:54.763496    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b52f9c73-60cf-4897-a8c1-2ae824819a23-cabundle-cert\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:07:55.261957 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:55.261922    2583 generic.go:356] "Generic (PLEG): container finished" podID="09cccda9-2bc2-4e97-99a9-ef8097cb931e" containerID="c52f58b6599f1a9f905298ffaf860f5a953c5b47fbf6d0ce3e65fe0a352de945" exitCode=0
Jul 07 17:07:55.262164 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:55.261997    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc" event={"ID":"09cccda9-2bc2-4e97-99a9-ef8097cb931e","Type":"ContainerDied","Data":"c52f58b6599f1a9f905298ffaf860f5a953c5b47fbf6d0ce3e65fe0a352de945"}
Jul 07 17:07:55.264064 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:55.264038    2583 generic.go:356] "Generic (PLEG): container finished" podID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerID="3823c3e7ff89fbd6fe5cf21c7433dda4b97613bdf59fd47fbe641e9e2c1e3e42" exitCode=0
Jul 07 17:07:55.264191 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:55.264092    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"
Jul 07 17:07:55.264191 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:55.264090    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" event={"ID":"b52f9c73-60cf-4897-a8c1-2ae824819a23","Type":"ContainerDied","Data":"3823c3e7ff89fbd6fe5cf21c7433dda4b97613bdf59fd47fbe641e9e2c1e3e42"}
Jul 07 17:07:55.264191 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:55.264167    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc" event={"ID":"b52f9c73-60cf-4897-a8c1-2ae824819a23","Type":"ContainerDied","Data":"ad1145e0466b52ba4897521a2121665bf4f502c4eff8d7173879b141c2433f5a"}
Jul 07 17:07:55.264309 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:55.264207    2583 scope.go:117] "RemoveContainer" containerID="7c71be809aa31fa3f7b48896b4a9a8323a89926d442b664ac77701882fe2536b"
Jul 07 17:07:55.278958 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:55.278937    2583 scope.go:117] "RemoveContainer" containerID="3823c3e7ff89fbd6fe5cf21c7433dda4b97613bdf59fd47fbe641e9e2c1e3e42"
Jul 07 17:07:55.287234 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:55.287216    2583 scope.go:117] "RemoveContainer" containerID="1c2d7d07bdb2b31ac8472c3e78a170fdea3559c8497ef2884752b346a3a002e1"
Jul 07 17:07:55.298899 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:55.298877    2583 scope.go:117] "RemoveContainer" containerID="7c71be809aa31fa3f7b48896b4a9a8323a89926d442b664ac77701882fe2536b"
Jul 07 17:07:55.299207 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:07:55.299185    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c71be809aa31fa3f7b48896b4a9a8323a89926d442b664ac77701882fe2536b\": container with ID starting with 7c71be809aa31fa3f7b48896b4a9a8323a89926d442b664ac77701882fe2536b not found: ID does not exist" containerID="7c71be809aa31fa3f7b48896b4a9a8323a89926d442b664ac77701882fe2536b"
Jul 07 17:07:55.299301 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:55.299215    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c71be809aa31fa3f7b48896b4a9a8323a89926d442b664ac77701882fe2536b"} err="failed to get container status \"7c71be809aa31fa3f7b48896b4a9a8323a89926d442b664ac77701882fe2536b\": rpc error: code = NotFound desc = could not find container \"7c71be809aa31fa3f7b48896b4a9a8323a89926d442b664ac77701882fe2536b\": container with ID starting with 7c71be809aa31fa3f7b48896b4a9a8323a89926d442b664ac77701882fe2536b not found: ID does not exist"
Jul 07 17:07:55.299301 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:55.299234    2583 scope.go:117] "RemoveContainer" containerID="3823c3e7ff89fbd6fe5cf21c7433dda4b97613bdf59fd47fbe641e9e2c1e3e42"
Jul 07 17:07:55.299523 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:07:55.299501    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3823c3e7ff89fbd6fe5cf21c7433dda4b97613bdf59fd47fbe641e9e2c1e3e42\": container with ID starting with 3823c3e7ff89fbd6fe5cf21c7433dda4b97613bdf59fd47fbe641e9e2c1e3e42 not found: ID does not exist" containerID="3823c3e7ff89fbd6fe5cf21c7433dda4b97613bdf59fd47fbe641e9e2c1e3e42"
Jul 07 17:07:55.299623 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:55.299530    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3823c3e7ff89fbd6fe5cf21c7433dda4b97613bdf59fd47fbe641e9e2c1e3e42"} err="failed to get container status \"3823c3e7ff89fbd6fe5cf21c7433dda4b97613bdf59fd47fbe641e9e2c1e3e42\": rpc error: code = NotFound desc = could not find container \"3823c3e7ff89fbd6fe5cf21c7433dda4b97613bdf59fd47fbe641e9e2c1e3e42\": container with ID starting with 3823c3e7ff89fbd6fe5cf21c7433dda4b97613bdf59fd47fbe641e9e2c1e3e42 not found: ID does not exist"
Jul 07 17:07:55.299623 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:55.299546    2583 scope.go:117] "RemoveContainer" containerID="1c2d7d07bdb2b31ac8472c3e78a170fdea3559c8497ef2884752b346a3a002e1"
Jul 07 17:07:55.299829 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:07:55.299809    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c2d7d07bdb2b31ac8472c3e78a170fdea3559c8497ef2884752b346a3a002e1\": container with ID starting with 1c2d7d07bdb2b31ac8472c3e78a170fdea3559c8497ef2884752b346a3a002e1 not found: ID does not exist" containerID="1c2d7d07bdb2b31ac8472c3e78a170fdea3559c8497ef2884752b346a3a002e1"
Jul 07 17:07:55.299888 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:55.299837    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c2d7d07bdb2b31ac8472c3e78a170fdea3559c8497ef2884752b346a3a002e1"} err="failed to get container status \"1c2d7d07bdb2b31ac8472c3e78a170fdea3559c8497ef2884752b346a3a002e1\": rpc error: code = NotFound desc = could not find container \"1c2d7d07bdb2b31ac8472c3e78a170fdea3559c8497ef2884752b346a3a002e1\": container with ID starting with 1c2d7d07bdb2b31ac8472c3e78a170fdea3559c8497ef2884752b346a3a002e1 not found: ID does not exist"
Jul 07 17:07:55.310263 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:55.310242    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"]
Jul 07 17:07:55.316590 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:55.316560    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dd75c-predictor-676579bd58-wfblc"]
Jul 07 17:07:55.586299 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:55.586225    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" path="/var/lib/kubelet/pods/b52f9c73-60cf-4897-a8c1-2ae824819a23/volumes"
Jul 07 17:07:56.271318 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:56.271286    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc" event={"ID":"09cccda9-2bc2-4e97-99a9-ef8097cb931e","Type":"ContainerStarted","Data":"d850db5bff1ada7b3367a0fe0e5047460721343cfef33c189e4507c2641ec7a8"}
Jul 07 17:07:56.271495 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:56.271347    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc" event={"ID":"09cccda9-2bc2-4e97-99a9-ef8097cb931e","Type":"ContainerStarted","Data":"1d0410f13c09859e79c4a927551e3c50dbff2a3aa4068aab4123d5ee7f8be0b0"}
Jul 07 17:07:56.271607 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:56.271588    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"
Jul 07 17:07:56.300539 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:56.300484    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc" podStartSLOduration=3.748665324 podStartE2EDuration="7.300468536s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.551803212s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:07:56.298718321 +0000 UTC m=+1269.311728141" watchObservedRunningTime="2026-07-07 17:07:56.300468536 +0000 UTC m=+1269.313478354"
Jul 07 17:07:57.275718 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:57.275693    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"
Jul 07 17:07:57.277044 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:57.277014    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc" podUID="09cccda9-2bc2-4e97-99a9-ef8097cb931e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 17:07:58.279595 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:07:58.279554    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc" podUID="09cccda9-2bc2-4e97-99a9-ef8097cb931e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 17:08:03.284451 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:08:03.284422    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"
Jul 07 17:08:03.285090 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:08:03.285060    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc" podUID="09cccda9-2bc2-4e97-99a9-ef8097cb931e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 17:08:13.285277 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:08:13.285231    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc" podUID="09cccda9-2bc2-4e97-99a9-ef8097cb931e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 17:08:23.285390 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:08:23.285347    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc" podUID="09cccda9-2bc2-4e97-99a9-ef8097cb931e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 17:08:33.285321 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:08:33.285276    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc" podUID="09cccda9-2bc2-4e97-99a9-ef8097cb931e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 17:08:43.285545 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:08:43.285504    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc" podUID="09cccda9-2bc2-4e97-99a9-ef8097cb931e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 17:08:53.285871 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:08:53.285832    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc" podUID="09cccda9-2bc2-4e97-99a9-ef8097cb931e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 17:09:03.285518 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:03.285470    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc" podUID="09cccda9-2bc2-4e97-99a9-ef8097cb931e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 17:09:13.286411 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:13.286327    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"
Jul 07 17:09:19.948061 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:19.948023    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn"]
Jul 07 17:09:19.948541 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:19.948408    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kserve-container"
Jul 07 17:09:19.948541 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:19.948420    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kserve-container"
Jul 07 17:09:19.948541 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:19.948447    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="storage-initializer"
Jul 07 17:09:19.948541 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:19.948453    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="storage-initializer"
Jul 07 17:09:19.948541 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:19.948463    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kube-rbac-proxy"
Jul 07 17:09:19.948541 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:19.948471    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kube-rbac-proxy"
Jul 07 17:09:19.948541 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:19.948524    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kserve-container"
Jul 07 17:09:19.948541 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:19.948535    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b52f9c73-60cf-4897-a8c1-2ae824819a23" containerName="kube-rbac-proxy"
Jul 07 17:09:19.951841 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:19.951824    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn"
Jul 07 17:09:19.955795 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:19.955772    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-0c9a81-predictor-serving-cert\""
Jul 07 17:09:19.955917 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:19.955811    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-0c9a81-dockercfg-8smql\""
Jul 07 17:09:19.956365 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:19.956344    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-0c9a81-kube-rbac-proxy-sar-config\""
Jul 07 17:09:19.956484 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:19.956385    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-0c9a81\""
Jul 07 17:09:19.967994 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:19.967970    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn"]
Jul 07 17:09:20.016870 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:20.016837    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4a027b2-defb-498c-adb5-61d2f95e6514-cabundle-cert\") pod \"isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn\" (UID: \"e4a027b2-defb-498c-adb5-61d2f95e6514\") " pod="kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn"
Jul 07 17:09:20.017036 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:20.016892    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4a027b2-defb-498c-adb5-61d2f95e6514-kserve-provision-location\") pod \"isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn\" (UID: \"e4a027b2-defb-498c-adb5-61d2f95e6514\") " pod="kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn"
Jul 07 17:09:20.017036 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:20.016923    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvdcl\" (UniqueName: \"kubernetes.io/projected/e4a027b2-defb-498c-adb5-61d2f95e6514-kube-api-access-lvdcl\") pod \"isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn\" (UID: \"e4a027b2-defb-498c-adb5-61d2f95e6514\") " pod="kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn"
Jul 07 17:09:20.017036 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:20.016956    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4a027b2-defb-498c-adb5-61d2f95e6514-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn\" (UID: \"e4a027b2-defb-498c-adb5-61d2f95e6514\") " pod="kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn"
Jul 07 17:09:20.017183 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:20.017038    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4a027b2-defb-498c-adb5-61d2f95e6514-proxy-tls\") pod \"isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn\" (UID: \"e4a027b2-defb-498c-adb5-61d2f95e6514\") " pod="kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn"
Jul 07 17:09:20.117471 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:20.117434    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4a027b2-defb-498c-adb5-61d2f95e6514-cabundle-cert\") pod \"isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn\" (UID: \"e4a027b2-defb-498c-adb5-61d2f95e6514\") " pod="kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn"
Jul 07 17:09:20.117673 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:20.117483    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4a027b2-defb-498c-adb5-61d2f95e6514-kserve-provision-location\") pod \"isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn\" (UID: \"e4a027b2-defb-498c-adb5-61d2f95e6514\") " pod="kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn"
Jul 07 17:09:20.117673 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:20.117505    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvdcl\" (UniqueName: \"kubernetes.io/projected/e4a027b2-defb-498c-adb5-61d2f95e6514-kube-api-access-lvdcl\") pod \"isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn\" (UID: \"e4a027b2-defb-498c-adb5-61d2f95e6514\") " pod="kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn"
Jul 07 17:09:20.117673 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:20.117530    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4a027b2-defb-498c-adb5-61d2f95e6514-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn\" (UID: \"e4a027b2-defb-498c-adb5-61d2f95e6514\") " pod="kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn"
Jul 07 17:09:20.117673 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:20.117573    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4a027b2-defb-498c-adb5-61d2f95e6514-proxy-tls\") pod \"isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn\" (UID: \"e4a027b2-defb-498c-adb5-61d2f95e6514\") " pod="kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn"
Jul 07 17:09:20.117673 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:09:20.117672    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-serving-cert: secret "isvc-secondary-0c9a81-predictor-serving-cert" not found
Jul 07 17:09:20.117950 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:09:20.117761    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4a027b2-defb-498c-adb5-61d2f95e6514-proxy-tls podName:e4a027b2-defb-498c-adb5-61d2f95e6514 nodeName:}" failed. No retries permitted until 2026-07-07 17:09:20.617741554 +0000 UTC m=+1353.630751366 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e4a027b2-defb-498c-adb5-61d2f95e6514-proxy-tls") pod "isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn" (UID: "e4a027b2-defb-498c-adb5-61d2f95e6514") : secret "isvc-secondary-0c9a81-predictor-serving-cert" not found
Jul 07 17:09:20.117950 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:20.117896    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4a027b2-defb-498c-adb5-61d2f95e6514-kserve-provision-location\") pod \"isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn\" (UID: \"e4a027b2-defb-498c-adb5-61d2f95e6514\") " pod="kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn"
Jul 07 17:09:20.118259 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:20.118237    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4a027b2-defb-498c-adb5-61d2f95e6514-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn\" (UID: \"e4a027b2-defb-498c-adb5-61d2f95e6514\") " pod="kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn"
Jul 07 17:09:20.118345 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:20.118273    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4a027b2-defb-498c-adb5-61d2f95e6514-cabundle-cert\") pod \"isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn\" (UID: \"e4a027b2-defb-498c-adb5-61d2f95e6514\") " pod="kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn"
Jul 07 17:09:20.127767 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:20.127737    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvdcl\" (UniqueName: \"kubernetes.io/projected/e4a027b2-defb-498c-adb5-61d2f95e6514-kube-api-access-lvdcl\") pod \"isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn\" (UID: \"e4a027b2-defb-498c-adb5-61d2f95e6514\") " pod="kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn"
Jul 07 17:09:20.621561 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:20.621530    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4a027b2-defb-498c-adb5-61d2f95e6514-proxy-tls\") pod \"isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn\" (UID: \"e4a027b2-defb-498c-adb5-61d2f95e6514\") " pod="kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn"
Jul 07 17:09:20.624055 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:20.624024    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4a027b2-defb-498c-adb5-61d2f95e6514-proxy-tls\") pod \"isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn\" (UID: \"e4a027b2-defb-498c-adb5-61d2f95e6514\") " pod="kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn"
Jul 07 17:09:20.862544 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:20.862509    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn"
Jul 07 17:09:21.001034 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:21.001012    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn"]
Jul 07 17:09:21.002736 ip-10-0-141-189 kubenswrapper[2583]: W0707 17:09:21.002708    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4a027b2_defb_498c_adb5_61d2f95e6514.slice/crio-08778c2f50842183c44e3f586afc7682596e76c54eb1a1f3c4f79cb8a35c2e3a WatchSource:0}: Error finding container 08778c2f50842183c44e3f586afc7682596e76c54eb1a1f3c4f79cb8a35c2e3a: Status 404 returned error can't find the container with id 08778c2f50842183c44e3f586afc7682596e76c54eb1a1f3c4f79cb8a35c2e3a
Jul 07 17:09:21.634997 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:21.634963    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn" event={"ID":"e4a027b2-defb-498c-adb5-61d2f95e6514","Type":"ContainerStarted","Data":"cf3fa25498cb765018f8a5f7b3b4a2a923a2a30b671fe853734a7a97a0da43fc"}
Jul 07 17:09:21.635182 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:21.635018    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn" event={"ID":"e4a027b2-defb-498c-adb5-61d2f95e6514","Type":"ContainerStarted","Data":"08778c2f50842183c44e3f586afc7682596e76c54eb1a1f3c4f79cb8a35c2e3a"}
Jul 07 17:09:28.673017 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:28.672989    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn_e4a027b2-defb-498c-adb5-61d2f95e6514/storage-initializer/0.log"
Jul 07 17:09:28.673419 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:28.673030    2583 generic.go:356] "Generic (PLEG): container finished" podID="e4a027b2-defb-498c-adb5-61d2f95e6514" containerID="cf3fa25498cb765018f8a5f7b3b4a2a923a2a30b671fe853734a7a97a0da43fc" exitCode=1
Jul 07 17:09:28.673419 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:28.673090    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn" event={"ID":"e4a027b2-defb-498c-adb5-61d2f95e6514","Type":"ContainerDied","Data":"cf3fa25498cb765018f8a5f7b3b4a2a923a2a30b671fe853734a7a97a0da43fc"}
Jul 07 17:09:29.679922 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:29.679895    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn_e4a027b2-defb-498c-adb5-61d2f95e6514/storage-initializer/0.log"
Jul 07 17:09:29.680330 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:29.680016    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn" event={"ID":"e4a027b2-defb-498c-adb5-61d2f95e6514","Type":"ContainerStarted","Data":"ecbe3e94cfbf5301d8830512873d56342bb21624639031014581338d8c0fedfb"}
Jul 07 17:09:34.702940 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:34.702905    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn_e4a027b2-defb-498c-adb5-61d2f95e6514/storage-initializer/1.log"
Jul 07 17:09:34.703354 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:34.703301    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn_e4a027b2-defb-498c-adb5-61d2f95e6514/storage-initializer/0.log"
Jul 07 17:09:34.703354 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:34.703333    2583 generic.go:356] "Generic (PLEG): container finished" podID="e4a027b2-defb-498c-adb5-61d2f95e6514" containerID="ecbe3e94cfbf5301d8830512873d56342bb21624639031014581338d8c0fedfb" exitCode=1
Jul 07 17:09:34.703457 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:34.703373    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn" event={"ID":"e4a027b2-defb-498c-adb5-61d2f95e6514","Type":"ContainerDied","Data":"ecbe3e94cfbf5301d8830512873d56342bb21624639031014581338d8c0fedfb"}
Jul 07 17:09:34.703457 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:34.703428    2583 scope.go:117] "RemoveContainer" containerID="cf3fa25498cb765018f8a5f7b3b4a2a923a2a30b671fe853734a7a97a0da43fc"
Jul 07 17:09:34.703829 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:34.703809    2583 scope.go:117] "RemoveContainer" containerID="cf3fa25498cb765018f8a5f7b3b4a2a923a2a30b671fe853734a7a97a0da43fc"
Jul 07 17:09:34.714824 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:09:34.714792    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn_kserve-ci-e2e-test(e4a027b2-defb-498c-adb5-61d2f95e6514)\"" pod="kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn" podUID="e4a027b2-defb-498c-adb5-61d2f95e6514"
Jul 07 17:09:34.714824 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:09:34.714806    2583 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn_kserve-ci-e2e-test_e4a027b2-defb-498c-adb5-61d2f95e6514_0 in pod sandbox 08778c2f50842183c44e3f586afc7682596e76c54eb1a1f3c4f79cb8a35c2e3a from index: no such id: 'cf3fa25498cb765018f8a5f7b3b4a2a923a2a30b671fe853734a7a97a0da43fc'" containerID="cf3fa25498cb765018f8a5f7b3b4a2a923a2a30b671fe853734a7a97a0da43fc"
Jul 07 17:09:34.714963 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:34.714847    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf3fa25498cb765018f8a5f7b3b4a2a923a2a30b671fe853734a7a97a0da43fc"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn_kserve-ci-e2e-test_e4a027b2-defb-498c-adb5-61d2f95e6514_0 in pod sandbox 08778c2f50842183c44e3f586afc7682596e76c54eb1a1f3c4f79cb8a35c2e3a from index: no such id: 'cf3fa25498cb765018f8a5f7b3b4a2a923a2a30b671fe853734a7a97a0da43fc'"
Jul 07 17:09:35.709064 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:35.709035    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn_e4a027b2-defb-498c-adb5-61d2f95e6514/storage-initializer/1.log"
Jul 07 17:09:40.064517 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.064479    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn"]
Jul 07 17:09:40.187106 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.187073    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"]
Jul 07 17:09:40.187420 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.187393    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc" podUID="09cccda9-2bc2-4e97-99a9-ef8097cb931e" containerName="kserve-container" containerID="cri-o://1d0410f13c09859e79c4a927551e3c50dbff2a3aa4068aab4123d5ee7f8be0b0" gracePeriod=30
Jul 07 17:09:40.187511 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.187483    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc" podUID="09cccda9-2bc2-4e97-99a9-ef8097cb931e" containerName="kube-rbac-proxy" containerID="cri-o://d850db5bff1ada7b3367a0fe0e5047460721343cfef33c189e4507c2641ec7a8" gracePeriod=30
Jul 07 17:09:40.211782 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.211764    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn_e4a027b2-defb-498c-adb5-61d2f95e6514/storage-initializer/1.log"
Jul 07 17:09:40.215273 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.215256    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn"
Jul 07 17:09:40.297031 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.296999    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4a027b2-defb-498c-adb5-61d2f95e6514-kube-rbac-proxy-sar-config\") pod \"e4a027b2-defb-498c-adb5-61d2f95e6514\" (UID: \"e4a027b2-defb-498c-adb5-61d2f95e6514\") "
Jul 07 17:09:40.297244 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.297057    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4a027b2-defb-498c-adb5-61d2f95e6514-proxy-tls\") pod \"e4a027b2-defb-498c-adb5-61d2f95e6514\" (UID: \"e4a027b2-defb-498c-adb5-61d2f95e6514\") "
Jul 07 17:09:40.297244 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.297085    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4a027b2-defb-498c-adb5-61d2f95e6514-cabundle-cert\") pod \"e4a027b2-defb-498c-adb5-61d2f95e6514\" (UID: \"e4a027b2-defb-498c-adb5-61d2f95e6514\") "
Jul 07 17:09:40.297244 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.297117    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvdcl\" (UniqueName: \"kubernetes.io/projected/e4a027b2-defb-498c-adb5-61d2f95e6514-kube-api-access-lvdcl\") pod \"e4a027b2-defb-498c-adb5-61d2f95e6514\" (UID: \"e4a027b2-defb-498c-adb5-61d2f95e6514\") "
Jul 07 17:09:40.297244 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.297164    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4a027b2-defb-498c-adb5-61d2f95e6514-kserve-provision-location\") pod \"e4a027b2-defb-498c-adb5-61d2f95e6514\" (UID: \"e4a027b2-defb-498c-adb5-61d2f95e6514\") "
Jul 07 17:09:40.297488 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.297466    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4a027b2-defb-498c-adb5-61d2f95e6514-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e4a027b2-defb-498c-adb5-61d2f95e6514" (UID: "e4a027b2-defb-498c-adb5-61d2f95e6514"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:09:40.297488 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.297467    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4a027b2-defb-498c-adb5-61d2f95e6514-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e4a027b2-defb-498c-adb5-61d2f95e6514" (UID: "e4a027b2-defb-498c-adb5-61d2f95e6514"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:09:40.297563 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.297490    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4a027b2-defb-498c-adb5-61d2f95e6514-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e4a027b2-defb-498c-adb5-61d2f95e6514" (UID: "e4a027b2-defb-498c-adb5-61d2f95e6514"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:09:40.299163 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.299124    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4a027b2-defb-498c-adb5-61d2f95e6514-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e4a027b2-defb-498c-adb5-61d2f95e6514" (UID: "e4a027b2-defb-498c-adb5-61d2f95e6514"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:09:40.299441 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.299425    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4a027b2-defb-498c-adb5-61d2f95e6514-kube-api-access-lvdcl" (OuterVolumeSpecName: "kube-api-access-lvdcl") pod "e4a027b2-defb-498c-adb5-61d2f95e6514" (UID: "e4a027b2-defb-498c-adb5-61d2f95e6514"). InnerVolumeSpecName "kube-api-access-lvdcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:09:40.398437 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.398350    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4a027b2-defb-498c-adb5-61d2f95e6514-kserve-provision-location\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:09:40.398437 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.398380    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4a027b2-defb-498c-adb5-61d2f95e6514-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:09:40.398437 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.398391    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4a027b2-defb-498c-adb5-61d2f95e6514-proxy-tls\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:09:40.398437 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.398400    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4a027b2-defb-498c-adb5-61d2f95e6514-cabundle-cert\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:09:40.398437 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.398411    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvdcl\" (UniqueName: \"kubernetes.io/projected/e4a027b2-defb-498c-adb5-61d2f95e6514-kube-api-access-lvdcl\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:09:40.730889 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.730864    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn_e4a027b2-defb-498c-adb5-61d2f95e6514/storage-initializer/1.log"
Jul 07 17:09:40.731089 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.730989    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn"
Jul 07 17:09:40.731089 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.731004    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn" event={"ID":"e4a027b2-defb-498c-adb5-61d2f95e6514","Type":"ContainerDied","Data":"08778c2f50842183c44e3f586afc7682596e76c54eb1a1f3c4f79cb8a35c2e3a"}
Jul 07 17:09:40.731089 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.731076    2583 scope.go:117] "RemoveContainer" containerID="ecbe3e94cfbf5301d8830512873d56342bb21624639031014581338d8c0fedfb"
Jul 07 17:09:40.733655 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.733624    2583 generic.go:356] "Generic (PLEG): container finished" podID="09cccda9-2bc2-4e97-99a9-ef8097cb931e" containerID="d850db5bff1ada7b3367a0fe0e5047460721343cfef33c189e4507c2641ec7a8" exitCode=2
Jul 07 17:09:40.733771 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.733688    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc" event={"ID":"09cccda9-2bc2-4e97-99a9-ef8097cb931e","Type":"ContainerDied","Data":"d850db5bff1ada7b3367a0fe0e5047460721343cfef33c189e4507c2641ec7a8"}
Jul 07 17:09:40.781428 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.781392    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn"]
Jul 07 17:09:40.786109 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:40.786080    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-0c9a81-predictor-64c9c78b78-v96zn"]
Jul 07 17:09:41.586181 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:41.586122    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4a027b2-defb-498c-adb5-61d2f95e6514" path="/var/lib/kubelet/pods/e4a027b2-defb-498c-adb5-61d2f95e6514/volumes"
Jul 07 17:09:43.280466 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:43.280431    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc" podUID="09cccda9-2bc2-4e97-99a9-ef8097cb931e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 07 17:09:43.285803 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:43.285775    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc" podUID="09cccda9-2bc2-4e97-99a9-ef8097cb931e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 17:09:44.934511 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:44.934485    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"
Jul 07 17:09:45.039969 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.039866    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09cccda9-2bc2-4e97-99a9-ef8097cb931e-kserve-provision-location\") pod \"09cccda9-2bc2-4e97-99a9-ef8097cb931e\" (UID: \"09cccda9-2bc2-4e97-99a9-ef8097cb931e\") "
Jul 07 17:09:45.039969 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.039940    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09cccda9-2bc2-4e97-99a9-ef8097cb931e-kube-rbac-proxy-sar-config\") pod \"09cccda9-2bc2-4e97-99a9-ef8097cb931e\" (UID: \"09cccda9-2bc2-4e97-99a9-ef8097cb931e\") "
Jul 07 17:09:45.040249 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.040023    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09cccda9-2bc2-4e97-99a9-ef8097cb931e-proxy-tls\") pod \"09cccda9-2bc2-4e97-99a9-ef8097cb931e\" (UID: \"09cccda9-2bc2-4e97-99a9-ef8097cb931e\") "
Jul 07 17:09:45.040249 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.040097    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wnhj\" (UniqueName: \"kubernetes.io/projected/09cccda9-2bc2-4e97-99a9-ef8097cb931e-kube-api-access-7wnhj\") pod \"09cccda9-2bc2-4e97-99a9-ef8097cb931e\" (UID: \"09cccda9-2bc2-4e97-99a9-ef8097cb931e\") "
Jul 07 17:09:45.040249 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.040118    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09cccda9-2bc2-4e97-99a9-ef8097cb931e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "09cccda9-2bc2-4e97-99a9-ef8097cb931e" (UID: "09cccda9-2bc2-4e97-99a9-ef8097cb931e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:09:45.040249 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.040178    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09cccda9-2bc2-4e97-99a9-ef8097cb931e-cabundle-cert\") pod \"09cccda9-2bc2-4e97-99a9-ef8097cb931e\" (UID: \"09cccda9-2bc2-4e97-99a9-ef8097cb931e\") "
Jul 07 17:09:45.040452 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.040363    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09cccda9-2bc2-4e97-99a9-ef8097cb931e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "09cccda9-2bc2-4e97-99a9-ef8097cb931e" (UID: "09cccda9-2bc2-4e97-99a9-ef8097cb931e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:09:45.040508 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.040482    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09cccda9-2bc2-4e97-99a9-ef8097cb931e-kserve-provision-location\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:09:45.040508 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.040499    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09cccda9-2bc2-4e97-99a9-ef8097cb931e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:09:45.040725 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.040700    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09cccda9-2bc2-4e97-99a9-ef8097cb931e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "09cccda9-2bc2-4e97-99a9-ef8097cb931e" (UID: "09cccda9-2bc2-4e97-99a9-ef8097cb931e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:09:45.042468 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.042449    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09cccda9-2bc2-4e97-99a9-ef8097cb931e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "09cccda9-2bc2-4e97-99a9-ef8097cb931e" (UID: "09cccda9-2bc2-4e97-99a9-ef8097cb931e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:09:45.042541 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.042483    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09cccda9-2bc2-4e97-99a9-ef8097cb931e-kube-api-access-7wnhj" (OuterVolumeSpecName: "kube-api-access-7wnhj") pod "09cccda9-2bc2-4e97-99a9-ef8097cb931e" (UID: "09cccda9-2bc2-4e97-99a9-ef8097cb931e"). InnerVolumeSpecName "kube-api-access-7wnhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:09:45.141414 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.141376    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09cccda9-2bc2-4e97-99a9-ef8097cb931e-proxy-tls\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:09:45.141414 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.141409    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wnhj\" (UniqueName: \"kubernetes.io/projected/09cccda9-2bc2-4e97-99a9-ef8097cb931e-kube-api-access-7wnhj\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:09:45.141414 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.141422    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09cccda9-2bc2-4e97-99a9-ef8097cb931e-cabundle-cert\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:09:45.204277 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.204242    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb"]
Jul 07 17:09:45.204673 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.204652    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4a027b2-defb-498c-adb5-61d2f95e6514" containerName="storage-initializer"
Jul 07 17:09:45.204673 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.204672    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4a027b2-defb-498c-adb5-61d2f95e6514" containerName="storage-initializer"
Jul 07 17:09:45.204834 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.204681    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09cccda9-2bc2-4e97-99a9-ef8097cb931e" containerName="kube-rbac-proxy"
Jul 07 17:09:45.204834 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.204686    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="09cccda9-2bc2-4e97-99a9-ef8097cb931e" containerName="kube-rbac-proxy"
Jul 07 17:09:45.204834 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.204695    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09cccda9-2bc2-4e97-99a9-ef8097cb931e" containerName="kserve-container"
Jul 07 17:09:45.204834 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.204700    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="09cccda9-2bc2-4e97-99a9-ef8097cb931e" containerName="kserve-container"
Jul 07 17:09:45.204834 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.204718    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09cccda9-2bc2-4e97-99a9-ef8097cb931e" containerName="storage-initializer"
Jul 07 17:09:45.204834 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.204724    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="09cccda9-2bc2-4e97-99a9-ef8097cb931e" containerName="storage-initializer"
Jul 07 17:09:45.204834 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.204781    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="09cccda9-2bc2-4e97-99a9-ef8097cb931e" containerName="kube-rbac-proxy"
Jul 07 17:09:45.204834 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.204790    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="09cccda9-2bc2-4e97-99a9-ef8097cb931e" containerName="kserve-container"
Jul 07 17:09:45.204834 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.204795    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4a027b2-defb-498c-adb5-61d2f95e6514" containerName="storage-initializer"
Jul 07 17:09:45.205100 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.204865    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4a027b2-defb-498c-adb5-61d2f95e6514" containerName="storage-initializer"
Jul 07 17:09:45.205100 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.204872    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4a027b2-defb-498c-adb5-61d2f95e6514" containerName="storage-initializer"
Jul 07 17:09:45.205100 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.204909    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4a027b2-defb-498c-adb5-61d2f95e6514" containerName="storage-initializer"
Jul 07 17:09:45.207983 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.207964    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb"
Jul 07 17:09:45.210789 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.210762    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-749524-predictor-serving-cert\""
Jul 07 17:09:45.210789 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.210766    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-749524\""
Jul 07 17:09:45.210967 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.210794    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-749524-dockercfg-xm527\""
Jul 07 17:09:45.211110 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.211094    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-749524-kube-rbac-proxy-sar-config\""
Jul 07 17:09:45.220641 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.220620    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb"]
Jul 07 17:09:45.342905 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.342802    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e527ba40-bcae-4734-9a3f-0bed7927b15a-cabundle-cert\") pod \"isvc-init-fail-749524-predictor-859677d44b-hsbzb\" (UID: \"e527ba40-bcae-4734-9a3f-0bed7927b15a\") " pod="kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb"
Jul 07 17:09:45.342905 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.342870    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e527ba40-bcae-4734-9a3f-0bed7927b15a-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-749524-predictor-859677d44b-hsbzb\" (UID: \"e527ba40-bcae-4734-9a3f-0bed7927b15a\") " pod="kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb"
Jul 07 17:09:45.343100 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.342981    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e527ba40-bcae-4734-9a3f-0bed7927b15a-proxy-tls\") pod \"isvc-init-fail-749524-predictor-859677d44b-hsbzb\" (UID: \"e527ba40-bcae-4734-9a3f-0bed7927b15a\") " pod="kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb"
Jul 07 17:09:45.343100 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.343037    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwzs5\" (UniqueName: \"kubernetes.io/projected/e527ba40-bcae-4734-9a3f-0bed7927b15a-kube-api-access-hwzs5\") pod \"isvc-init-fail-749524-predictor-859677d44b-hsbzb\" (UID: \"e527ba40-bcae-4734-9a3f-0bed7927b15a\") " pod="kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb"
Jul 07 17:09:45.343100 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.343072    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e527ba40-bcae-4734-9a3f-0bed7927b15a-kserve-provision-location\") pod \"isvc-init-fail-749524-predictor-859677d44b-hsbzb\" (UID: \"e527ba40-bcae-4734-9a3f-0bed7927b15a\") " pod="kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb"
Jul 07 17:09:45.443884 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.443845    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e527ba40-bcae-4734-9a3f-0bed7927b15a-proxy-tls\") pod \"isvc-init-fail-749524-predictor-859677d44b-hsbzb\" (UID: \"e527ba40-bcae-4734-9a3f-0bed7927b15a\") " pod="kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb"
Jul 07 17:09:45.444061 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.443897    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwzs5\" (UniqueName: \"kubernetes.io/projected/e527ba40-bcae-4734-9a3f-0bed7927b15a-kube-api-access-hwzs5\") pod \"isvc-init-fail-749524-predictor-859677d44b-hsbzb\" (UID: \"e527ba40-bcae-4734-9a3f-0bed7927b15a\") " pod="kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb"
Jul 07 17:09:45.444061 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.443918    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e527ba40-bcae-4734-9a3f-0bed7927b15a-kserve-provision-location\") pod \"isvc-init-fail-749524-predictor-859677d44b-hsbzb\" (UID: \"e527ba40-bcae-4734-9a3f-0bed7927b15a\") " pod="kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb"
Jul 07 17:09:45.444061 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.443975    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e527ba40-bcae-4734-9a3f-0bed7927b15a-cabundle-cert\") pod \"isvc-init-fail-749524-predictor-859677d44b-hsbzb\" (UID: \"e527ba40-bcae-4734-9a3f-0bed7927b15a\") " pod="kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb"
Jul 07 17:09:45.444061 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:09:45.444001    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-749524-predictor-serving-cert: secret "isvc-init-fail-749524-predictor-serving-cert" not found
Jul 07 17:09:45.444061 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:09:45.444057    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e527ba40-bcae-4734-9a3f-0bed7927b15a-proxy-tls podName:e527ba40-bcae-4734-9a3f-0bed7927b15a nodeName:}" failed. No retries permitted until 2026-07-07 17:09:45.944039217 +0000 UTC m=+1378.957049015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e527ba40-bcae-4734-9a3f-0bed7927b15a-proxy-tls") pod "isvc-init-fail-749524-predictor-859677d44b-hsbzb" (UID: "e527ba40-bcae-4734-9a3f-0bed7927b15a") : secret "isvc-init-fail-749524-predictor-serving-cert" not found
Jul 07 17:09:45.444061 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.444045    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e527ba40-bcae-4734-9a3f-0bed7927b15a-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-749524-predictor-859677d44b-hsbzb\" (UID: \"e527ba40-bcae-4734-9a3f-0bed7927b15a\") " pod="kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb"
Jul 07 17:09:45.444401 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.444331    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e527ba40-bcae-4734-9a3f-0bed7927b15a-kserve-provision-location\") pod \"isvc-init-fail-749524-predictor-859677d44b-hsbzb\" (UID: \"e527ba40-bcae-4734-9a3f-0bed7927b15a\") " pod="kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb"
Jul 07 17:09:45.444673 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.444650    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e527ba40-bcae-4734-9a3f-0bed7927b15a-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-749524-predictor-859677d44b-hsbzb\" (UID: \"e527ba40-bcae-4734-9a3f-0bed7927b15a\") " pod="kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb"
Jul 07 17:09:45.444763 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.444682    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e527ba40-bcae-4734-9a3f-0bed7927b15a-cabundle-cert\") pod \"isvc-init-fail-749524-predictor-859677d44b-hsbzb\" (UID: \"e527ba40-bcae-4734-9a3f-0bed7927b15a\") " pod="kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb"
Jul 07 17:09:45.458401 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.458379    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwzs5\" (UniqueName: \"kubernetes.io/projected/e527ba40-bcae-4734-9a3f-0bed7927b15a-kube-api-access-hwzs5\") pod \"isvc-init-fail-749524-predictor-859677d44b-hsbzb\" (UID: \"e527ba40-bcae-4734-9a3f-0bed7927b15a\") " pod="kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb"
Jul 07 17:09:45.759745 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.759709    2583 generic.go:356] "Generic (PLEG): container finished" podID="09cccda9-2bc2-4e97-99a9-ef8097cb931e" containerID="1d0410f13c09859e79c4a927551e3c50dbff2a3aa4068aab4123d5ee7f8be0b0" exitCode=0
Jul 07 17:09:45.759745 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.759749    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc" event={"ID":"09cccda9-2bc2-4e97-99a9-ef8097cb931e","Type":"ContainerDied","Data":"1d0410f13c09859e79c4a927551e3c50dbff2a3aa4068aab4123d5ee7f8be0b0"}
Jul 07 17:09:45.759950 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.759788    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"
Jul 07 17:09:45.759950 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.759832    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc" event={"ID":"09cccda9-2bc2-4e97-99a9-ef8097cb931e","Type":"ContainerDied","Data":"69d0317c89dd9ec0c89781ee1e8e6c1201fbc3331e09655ecbcbe6d920a9270c"}
Jul 07 17:09:45.759950 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.759860    2583 scope.go:117] "RemoveContainer" containerID="d850db5bff1ada7b3367a0fe0e5047460721343cfef33c189e4507c2641ec7a8"
Jul 07 17:09:45.767993 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.767977    2583 scope.go:117] "RemoveContainer" containerID="1d0410f13c09859e79c4a927551e3c50dbff2a3aa4068aab4123d5ee7f8be0b0"
Jul 07 17:09:45.775530 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.775513    2583 scope.go:117] "RemoveContainer" containerID="c52f58b6599f1a9f905298ffaf860f5a953c5b47fbf6d0ce3e65fe0a352de945"
Jul 07 17:09:45.782905 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.782883    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"]
Jul 07 17:09:45.784022 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.784005    2583 scope.go:117] "RemoveContainer" containerID="d850db5bff1ada7b3367a0fe0e5047460721343cfef33c189e4507c2641ec7a8"
Jul 07 17:09:45.784392 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:09:45.784365    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d850db5bff1ada7b3367a0fe0e5047460721343cfef33c189e4507c2641ec7a8\": container with ID starting with d850db5bff1ada7b3367a0fe0e5047460721343cfef33c189e4507c2641ec7a8 not found: ID does not exist" containerID="d850db5bff1ada7b3367a0fe0e5047460721343cfef33c189e4507c2641ec7a8"
Jul 07 17:09:45.784469 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.784401    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d850db5bff1ada7b3367a0fe0e5047460721343cfef33c189e4507c2641ec7a8"} err="failed to get container status \"d850db5bff1ada7b3367a0fe0e5047460721343cfef33c189e4507c2641ec7a8\": rpc error: code = NotFound desc = could not find container \"d850db5bff1ada7b3367a0fe0e5047460721343cfef33c189e4507c2641ec7a8\": container with ID starting with d850db5bff1ada7b3367a0fe0e5047460721343cfef33c189e4507c2641ec7a8 not found: ID does not exist"
Jul 07 17:09:45.784469 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.784423    2583 scope.go:117] "RemoveContainer" containerID="1d0410f13c09859e79c4a927551e3c50dbff2a3aa4068aab4123d5ee7f8be0b0"
Jul 07 17:09:45.784718 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:09:45.784700    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d0410f13c09859e79c4a927551e3c50dbff2a3aa4068aab4123d5ee7f8be0b0\": container with ID starting with 1d0410f13c09859e79c4a927551e3c50dbff2a3aa4068aab4123d5ee7f8be0b0 not found: ID does not exist" containerID="1d0410f13c09859e79c4a927551e3c50dbff2a3aa4068aab4123d5ee7f8be0b0"
Jul 07 17:09:45.784758 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.784724    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d0410f13c09859e79c4a927551e3c50dbff2a3aa4068aab4123d5ee7f8be0b0"} err="failed to get container status \"1d0410f13c09859e79c4a927551e3c50dbff2a3aa4068aab4123d5ee7f8be0b0\": rpc error: code = NotFound desc = could not find container \"1d0410f13c09859e79c4a927551e3c50dbff2a3aa4068aab4123d5ee7f8be0b0\": container with ID starting with 1d0410f13c09859e79c4a927551e3c50dbff2a3aa4068aab4123d5ee7f8be0b0 not found: ID does not exist"
Jul 07 17:09:45.784758 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.784739    2583 scope.go:117] "RemoveContainer" containerID="c52f58b6599f1a9f905298ffaf860f5a953c5b47fbf6d0ce3e65fe0a352de945"
Jul 07 17:09:45.784979 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:09:45.784949    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c52f58b6599f1a9f905298ffaf860f5a953c5b47fbf6d0ce3e65fe0a352de945\": container with ID starting with c52f58b6599f1a9f905298ffaf860f5a953c5b47fbf6d0ce3e65fe0a352de945 not found: ID does not exist" containerID="c52f58b6599f1a9f905298ffaf860f5a953c5b47fbf6d0ce3e65fe0a352de945"
Jul 07 17:09:45.785021 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.784979    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c52f58b6599f1a9f905298ffaf860f5a953c5b47fbf6d0ce3e65fe0a352de945"} err="failed to get container status \"c52f58b6599f1a9f905298ffaf860f5a953c5b47fbf6d0ce3e65fe0a352de945\": rpc error: code = NotFound desc = could not find container \"c52f58b6599f1a9f905298ffaf860f5a953c5b47fbf6d0ce3e65fe0a352de945\": container with ID starting with c52f58b6599f1a9f905298ffaf860f5a953c5b47fbf6d0ce3e65fe0a352de945 not found: ID does not exist"
Jul 07 17:09:45.785445 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.785427    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-0c9a81-predictor-5566d97cdf-5prxc"]
Jul 07 17:09:45.948295 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.948262    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e527ba40-bcae-4734-9a3f-0bed7927b15a-proxy-tls\") pod \"isvc-init-fail-749524-predictor-859677d44b-hsbzb\" (UID: \"e527ba40-bcae-4734-9a3f-0bed7927b15a\") " pod="kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb"
Jul 07 17:09:45.950866 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:45.950840    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e527ba40-bcae-4734-9a3f-0bed7927b15a-proxy-tls\") pod \"isvc-init-fail-749524-predictor-859677d44b-hsbzb\" (UID: \"e527ba40-bcae-4734-9a3f-0bed7927b15a\") " pod="kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb"
Jul 07 17:09:46.118125 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:46.118083    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb"
Jul 07 17:09:46.256770 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:46.256745    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb"]
Jul 07 17:09:46.259916 ip-10-0-141-189 kubenswrapper[2583]: W0707 17:09:46.259883    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode527ba40_bcae_4734_9a3f_0bed7927b15a.slice/crio-d5f800a4481241e9041050639b9573f53862427c1e51439b9bdb9c3b996a4a80 WatchSource:0}: Error finding container d5f800a4481241e9041050639b9573f53862427c1e51439b9bdb9c3b996a4a80: Status 404 returned error can't find the container with id d5f800a4481241e9041050639b9573f53862427c1e51439b9bdb9c3b996a4a80
Jul 07 17:09:46.766421 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:46.766386    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb" event={"ID":"e527ba40-bcae-4734-9a3f-0bed7927b15a","Type":"ContainerStarted","Data":"ceaff5beb66d2dbe11ce5240787bda40f3cb427ec6ef5f375a5b6f8cb738e9d4"}
Jul 07 17:09:46.766602 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:46.766497    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb" event={"ID":"e527ba40-bcae-4734-9a3f-0bed7927b15a","Type":"ContainerStarted","Data":"d5f800a4481241e9041050639b9573f53862427c1e51439b9bdb9c3b996a4a80"}
Jul 07 17:09:47.587691 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:47.587658    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09cccda9-2bc2-4e97-99a9-ef8097cb931e" path="/var/lib/kubelet/pods/09cccda9-2bc2-4e97-99a9-ef8097cb931e/volumes"
Jul 07 17:09:50.785397 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:50.785316    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-749524-predictor-859677d44b-hsbzb_e527ba40-bcae-4734-9a3f-0bed7927b15a/storage-initializer/0.log"
Jul 07 17:09:50.785397 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:50.785357    2583 generic.go:356] "Generic (PLEG): container finished" podID="e527ba40-bcae-4734-9a3f-0bed7927b15a" containerID="ceaff5beb66d2dbe11ce5240787bda40f3cb427ec6ef5f375a5b6f8cb738e9d4" exitCode=1
Jul 07 17:09:50.785828 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:50.785398    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb" event={"ID":"e527ba40-bcae-4734-9a3f-0bed7927b15a","Type":"ContainerDied","Data":"ceaff5beb66d2dbe11ce5240787bda40f3cb427ec6ef5f375a5b6f8cb738e9d4"}
Jul 07 17:09:51.790815 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:51.790781    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-749524-predictor-859677d44b-hsbzb_e527ba40-bcae-4734-9a3f-0bed7927b15a/storage-initializer/0.log"
Jul 07 17:09:51.791425 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:51.790889    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb" event={"ID":"e527ba40-bcae-4734-9a3f-0bed7927b15a","Type":"ContainerStarted","Data":"62efac70737ad3bc5339847ebe6fda98e73fdb5b3356a64b2dbb7d712bcb7eaa"}
Jul 07 17:09:55.209113 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.209082    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb"]
Jul 07 17:09:55.209504 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.209304    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb" podUID="e527ba40-bcae-4734-9a3f-0bed7927b15a" containerName="storage-initializer" containerID="cri-o://62efac70737ad3bc5339847ebe6fda98e73fdb5b3356a64b2dbb7d712bcb7eaa" gracePeriod=30
Jul 07 17:09:55.343817 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.343795    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-749524-predictor-859677d44b-hsbzb_e527ba40-bcae-4734-9a3f-0bed7927b15a/storage-initializer/1.log"
Jul 07 17:09:55.344260 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.344245    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-749524-predictor-859677d44b-hsbzb_e527ba40-bcae-4734-9a3f-0bed7927b15a/storage-initializer/0.log"
Jul 07 17:09:55.346503 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.346488    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb"
Jul 07 17:09:55.431629 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.431538    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e527ba40-bcae-4734-9a3f-0bed7927b15a-kube-rbac-proxy-sar-config\") pod \"e527ba40-bcae-4734-9a3f-0bed7927b15a\" (UID: \"e527ba40-bcae-4734-9a3f-0bed7927b15a\") "
Jul 07 17:09:55.431629 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.431589    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e527ba40-bcae-4734-9a3f-0bed7927b15a-proxy-tls\") pod \"e527ba40-bcae-4734-9a3f-0bed7927b15a\" (UID: \"e527ba40-bcae-4734-9a3f-0bed7927b15a\") "
Jul 07 17:09:55.431855 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.431636    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e527ba40-bcae-4734-9a3f-0bed7927b15a-cabundle-cert\") pod \"e527ba40-bcae-4734-9a3f-0bed7927b15a\" (UID: \"e527ba40-bcae-4734-9a3f-0bed7927b15a\") "
Jul 07 17:09:55.431855 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.431688    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e527ba40-bcae-4734-9a3f-0bed7927b15a-kserve-provision-location\") pod \"e527ba40-bcae-4734-9a3f-0bed7927b15a\" (UID: \"e527ba40-bcae-4734-9a3f-0bed7927b15a\") "
Jul 07 17:09:55.431855 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.431734    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwzs5\" (UniqueName: \"kubernetes.io/projected/e527ba40-bcae-4734-9a3f-0bed7927b15a-kube-api-access-hwzs5\") pod \"e527ba40-bcae-4734-9a3f-0bed7927b15a\" (UID: \"e527ba40-bcae-4734-9a3f-0bed7927b15a\") "
Jul 07 17:09:55.432020 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.431940    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e527ba40-bcae-4734-9a3f-0bed7927b15a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e527ba40-bcae-4734-9a3f-0bed7927b15a" (UID: "e527ba40-bcae-4734-9a3f-0bed7927b15a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:09:55.432020 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.431984    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e527ba40-bcae-4734-9a3f-0bed7927b15a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e527ba40-bcae-4734-9a3f-0bed7927b15a" (UID: "e527ba40-bcae-4734-9a3f-0bed7927b15a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:09:55.432257 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.432038    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e527ba40-bcae-4734-9a3f-0bed7927b15a-kserve-provision-location\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:09:55.432257 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.432073    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e527ba40-bcae-4734-9a3f-0bed7927b15a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e527ba40-bcae-4734-9a3f-0bed7927b15a" (UID: "e527ba40-bcae-4734-9a3f-0bed7927b15a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:09:55.433938 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.433917    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e527ba40-bcae-4734-9a3f-0bed7927b15a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e527ba40-bcae-4734-9a3f-0bed7927b15a" (UID: "e527ba40-bcae-4734-9a3f-0bed7927b15a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:09:55.433997 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.433965    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e527ba40-bcae-4734-9a3f-0bed7927b15a-kube-api-access-hwzs5" (OuterVolumeSpecName: "kube-api-access-hwzs5") pod "e527ba40-bcae-4734-9a3f-0bed7927b15a" (UID: "e527ba40-bcae-4734-9a3f-0bed7927b15a"). InnerVolumeSpecName "kube-api-access-hwzs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:09:55.532588 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.532554    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwzs5\" (UniqueName: \"kubernetes.io/projected/e527ba40-bcae-4734-9a3f-0bed7927b15a-kube-api-access-hwzs5\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:09:55.532588 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.532582    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e527ba40-bcae-4734-9a3f-0bed7927b15a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:09:55.532588 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.532592    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e527ba40-bcae-4734-9a3f-0bed7927b15a-proxy-tls\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:09:55.532588 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.532601    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e527ba40-bcae-4734-9a3f-0bed7927b15a-cabundle-cert\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:09:55.809107 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.809075    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-749524-predictor-859677d44b-hsbzb_e527ba40-bcae-4734-9a3f-0bed7927b15a/storage-initializer/1.log"
Jul 07 17:09:55.809523 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.809504    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-749524-predictor-859677d44b-hsbzb_e527ba40-bcae-4734-9a3f-0bed7927b15a/storage-initializer/0.log"
Jul 07 17:09:55.809611 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.809543    2583 generic.go:356] "Generic (PLEG): container finished" podID="e527ba40-bcae-4734-9a3f-0bed7927b15a" containerID="62efac70737ad3bc5339847ebe6fda98e73fdb5b3356a64b2dbb7d712bcb7eaa" exitCode=1
Jul 07 17:09:55.809611 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.809573    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb" event={"ID":"e527ba40-bcae-4734-9a3f-0bed7927b15a","Type":"ContainerDied","Data":"62efac70737ad3bc5339847ebe6fda98e73fdb5b3356a64b2dbb7d712bcb7eaa"}
Jul 07 17:09:55.809706 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.809620    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb" event={"ID":"e527ba40-bcae-4734-9a3f-0bed7927b15a","Type":"ContainerDied","Data":"d5f800a4481241e9041050639b9573f53862427c1e51439b9bdb9c3b996a4a80"}
Jul 07 17:09:55.809706 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.809622    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb"
Jul 07 17:09:55.809706 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.809637    2583 scope.go:117] "RemoveContainer" containerID="62efac70737ad3bc5339847ebe6fda98e73fdb5b3356a64b2dbb7d712bcb7eaa"
Jul 07 17:09:55.818167 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.818154    2583 scope.go:117] "RemoveContainer" containerID="ceaff5beb66d2dbe11ce5240787bda40f3cb427ec6ef5f375a5b6f8cb738e9d4"
Jul 07 17:09:55.825569 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.825553    2583 scope.go:117] "RemoveContainer" containerID="62efac70737ad3bc5339847ebe6fda98e73fdb5b3356a64b2dbb7d712bcb7eaa"
Jul 07 17:09:55.825804 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:09:55.825788    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62efac70737ad3bc5339847ebe6fda98e73fdb5b3356a64b2dbb7d712bcb7eaa\": container with ID starting with 62efac70737ad3bc5339847ebe6fda98e73fdb5b3356a64b2dbb7d712bcb7eaa not found: ID does not exist" containerID="62efac70737ad3bc5339847ebe6fda98e73fdb5b3356a64b2dbb7d712bcb7eaa"
Jul 07 17:09:55.825857 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.825812    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62efac70737ad3bc5339847ebe6fda98e73fdb5b3356a64b2dbb7d712bcb7eaa"} err="failed to get container status \"62efac70737ad3bc5339847ebe6fda98e73fdb5b3356a64b2dbb7d712bcb7eaa\": rpc error: code = NotFound desc = could not find container \"62efac70737ad3bc5339847ebe6fda98e73fdb5b3356a64b2dbb7d712bcb7eaa\": container with ID starting with 62efac70737ad3bc5339847ebe6fda98e73fdb5b3356a64b2dbb7d712bcb7eaa not found: ID does not exist"
Jul 07 17:09:55.825857 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.825828    2583 scope.go:117] "RemoveContainer" containerID="ceaff5beb66d2dbe11ce5240787bda40f3cb427ec6ef5f375a5b6f8cb738e9d4"
Jul 07 17:09:55.826051 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:09:55.826033    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceaff5beb66d2dbe11ce5240787bda40f3cb427ec6ef5f375a5b6f8cb738e9d4\": container with ID starting with ceaff5beb66d2dbe11ce5240787bda40f3cb427ec6ef5f375a5b6f8cb738e9d4 not found: ID does not exist" containerID="ceaff5beb66d2dbe11ce5240787bda40f3cb427ec6ef5f375a5b6f8cb738e9d4"
Jul 07 17:09:55.826107 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.826057    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceaff5beb66d2dbe11ce5240787bda40f3cb427ec6ef5f375a5b6f8cb738e9d4"} err="failed to get container status \"ceaff5beb66d2dbe11ce5240787bda40f3cb427ec6ef5f375a5b6f8cb738e9d4\": rpc error: code = NotFound desc = could not find container \"ceaff5beb66d2dbe11ce5240787bda40f3cb427ec6ef5f375a5b6f8cb738e9d4\": container with ID starting with ceaff5beb66d2dbe11ce5240787bda40f3cb427ec6ef5f375a5b6f8cb738e9d4 not found: ID does not exist"
Jul 07 17:09:55.852781 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.852750    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb"]
Jul 07 17:09:55.858592 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:55.858566    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-749524-predictor-859677d44b-hsbzb"]
Jul 07 17:09:57.586719 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:09:57.586688    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e527ba40-bcae-4734-9a3f-0bed7927b15a" path="/var/lib/kubelet/pods/e527ba40-bcae-4734-9a3f-0bed7927b15a/volumes"
Jul 07 17:10:00.419642 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.419603    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"]
Jul 07 17:10:00.420011 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.419962    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e527ba40-bcae-4734-9a3f-0bed7927b15a" containerName="storage-initializer"
Jul 07 17:10:00.420011 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.419975    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e527ba40-bcae-4734-9a3f-0bed7927b15a" containerName="storage-initializer"
Jul 07 17:10:00.420011 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.419988    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e527ba40-bcae-4734-9a3f-0bed7927b15a" containerName="storage-initializer"
Jul 07 17:10:00.420011 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.420001    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e527ba40-bcae-4734-9a3f-0bed7927b15a" containerName="storage-initializer"
Jul 07 17:10:00.420188 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.420064    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e527ba40-bcae-4734-9a3f-0bed7927b15a" containerName="storage-initializer"
Jul 07 17:10:00.420188 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.420072    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e527ba40-bcae-4734-9a3f-0bed7927b15a" containerName="storage-initializer"
Jul 07 17:10:00.424694 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.424672    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"
Jul 07 17:10:00.427166 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.427128    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-msz67\""
Jul 07 17:10:00.427301 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.427244    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-963f7-predictor-serving-cert\""
Jul 07 17:10:00.427635 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.427617    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 17:10:00.427746 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.427619    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-963f7-kube-rbac-proxy-sar-config\""
Jul 07 17:10:00.427844 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.427829    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 17:10:00.428720 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.428703    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 17:10:00.428803 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.428702    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 17:10:00.434592 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.434566    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"]
Jul 07 17:10:00.474619 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.474584    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da060526-210b-45d0-a402-d76141a73705-kserve-provision-location\") pod \"raw-sklearn-963f7-predictor-cb9b4fb64-dz962\" (UID: \"da060526-210b-45d0-a402-d76141a73705\") " pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"
Jul 07 17:10:00.474619 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.474621    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da060526-210b-45d0-a402-d76141a73705-cabundle-cert\") pod \"raw-sklearn-963f7-predictor-cb9b4fb64-dz962\" (UID: \"da060526-210b-45d0-a402-d76141a73705\") " pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"
Jul 07 17:10:00.474836 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.474642    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da060526-210b-45d0-a402-d76141a73705-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-963f7-predictor-cb9b4fb64-dz962\" (UID: \"da060526-210b-45d0-a402-d76141a73705\") " pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"
Jul 07 17:10:00.474836 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.474731    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da060526-210b-45d0-a402-d76141a73705-proxy-tls\") pod \"raw-sklearn-963f7-predictor-cb9b4fb64-dz962\" (UID: \"da060526-210b-45d0-a402-d76141a73705\") " pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"
Jul 07 17:10:00.474836 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.474756    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l5kd\" (UniqueName: \"kubernetes.io/projected/da060526-210b-45d0-a402-d76141a73705-kube-api-access-8l5kd\") pod \"raw-sklearn-963f7-predictor-cb9b4fb64-dz962\" (UID: \"da060526-210b-45d0-a402-d76141a73705\") " pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"
Jul 07 17:10:00.575636 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.575602    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da060526-210b-45d0-a402-d76141a73705-proxy-tls\") pod \"raw-sklearn-963f7-predictor-cb9b4fb64-dz962\" (UID: \"da060526-210b-45d0-a402-d76141a73705\") " pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"
Jul 07 17:10:00.575636 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.575638    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l5kd\" (UniqueName: \"kubernetes.io/projected/da060526-210b-45d0-a402-d76141a73705-kube-api-access-8l5kd\") pod \"raw-sklearn-963f7-predictor-cb9b4fb64-dz962\" (UID: \"da060526-210b-45d0-a402-d76141a73705\") " pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"
Jul 07 17:10:00.575886 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.575710    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da060526-210b-45d0-a402-d76141a73705-kserve-provision-location\") pod \"raw-sklearn-963f7-predictor-cb9b4fb64-dz962\" (UID: \"da060526-210b-45d0-a402-d76141a73705\") " pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"
Jul 07 17:10:00.575886 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.575731    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da060526-210b-45d0-a402-d76141a73705-cabundle-cert\") pod \"raw-sklearn-963f7-predictor-cb9b4fb64-dz962\" (UID: \"da060526-210b-45d0-a402-d76141a73705\") " pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"
Jul 07 17:10:00.575886 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.575747    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da060526-210b-45d0-a402-d76141a73705-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-963f7-predictor-cb9b4fb64-dz962\" (UID: \"da060526-210b-45d0-a402-d76141a73705\") " pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"
Jul 07 17:10:00.576188 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.576167    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da060526-210b-45d0-a402-d76141a73705-kserve-provision-location\") pod \"raw-sklearn-963f7-predictor-cb9b4fb64-dz962\" (UID: \"da060526-210b-45d0-a402-d76141a73705\") " pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"
Jul 07 17:10:00.576462 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.576443    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da060526-210b-45d0-a402-d76141a73705-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-963f7-predictor-cb9b4fb64-dz962\" (UID: \"da060526-210b-45d0-a402-d76141a73705\") " pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"
Jul 07 17:10:00.576511 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.576464    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da060526-210b-45d0-a402-d76141a73705-cabundle-cert\") pod \"raw-sklearn-963f7-predictor-cb9b4fb64-dz962\" (UID: \"da060526-210b-45d0-a402-d76141a73705\") " pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"
Jul 07 17:10:00.578194 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.578173    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da060526-210b-45d0-a402-d76141a73705-proxy-tls\") pod \"raw-sklearn-963f7-predictor-cb9b4fb64-dz962\" (UID: \"da060526-210b-45d0-a402-d76141a73705\") " pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"
Jul 07 17:10:00.587697 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.587670    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l5kd\" (UniqueName: \"kubernetes.io/projected/da060526-210b-45d0-a402-d76141a73705-kube-api-access-8l5kd\") pod \"raw-sklearn-963f7-predictor-cb9b4fb64-dz962\" (UID: \"da060526-210b-45d0-a402-d76141a73705\") " pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"
Jul 07 17:10:00.735703 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.735673    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"
Jul 07 17:10:00.879325 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:00.879299    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"]
Jul 07 17:10:00.881149 ip-10-0-141-189 kubenswrapper[2583]: W0707 17:10:00.881109    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda060526_210b_45d0_a402_d76141a73705.slice/crio-bbc5bd827c64c901f78ac3c2ba24ac28ec95d1afda3e6df59158b35ab57b038a WatchSource:0}: Error finding container bbc5bd827c64c901f78ac3c2ba24ac28ec95d1afda3e6df59158b35ab57b038a: Status 404 returned error can't find the container with id bbc5bd827c64c901f78ac3c2ba24ac28ec95d1afda3e6df59158b35ab57b038a
Jul 07 17:10:01.835526 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:01.835487    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962" event={"ID":"da060526-210b-45d0-a402-d76141a73705","Type":"ContainerStarted","Data":"47538f82d8a7fb3648c623f8b6ff6bb0b56d695cae20f997c16297268bf66cc9"}
Jul 07 17:10:01.835955 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:01.835551    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962" event={"ID":"da060526-210b-45d0-a402-d76141a73705","Type":"ContainerStarted","Data":"bbc5bd827c64c901f78ac3c2ba24ac28ec95d1afda3e6df59158b35ab57b038a"}
Jul 07 17:10:04.849763 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:04.849671    2583 generic.go:356] "Generic (PLEG): container finished" podID="da060526-210b-45d0-a402-d76141a73705" containerID="47538f82d8a7fb3648c623f8b6ff6bb0b56d695cae20f997c16297268bf66cc9" exitCode=0
Jul 07 17:10:04.850235 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:04.849749    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962" event={"ID":"da060526-210b-45d0-a402-d76141a73705","Type":"ContainerDied","Data":"47538f82d8a7fb3648c623f8b6ff6bb0b56d695cae20f997c16297268bf66cc9"}
Jul 07 17:10:05.857617 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:05.857584    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962" event={"ID":"da060526-210b-45d0-a402-d76141a73705","Type":"ContainerStarted","Data":"8f4b1d536b646e6d96e3148cd111131e73149e36f5d50f0bac5062d05fbb8bac"}
Jul 07 17:10:05.858003 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:05.857649    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962" event={"ID":"da060526-210b-45d0-a402-d76141a73705","Type":"ContainerStarted","Data":"a0fda1a37bc86af9f89081b16626552b7dcaca502011657ca5fed8c66bc74c9e"}
Jul 07 17:10:05.858003 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:05.857863    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"
Jul 07 17:10:05.882207 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:05.882161    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962" podStartSLOduration=2.2869431860000002 podStartE2EDuration="5.882125625s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.595182439s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:10:05.881159683 +0000 UTC m=+1398.894169493" watchObservedRunningTime="2026-07-07 17:10:05.882125625 +0000 UTC m=+1398.895135444"
Jul 07 17:10:06.862988 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:06.862951    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"
Jul 07 17:10:06.864122 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:06.864094    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962" podUID="da060526-210b-45d0-a402-d76141a73705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 17:10:07.867016 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:07.866970    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962" podUID="da060526-210b-45d0-a402-d76141a73705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 17:10:12.871360 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:12.871329    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"
Jul 07 17:10:12.871866 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:12.871839    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962" podUID="da060526-210b-45d0-a402-d76141a73705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 17:10:22.872430 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:22.872384    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962" podUID="da060526-210b-45d0-a402-d76141a73705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 17:10:32.872220 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:32.872180    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962" podUID="da060526-210b-45d0-a402-d76141a73705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 17:10:42.872443 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:42.872359    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962" podUID="da060526-210b-45d0-a402-d76141a73705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 17:10:52.872114 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:10:52.872074    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962" podUID="da060526-210b-45d0-a402-d76141a73705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 17:11:02.872239 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:02.872186    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962" podUID="da060526-210b-45d0-a402-d76141a73705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 17:11:12.872377 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:12.872334    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962" podUID="da060526-210b-45d0-a402-d76141a73705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 17:11:22.873086 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:22.873056    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"
Jul 07 17:11:30.645095 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:30.645063    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"]
Jul 07 17:11:30.645616 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:30.645336    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962" podUID="da060526-210b-45d0-a402-d76141a73705" containerName="kserve-container" containerID="cri-o://a0fda1a37bc86af9f89081b16626552b7dcaca502011657ca5fed8c66bc74c9e" gracePeriod=30
Jul 07 17:11:30.645616 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:30.645384    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962" podUID="da060526-210b-45d0-a402-d76141a73705" containerName="kube-rbac-proxy" containerID="cri-o://8f4b1d536b646e6d96e3148cd111131e73149e36f5d50f0bac5062d05fbb8bac" gracePeriod=30
Jul 07 17:11:31.224276 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:31.224243    2583 generic.go:356] "Generic (PLEG): container finished" podID="da060526-210b-45d0-a402-d76141a73705" containerID="8f4b1d536b646e6d96e3148cd111131e73149e36f5d50f0bac5062d05fbb8bac" exitCode=2
Jul 07 17:11:31.224451 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:31.224306    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962" event={"ID":"da060526-210b-45d0-a402-d76141a73705","Type":"ContainerDied","Data":"8f4b1d536b646e6d96e3148cd111131e73149e36f5d50f0bac5062d05fbb8bac"}
Jul 07 17:11:32.867277 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:32.867236    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962" podUID="da060526-210b-45d0-a402-d76141a73705" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Jul 07 17:11:32.872601 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:32.872572    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962" podUID="da060526-210b-45d0-a402-d76141a73705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 17:11:35.494835 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.494814    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"
Jul 07 17:11:35.600354 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.600270    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da060526-210b-45d0-a402-d76141a73705-cabundle-cert\") pod \"da060526-210b-45d0-a402-d76141a73705\" (UID: \"da060526-210b-45d0-a402-d76141a73705\") "
Jul 07 17:11:35.600354 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.600344    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da060526-210b-45d0-a402-d76141a73705-proxy-tls\") pod \"da060526-210b-45d0-a402-d76141a73705\" (UID: \"da060526-210b-45d0-a402-d76141a73705\") "
Jul 07 17:11:35.600619 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.600397    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da060526-210b-45d0-a402-d76141a73705-kube-rbac-proxy-sar-config\") pod \"da060526-210b-45d0-a402-d76141a73705\" (UID: \"da060526-210b-45d0-a402-d76141a73705\") "
Jul 07 17:11:35.600619 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.600433    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8l5kd\" (UniqueName: \"kubernetes.io/projected/da060526-210b-45d0-a402-d76141a73705-kube-api-access-8l5kd\") pod \"da060526-210b-45d0-a402-d76141a73705\" (UID: \"da060526-210b-45d0-a402-d76141a73705\") "
Jul 07 17:11:35.600619 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.600466    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da060526-210b-45d0-a402-d76141a73705-kserve-provision-location\") pod \"da060526-210b-45d0-a402-d76141a73705\" (UID: \"da060526-210b-45d0-a402-d76141a73705\") "
Jul 07 17:11:35.600788 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.600687    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da060526-210b-45d0-a402-d76141a73705-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "da060526-210b-45d0-a402-d76141a73705" (UID: "da060526-210b-45d0-a402-d76141a73705"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:11:35.600788 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.600772    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da060526-210b-45d0-a402-d76141a73705-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "da060526-210b-45d0-a402-d76141a73705" (UID: "da060526-210b-45d0-a402-d76141a73705"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:11:35.600956 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.600930    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da060526-210b-45d0-a402-d76141a73705-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "da060526-210b-45d0-a402-d76141a73705" (UID: "da060526-210b-45d0-a402-d76141a73705"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:11:35.602657 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.602630    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da060526-210b-45d0-a402-d76141a73705-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "da060526-210b-45d0-a402-d76141a73705" (UID: "da060526-210b-45d0-a402-d76141a73705"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:11:35.602799 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.602731    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da060526-210b-45d0-a402-d76141a73705-kube-api-access-8l5kd" (OuterVolumeSpecName: "kube-api-access-8l5kd") pod "da060526-210b-45d0-a402-d76141a73705" (UID: "da060526-210b-45d0-a402-d76141a73705"). InnerVolumeSpecName "kube-api-access-8l5kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:11:35.701644 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.701614    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da060526-210b-45d0-a402-d76141a73705-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:11:35.701644 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.701642    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l5kd\" (UniqueName: \"kubernetes.io/projected/da060526-210b-45d0-a402-d76141a73705-kube-api-access-8l5kd\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:11:35.701844 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.701653    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da060526-210b-45d0-a402-d76141a73705-kserve-provision-location\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:11:35.701844 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.701663    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da060526-210b-45d0-a402-d76141a73705-cabundle-cert\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:11:35.701844 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.701672    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da060526-210b-45d0-a402-d76141a73705-proxy-tls\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:11:35.754977 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.754938    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"]
Jul 07 17:11:35.755382 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.755364    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da060526-210b-45d0-a402-d76141a73705" containerName="kube-rbac-proxy"
Jul 07 17:11:35.755486 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.755385    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="da060526-210b-45d0-a402-d76141a73705" containerName="kube-rbac-proxy"
Jul 07 17:11:35.755486 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.755400    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da060526-210b-45d0-a402-d76141a73705" containerName="kserve-container"
Jul 07 17:11:35.755486 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.755408    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="da060526-210b-45d0-a402-d76141a73705" containerName="kserve-container"
Jul 07 17:11:35.755486 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.755441    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da060526-210b-45d0-a402-d76141a73705" containerName="storage-initializer"
Jul 07 17:11:35.755486 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.755460    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="da060526-210b-45d0-a402-d76141a73705" containerName="storage-initializer"
Jul 07 17:11:35.755757 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.755584    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="da060526-210b-45d0-a402-d76141a73705" containerName="kube-rbac-proxy"
Jul 07 17:11:35.755757 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.755599    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="da060526-210b-45d0-a402-d76141a73705" containerName="kserve-container"
Jul 07 17:11:35.758970 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.758949    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"
Jul 07 17:11:35.761718 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.761691    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-7894c-predictor-serving-cert\""
Jul 07 17:11:35.761718 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.761696    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-7894c-kube-rbac-proxy-sar-config\""
Jul 07 17:11:35.770845 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.770822    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"]
Jul 07 17:11:35.802705 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.802677    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afe99c2e-92b8-4091-a205-c4d622bc743a-proxy-tls\") pod \"raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv\" (UID: \"afe99c2e-92b8-4091-a205-c4d622bc743a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"
Jul 07 17:11:35.802896 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.802714    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afe99c2e-92b8-4091-a205-c4d622bc743a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv\" (UID: \"afe99c2e-92b8-4091-a205-c4d622bc743a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"
Jul 07 17:11:35.802896 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.802838    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpn76\" (UniqueName: \"kubernetes.io/projected/afe99c2e-92b8-4091-a205-c4d622bc743a-kube-api-access-lpn76\") pod \"raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv\" (UID: \"afe99c2e-92b8-4091-a205-c4d622bc743a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"
Jul 07 17:11:35.802896 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.802871    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afe99c2e-92b8-4091-a205-c4d622bc743a-cabundle-cert\") pod \"raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv\" (UID: \"afe99c2e-92b8-4091-a205-c4d622bc743a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"
Jul 07 17:11:35.803031 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.802956    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afe99c2e-92b8-4091-a205-c4d622bc743a-kserve-provision-location\") pod \"raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv\" (UID: \"afe99c2e-92b8-4091-a205-c4d622bc743a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"
Jul 07 17:11:35.904281 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.904196    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afe99c2e-92b8-4091-a205-c4d622bc743a-kserve-provision-location\") pod \"raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv\" (UID: \"afe99c2e-92b8-4091-a205-c4d622bc743a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"
Jul 07 17:11:35.904281 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.904260    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afe99c2e-92b8-4091-a205-c4d622bc743a-proxy-tls\") pod \"raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv\" (UID: \"afe99c2e-92b8-4091-a205-c4d622bc743a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"
Jul 07 17:11:35.904281 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.904282    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afe99c2e-92b8-4091-a205-c4d622bc743a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv\" (UID: \"afe99c2e-92b8-4091-a205-c4d622bc743a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"
Jul 07 17:11:35.904539 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.904327    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpn76\" (UniqueName: \"kubernetes.io/projected/afe99c2e-92b8-4091-a205-c4d622bc743a-kube-api-access-lpn76\") pod \"raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv\" (UID: \"afe99c2e-92b8-4091-a205-c4d622bc743a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"
Jul 07 17:11:35.904539 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.904350    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afe99c2e-92b8-4091-a205-c4d622bc743a-cabundle-cert\") pod \"raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv\" (UID: \"afe99c2e-92b8-4091-a205-c4d622bc743a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"
Jul 07 17:11:35.904684 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.904659    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afe99c2e-92b8-4091-a205-c4d622bc743a-kserve-provision-location\") pod \"raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv\" (UID: \"afe99c2e-92b8-4091-a205-c4d622bc743a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"
Jul 07 17:11:35.905062 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.905039    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afe99c2e-92b8-4091-a205-c4d622bc743a-cabundle-cert\") pod \"raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv\" (UID: \"afe99c2e-92b8-4091-a205-c4d622bc743a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"
Jul 07 17:11:35.905102 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.905039    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afe99c2e-92b8-4091-a205-c4d622bc743a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv\" (UID: \"afe99c2e-92b8-4091-a205-c4d622bc743a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"
Jul 07 17:11:35.906830 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.906811    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afe99c2e-92b8-4091-a205-c4d622bc743a-proxy-tls\") pod \"raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv\" (UID: \"afe99c2e-92b8-4091-a205-c4d622bc743a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"
Jul 07 17:11:35.914602 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:35.914575    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpn76\" (UniqueName: \"kubernetes.io/projected/afe99c2e-92b8-4091-a205-c4d622bc743a-kube-api-access-lpn76\") pod \"raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv\" (UID: \"afe99c2e-92b8-4091-a205-c4d622bc743a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"
Jul 07 17:11:36.070411 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:36.070375    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"
Jul 07 17:11:36.204777 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:36.204737    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"]
Jul 07 17:11:36.206941 ip-10-0-141-189 kubenswrapper[2583]: W0707 17:11:36.206916    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe99c2e_92b8_4091_a205_c4d622bc743a.slice/crio-fc3e49035bf9da30f10b2b85d3eb360e56b3600281a90d3ac2dc1c909e179871 WatchSource:0}: Error finding container fc3e49035bf9da30f10b2b85d3eb360e56b3600281a90d3ac2dc1c909e179871: Status 404 returned error can't find the container with id fc3e49035bf9da30f10b2b85d3eb360e56b3600281a90d3ac2dc1c909e179871
Jul 07 17:11:36.246614 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:36.246586    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv" event={"ID":"afe99c2e-92b8-4091-a205-c4d622bc743a","Type":"ContainerStarted","Data":"fc3e49035bf9da30f10b2b85d3eb360e56b3600281a90d3ac2dc1c909e179871"}
Jul 07 17:11:36.248603 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:36.248574    2583 generic.go:356] "Generic (PLEG): container finished" podID="da060526-210b-45d0-a402-d76141a73705" containerID="a0fda1a37bc86af9f89081b16626552b7dcaca502011657ca5fed8c66bc74c9e" exitCode=0
Jul 07 17:11:36.248703 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:36.248686    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962" event={"ID":"da060526-210b-45d0-a402-d76141a73705","Type":"ContainerDied","Data":"a0fda1a37bc86af9f89081b16626552b7dcaca502011657ca5fed8c66bc74c9e"}
Jul 07 17:11:36.248763 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:36.248750    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962" event={"ID":"da060526-210b-45d0-a402-d76141a73705","Type":"ContainerDied","Data":"bbc5bd827c64c901f78ac3c2ba24ac28ec95d1afda3e6df59158b35ab57b038a"}
Jul 07 17:11:36.248801 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:36.248689    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"
Jul 07 17:11:36.248801 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:36.248792    2583 scope.go:117] "RemoveContainer" containerID="8f4b1d536b646e6d96e3148cd111131e73149e36f5d50f0bac5062d05fbb8bac"
Jul 07 17:11:36.258595 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:36.258573    2583 scope.go:117] "RemoveContainer" containerID="a0fda1a37bc86af9f89081b16626552b7dcaca502011657ca5fed8c66bc74c9e"
Jul 07 17:11:36.266782 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:36.266758    2583 scope.go:117] "RemoveContainer" containerID="47538f82d8a7fb3648c623f8b6ff6bb0b56d695cae20f997c16297268bf66cc9"
Jul 07 17:11:36.273418 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:36.273396    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"]
Jul 07 17:11:36.278519 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:36.278493    2583 scope.go:117] "RemoveContainer" containerID="8f4b1d536b646e6d96e3148cd111131e73149e36f5d50f0bac5062d05fbb8bac"
Jul 07 17:11:36.278824 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:11:36.278804    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f4b1d536b646e6d96e3148cd111131e73149e36f5d50f0bac5062d05fbb8bac\": container with ID starting with 8f4b1d536b646e6d96e3148cd111131e73149e36f5d50f0bac5062d05fbb8bac not found: ID does not exist" containerID="8f4b1d536b646e6d96e3148cd111131e73149e36f5d50f0bac5062d05fbb8bac"
Jul 07 17:11:36.278905 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:36.278834    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f4b1d536b646e6d96e3148cd111131e73149e36f5d50f0bac5062d05fbb8bac"} err="failed to get container status \"8f4b1d536b646e6d96e3148cd111131e73149e36f5d50f0bac5062d05fbb8bac\": rpc error: code = NotFound desc = could not find container \"8f4b1d536b646e6d96e3148cd111131e73149e36f5d50f0bac5062d05fbb8bac\": container with ID starting with 8f4b1d536b646e6d96e3148cd111131e73149e36f5d50f0bac5062d05fbb8bac not found: ID does not exist"
Jul 07 17:11:36.278905 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:36.278853    2583 scope.go:117] "RemoveContainer" containerID="a0fda1a37bc86af9f89081b16626552b7dcaca502011657ca5fed8c66bc74c9e"
Jul 07 17:11:36.279107 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:11:36.279083    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0fda1a37bc86af9f89081b16626552b7dcaca502011657ca5fed8c66bc74c9e\": container with ID starting with a0fda1a37bc86af9f89081b16626552b7dcaca502011657ca5fed8c66bc74c9e not found: ID does not exist" containerID="a0fda1a37bc86af9f89081b16626552b7dcaca502011657ca5fed8c66bc74c9e"
Jul 07 17:11:36.279204 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:36.279117    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0fda1a37bc86af9f89081b16626552b7dcaca502011657ca5fed8c66bc74c9e"} err="failed to get container status \"a0fda1a37bc86af9f89081b16626552b7dcaca502011657ca5fed8c66bc74c9e\": rpc error: code = NotFound desc = could not find container \"a0fda1a37bc86af9f89081b16626552b7dcaca502011657ca5fed8c66bc74c9e\": container with ID starting with a0fda1a37bc86af9f89081b16626552b7dcaca502011657ca5fed8c66bc74c9e not found: ID does not exist"
Jul 07 17:11:36.279204 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:36.279155    2583 scope.go:117] "RemoveContainer" containerID="47538f82d8a7fb3648c623f8b6ff6bb0b56d695cae20f997c16297268bf66cc9"
Jul 07 17:11:36.279419 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:11:36.279402    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47538f82d8a7fb3648c623f8b6ff6bb0b56d695cae20f997c16297268bf66cc9\": container with ID starting with 47538f82d8a7fb3648c623f8b6ff6bb0b56d695cae20f997c16297268bf66cc9 not found: ID does not exist" containerID="47538f82d8a7fb3648c623f8b6ff6bb0b56d695cae20f997c16297268bf66cc9"
Jul 07 17:11:36.279491 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:36.279425    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47538f82d8a7fb3648c623f8b6ff6bb0b56d695cae20f997c16297268bf66cc9"} err="failed to get container status \"47538f82d8a7fb3648c623f8b6ff6bb0b56d695cae20f997c16297268bf66cc9\": rpc error: code = NotFound desc = could not find container \"47538f82d8a7fb3648c623f8b6ff6bb0b56d695cae20f997c16297268bf66cc9\": container with ID starting with 47538f82d8a7fb3648c623f8b6ff6bb0b56d695cae20f997c16297268bf66cc9 not found: ID does not exist"
Jul 07 17:11:36.279551 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:36.279512    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-963f7-predictor-cb9b4fb64-dz962"]
Jul 07 17:11:37.254206 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:37.254164    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv" event={"ID":"afe99c2e-92b8-4091-a205-c4d622bc743a","Type":"ContainerStarted","Data":"4ae67a3deabcbc2ff65c4a74dde4a194ebfa54b24814b5e9109a151566153698"}
Jul 07 17:11:37.586404 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:37.586324    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da060526-210b-45d0-a402-d76141a73705" path="/var/lib/kubelet/pods/da060526-210b-45d0-a402-d76141a73705/volumes"
Jul 07 17:11:40.268406 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:40.268375    2583 generic.go:356] "Generic (PLEG): container finished" podID="afe99c2e-92b8-4091-a205-c4d622bc743a" containerID="4ae67a3deabcbc2ff65c4a74dde4a194ebfa54b24814b5e9109a151566153698" exitCode=0
Jul 07 17:11:40.268802 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:40.268432    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv" event={"ID":"afe99c2e-92b8-4091-a205-c4d622bc743a","Type":"ContainerDied","Data":"4ae67a3deabcbc2ff65c4a74dde4a194ebfa54b24814b5e9109a151566153698"}
Jul 07 17:11:41.274761 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:41.274728    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv" event={"ID":"afe99c2e-92b8-4091-a205-c4d622bc743a","Type":"ContainerStarted","Data":"72832331159a4accb56e95eb5fa33a9f79a4742c66e9be2da3e6a6e69fb3b347"}
Jul 07 17:11:41.275169 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:41.274790    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv" event={"ID":"afe99c2e-92b8-4091-a205-c4d622bc743a","Type":"ContainerStarted","Data":"867e9ed89ad2ce3680b8dc38041454a54649177d90377e81412782994f72e0b6"}
Jul 07 17:11:41.275169 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:41.274973    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"
Jul 07 17:11:41.299406 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:41.299358    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv" podStartSLOduration=2.695166094 podStartE2EDuration="6.299343441s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.604177347s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:11:41.297805643 +0000 UTC m=+1494.310815464" watchObservedRunningTime="2026-07-07 17:11:41.299343441 +0000 UTC m=+1494.312353314"
Jul 07 17:11:42.278802 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:42.278764    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"
Jul 07 17:11:42.280130 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:42.280099    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv" podUID="afe99c2e-92b8-4091-a205-c4d622bc743a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 07 17:11:43.282492 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:43.282447    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv" podUID="afe99c2e-92b8-4091-a205-c4d622bc743a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 07 17:11:48.286685 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:48.286655    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"
Jul 07 17:11:48.287221 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:48.287193    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv" podUID="afe99c2e-92b8-4091-a205-c4d622bc743a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 07 17:11:58.287548 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:11:58.287508    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv" podUID="afe99c2e-92b8-4091-a205-c4d622bc743a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 07 17:12:08.287593 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:12:08.287503    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv" podUID="afe99c2e-92b8-4091-a205-c4d622bc743a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 07 17:12:18.287608 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:12:18.287565    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv" podUID="afe99c2e-92b8-4091-a205-c4d622bc743a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 07 17:12:28.287158 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:12:28.287106    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv" podUID="afe99c2e-92b8-4091-a205-c4d622bc743a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 07 17:12:38.287663 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:12:38.287621    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv" podUID="afe99c2e-92b8-4091-a205-c4d622bc743a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 07 17:12:48.288065 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:12:48.288019    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv" podUID="afe99c2e-92b8-4091-a205-c4d622bc743a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 07 17:12:58.288232 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:12:58.288203    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"
Jul 07 17:13:05.925311 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:05.925277    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"]
Jul 07 17:13:05.925800 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:05.925562    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv" podUID="afe99c2e-92b8-4091-a205-c4d622bc743a" containerName="kserve-container" containerID="cri-o://867e9ed89ad2ce3680b8dc38041454a54649177d90377e81412782994f72e0b6" gracePeriod=30
Jul 07 17:13:05.925800 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:05.925568    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv" podUID="afe99c2e-92b8-4091-a205-c4d622bc743a" containerName="kube-rbac-proxy" containerID="cri-o://72832331159a4accb56e95eb5fa33a9f79a4742c66e9be2da3e6a6e69fb3b347" gracePeriod=30
Jul 07 17:13:06.634037 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:06.634002    2583 generic.go:356] "Generic (PLEG): container finished" podID="afe99c2e-92b8-4091-a205-c4d622bc743a" containerID="72832331159a4accb56e95eb5fa33a9f79a4742c66e9be2da3e6a6e69fb3b347" exitCode=2
Jul 07 17:13:06.634248 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:06.634074    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv" event={"ID":"afe99c2e-92b8-4091-a205-c4d622bc743a","Type":"ContainerDied","Data":"72832331159a4accb56e95eb5fa33a9f79a4742c66e9be2da3e6a6e69fb3b347"}
Jul 07 17:13:08.282766 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:08.282718    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv" podUID="afe99c2e-92b8-4091-a205-c4d622bc743a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 07 17:13:08.288093 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:08.288062    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv" podUID="afe99c2e-92b8-4091-a205-c4d622bc743a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 07 17:13:10.653555 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:10.653528    2583 generic.go:356] "Generic (PLEG): container finished" podID="afe99c2e-92b8-4091-a205-c4d622bc743a" containerID="867e9ed89ad2ce3680b8dc38041454a54649177d90377e81412782994f72e0b6" exitCode=0
Jul 07 17:13:10.653835 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:10.653582    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv" event={"ID":"afe99c2e-92b8-4091-a205-c4d622bc743a","Type":"ContainerDied","Data":"867e9ed89ad2ce3680b8dc38041454a54649177d90377e81412782994f72e0b6"}
Jul 07 17:13:10.784437 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:10.784413    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"
Jul 07 17:13:10.857591 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:10.857561    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afe99c2e-92b8-4091-a205-c4d622bc743a-proxy-tls\") pod \"afe99c2e-92b8-4091-a205-c4d622bc743a\" (UID: \"afe99c2e-92b8-4091-a205-c4d622bc743a\") "
Jul 07 17:13:10.857782 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:10.857653    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afe99c2e-92b8-4091-a205-c4d622bc743a-kserve-provision-location\") pod \"afe99c2e-92b8-4091-a205-c4d622bc743a\" (UID: \"afe99c2e-92b8-4091-a205-c4d622bc743a\") "
Jul 07 17:13:10.857782 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:10.857679    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpn76\" (UniqueName: \"kubernetes.io/projected/afe99c2e-92b8-4091-a205-c4d622bc743a-kube-api-access-lpn76\") pod \"afe99c2e-92b8-4091-a205-c4d622bc743a\" (UID: \"afe99c2e-92b8-4091-a205-c4d622bc743a\") "
Jul 07 17:13:10.857782 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:10.857695    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afe99c2e-92b8-4091-a205-c4d622bc743a-cabundle-cert\") pod \"afe99c2e-92b8-4091-a205-c4d622bc743a\" (UID: \"afe99c2e-92b8-4091-a205-c4d622bc743a\") "
Jul 07 17:13:10.857782 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:10.857724    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afe99c2e-92b8-4091-a205-c4d622bc743a-kube-rbac-proxy-sar-config\") pod \"afe99c2e-92b8-4091-a205-c4d622bc743a\" (UID: \"afe99c2e-92b8-4091-a205-c4d622bc743a\") "
Jul 07 17:13:10.858067 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:10.858043    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe99c2e-92b8-4091-a205-c4d622bc743a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "afe99c2e-92b8-4091-a205-c4d622bc743a" (UID: "afe99c2e-92b8-4091-a205-c4d622bc743a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:13:10.858158 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:10.858059    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe99c2e-92b8-4091-a205-c4d622bc743a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "afe99c2e-92b8-4091-a205-c4d622bc743a" (UID: "afe99c2e-92b8-4091-a205-c4d622bc743a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:13:10.858202 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:10.858166    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe99c2e-92b8-4091-a205-c4d622bc743a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "afe99c2e-92b8-4091-a205-c4d622bc743a" (UID: "afe99c2e-92b8-4091-a205-c4d622bc743a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:13:10.859902 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:10.859883    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afe99c2e-92b8-4091-a205-c4d622bc743a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "afe99c2e-92b8-4091-a205-c4d622bc743a" (UID: "afe99c2e-92b8-4091-a205-c4d622bc743a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:13:10.859902 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:10.859890    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe99c2e-92b8-4091-a205-c4d622bc743a-kube-api-access-lpn76" (OuterVolumeSpecName: "kube-api-access-lpn76") pod "afe99c2e-92b8-4091-a205-c4d622bc743a" (UID: "afe99c2e-92b8-4091-a205-c4d622bc743a"). InnerVolumeSpecName "kube-api-access-lpn76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:13:10.958739 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:10.958709    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afe99c2e-92b8-4091-a205-c4d622bc743a-kserve-provision-location\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:13:10.958739 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:10.958732    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpn76\" (UniqueName: \"kubernetes.io/projected/afe99c2e-92b8-4091-a205-c4d622bc743a-kube-api-access-lpn76\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:13:10.958739 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:10.958742    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afe99c2e-92b8-4091-a205-c4d622bc743a-cabundle-cert\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:13:10.958972 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:10.958750    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afe99c2e-92b8-4091-a205-c4d622bc743a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:13:10.958972 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:10.958760    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afe99c2e-92b8-4091-a205-c4d622bc743a-proxy-tls\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:13:11.659091 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:11.659055    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv" event={"ID":"afe99c2e-92b8-4091-a205-c4d622bc743a","Type":"ContainerDied","Data":"fc3e49035bf9da30f10b2b85d3eb360e56b3600281a90d3ac2dc1c909e179871"}
Jul 07 17:13:11.659091 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:11.659092    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"
Jul 07 17:13:11.659620 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:11.659151    2583 scope.go:117] "RemoveContainer" containerID="72832331159a4accb56e95eb5fa33a9f79a4742c66e9be2da3e6a6e69fb3b347"
Jul 07 17:13:11.667923 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:11.667906    2583 scope.go:117] "RemoveContainer" containerID="867e9ed89ad2ce3680b8dc38041454a54649177d90377e81412782994f72e0b6"
Jul 07 17:13:11.675791 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:11.675770    2583 scope.go:117] "RemoveContainer" containerID="4ae67a3deabcbc2ff65c4a74dde4a194ebfa54b24814b5e9109a151566153698"
Jul 07 17:13:11.682388 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:11.682363    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"]
Jul 07 17:13:11.688436 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:11.688413    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-7894c-predictor-67f77f8dd-xq9vv"]
Jul 07 17:13:12.357803 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:12.357769    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r4mdr/must-gather-6nn4m"]
Jul 07 17:13:12.358164 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:12.358151    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afe99c2e-92b8-4091-a205-c4d622bc743a" containerName="kube-rbac-proxy"
Jul 07 17:13:12.358208 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:12.358167    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="afe99c2e-92b8-4091-a205-c4d622bc743a" containerName="kube-rbac-proxy"
Jul 07 17:13:12.358208 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:12.358182    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afe99c2e-92b8-4091-a205-c4d622bc743a" containerName="kserve-container"
Jul 07 17:13:12.358208 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:12.358191    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="afe99c2e-92b8-4091-a205-c4d622bc743a" containerName="kserve-container"
Jul 07 17:13:12.358208 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:12.358202    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afe99c2e-92b8-4091-a205-c4d622bc743a" containerName="storage-initializer"
Jul 07 17:13:12.358208 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:12.358207    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="afe99c2e-92b8-4091-a205-c4d622bc743a" containerName="storage-initializer"
Jul 07 17:13:12.358366 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:12.358268    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="afe99c2e-92b8-4091-a205-c4d622bc743a" containerName="kserve-container"
Jul 07 17:13:12.358366 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:12.358277    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="afe99c2e-92b8-4091-a205-c4d622bc743a" containerName="kube-rbac-proxy"
Jul 07 17:13:12.362816 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:12.362796    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r4mdr/must-gather-6nn4m"
Jul 07 17:13:12.365731 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:12.365708    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r4mdr\"/\"openshift-service-ca.crt\""
Jul 07 17:13:12.366723 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:12.366708    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r4mdr\"/\"kube-root-ca.crt\""
Jul 07 17:13:12.366825 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:12.366743    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-r4mdr\"/\"default-dockercfg-2f87k\""
Jul 07 17:13:12.376790 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:12.376760    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r4mdr/must-gather-6nn4m"]
Jul 07 17:13:12.473488 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:12.473440    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8rr4\" (UniqueName: \"kubernetes.io/projected/bb392c81-3e44-4f5f-826f-95a6a4032f71-kube-api-access-g8rr4\") pod \"must-gather-6nn4m\" (UID: \"bb392c81-3e44-4f5f-826f-95a6a4032f71\") " pod="openshift-must-gather-r4mdr/must-gather-6nn4m"
Jul 07 17:13:12.473674 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:12.473502    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bb392c81-3e44-4f5f-826f-95a6a4032f71-must-gather-output\") pod \"must-gather-6nn4m\" (UID: \"bb392c81-3e44-4f5f-826f-95a6a4032f71\") " pod="openshift-must-gather-r4mdr/must-gather-6nn4m"
Jul 07 17:13:12.574766 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:12.574713    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8rr4\" (UniqueName: \"kubernetes.io/projected/bb392c81-3e44-4f5f-826f-95a6a4032f71-kube-api-access-g8rr4\") pod \"must-gather-6nn4m\" (UID: \"bb392c81-3e44-4f5f-826f-95a6a4032f71\") " pod="openshift-must-gather-r4mdr/must-gather-6nn4m"
Jul 07 17:13:12.574920 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:12.574784    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bb392c81-3e44-4f5f-826f-95a6a4032f71-must-gather-output\") pod \"must-gather-6nn4m\" (UID: \"bb392c81-3e44-4f5f-826f-95a6a4032f71\") " pod="openshift-must-gather-r4mdr/must-gather-6nn4m"
Jul 07 17:13:12.575087 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:12.575067    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bb392c81-3e44-4f5f-826f-95a6a4032f71-must-gather-output\") pod \"must-gather-6nn4m\" (UID: \"bb392c81-3e44-4f5f-826f-95a6a4032f71\") " pod="openshift-must-gather-r4mdr/must-gather-6nn4m"
Jul 07 17:13:12.586636 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:12.586606    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8rr4\" (UniqueName: \"kubernetes.io/projected/bb392c81-3e44-4f5f-826f-95a6a4032f71-kube-api-access-g8rr4\") pod \"must-gather-6nn4m\" (UID: \"bb392c81-3e44-4f5f-826f-95a6a4032f71\") " pod="openshift-must-gather-r4mdr/must-gather-6nn4m"
Jul 07 17:13:12.680567 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:12.680488    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r4mdr/must-gather-6nn4m"
Jul 07 17:13:12.814226 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:12.814199    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r4mdr/must-gather-6nn4m"]
Jul 07 17:13:12.816326 ip-10-0-141-189 kubenswrapper[2583]: W0707 17:13:12.816292    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb392c81_3e44_4f5f_826f_95a6a4032f71.slice/crio-b879ec11cb376a0fde26839ffcb5ff3cef4dc81be9b2daaa8f3aaf767a2ff515 WatchSource:0}: Error finding container b879ec11cb376a0fde26839ffcb5ff3cef4dc81be9b2daaa8f3aaf767a2ff515: Status 404 returned error can't find the container with id b879ec11cb376a0fde26839ffcb5ff3cef4dc81be9b2daaa8f3aaf767a2ff515
Jul 07 17:13:12.818106 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:12.818087    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 17:13:13.590963 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:13.590375    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afe99c2e-92b8-4091-a205-c4d622bc743a" path="/var/lib/kubelet/pods/afe99c2e-92b8-4091-a205-c4d622bc743a/volumes"
Jul 07 17:13:13.671446 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:13.671383    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r4mdr/must-gather-6nn4m" event={"ID":"bb392c81-3e44-4f5f-826f-95a6a4032f71","Type":"ContainerStarted","Data":"b879ec11cb376a0fde26839ffcb5ff3cef4dc81be9b2daaa8f3aaf767a2ff515"}
Jul 07 17:13:19.705302 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:19.705253    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r4mdr/must-gather-6nn4m" event={"ID":"bb392c81-3e44-4f5f-826f-95a6a4032f71","Type":"ContainerStarted","Data":"eb14434ec6a79dc17a64df25efe6a2d2dea00f4483ce58b8eb4d46ed63555908"}
Jul 07 17:13:19.705705 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:19.705322    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r4mdr/must-gather-6nn4m" event={"ID":"bb392c81-3e44-4f5f-826f-95a6a4032f71","Type":"ContainerStarted","Data":"adb9300f64c40f47f1e6a636b0641810e7da81a840d4f68c7320777754a1847c"}
Jul 07 17:13:19.726059 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:19.725997    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-r4mdr/must-gather-6nn4m" podStartSLOduration=1.227070358 podStartE2EDuration="7.725973871s" totalImagesPullingTime="6.498903513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:13:19.724382652 +0000 UTC m=+1592.737392471" watchObservedRunningTime="2026-07-07 17:13:19.725973871 +0000 UTC m=+1592.738983693"
Jul 07 17:13:40.805152 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:40.805038    2583 generic.go:356] "Generic (PLEG): container finished" podID="bb392c81-3e44-4f5f-826f-95a6a4032f71" containerID="adb9300f64c40f47f1e6a636b0641810e7da81a840d4f68c7320777754a1847c" exitCode=0
Jul 07 17:13:40.805152 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:40.805087    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r4mdr/must-gather-6nn4m" event={"ID":"bb392c81-3e44-4f5f-826f-95a6a4032f71","Type":"ContainerDied","Data":"adb9300f64c40f47f1e6a636b0641810e7da81a840d4f68c7320777754a1847c"}
Jul 07 17:13:40.805583 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:40.805432    2583 scope.go:117] "RemoveContainer" containerID="adb9300f64c40f47f1e6a636b0641810e7da81a840d4f68c7320777754a1847c"
Jul 07 17:13:41.581006 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:41.580977    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-r4mdr_must-gather-6nn4m_bb392c81-3e44-4f5f-826f-95a6a4032f71/gather/0.log"
Jul 07 17:13:45.556852 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:45.556819    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x5chr_41c32952-21a9-4e19-bc8a-ceee6a08ceeb/global-pull-secret-syncer/0.log"
Jul 07 17:13:45.707402 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:45.707370    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kprb9_ae6f29a9-b432-4a71-a94a-27710a455ea6/konnectivity-agent/0.log"
Jul 07 17:13:45.806695 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:45.806665    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-189.ec2.internal_e3a5ffb0c2c44b15a7860c5703342945/haproxy/0.log"
Jul 07 17:13:47.141730 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:47.141690    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-r4mdr/must-gather-6nn4m"]
Jul 07 17:13:47.142194 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:47.141895    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-r4mdr/must-gather-6nn4m" podUID="bb392c81-3e44-4f5f-826f-95a6a4032f71" containerName="copy" containerID="cri-o://eb14434ec6a79dc17a64df25efe6a2d2dea00f4483ce58b8eb4d46ed63555908" gracePeriod=2
Jul 07 17:13:47.151594 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:47.151563    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-r4mdr/must-gather-6nn4m"]
Jul 07 17:13:47.375276 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:47.375250    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-r4mdr_must-gather-6nn4m_bb392c81-3e44-4f5f-826f-95a6a4032f71/copy/0.log"
Jul 07 17:13:47.376896 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:47.376879    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r4mdr/must-gather-6nn4m"
Jul 07 17:13:47.379337 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:13:47.379310    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-6nn4m\" is forbidden: User \"system:node:ip-10-0-141-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-r4mdr\": no relationship found between node 'ip-10-0-141-189.ec2.internal' and this object" podUID="bb392c81-3e44-4f5f-826f-95a6a4032f71" pod="openshift-must-gather-r4mdr/must-gather-6nn4m"
Jul 07 17:13:47.397715 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:47.397637    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8rr4\" (UniqueName: \"kubernetes.io/projected/bb392c81-3e44-4f5f-826f-95a6a4032f71-kube-api-access-g8rr4\") pod \"bb392c81-3e44-4f5f-826f-95a6a4032f71\" (UID: \"bb392c81-3e44-4f5f-826f-95a6a4032f71\") "
Jul 07 17:13:47.397715 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:47.397693    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bb392c81-3e44-4f5f-826f-95a6a4032f71-must-gather-output\") pod \"bb392c81-3e44-4f5f-826f-95a6a4032f71\" (UID: \"bb392c81-3e44-4f5f-826f-95a6a4032f71\") "
Jul 07 17:13:47.399707 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:47.399666    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb392c81-3e44-4f5f-826f-95a6a4032f71-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "bb392c81-3e44-4f5f-826f-95a6a4032f71" (UID: "bb392c81-3e44-4f5f-826f-95a6a4032f71"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:13:47.399928 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:47.399902    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb392c81-3e44-4f5f-826f-95a6a4032f71-kube-api-access-g8rr4" (OuterVolumeSpecName: "kube-api-access-g8rr4") pod "bb392c81-3e44-4f5f-826f-95a6a4032f71" (UID: "bb392c81-3e44-4f5f-826f-95a6a4032f71"). InnerVolumeSpecName "kube-api-access-g8rr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:13:47.499215 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:47.499180    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8rr4\" (UniqueName: \"kubernetes.io/projected/bb392c81-3e44-4f5f-826f-95a6a4032f71-kube-api-access-g8rr4\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:13:47.499215 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:47.499211    2583 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bb392c81-3e44-4f5f-826f-95a6a4032f71-must-gather-output\") on node \"ip-10-0-141-189.ec2.internal\" DevicePath \"\""
Jul 07 17:13:47.585891 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:13:47.585854    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-6nn4m\" is forbidden: User \"system:node:ip-10-0-141-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-r4mdr\": no relationship found between node 'ip-10-0-141-189.ec2.internal' and this object" podUID="bb392c81-3e44-4f5f-826f-95a6a4032f71" pod="openshift-must-gather-r4mdr/must-gather-6nn4m"
Jul 07 17:13:47.587085 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:47.587056    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb392c81-3e44-4f5f-826f-95a6a4032f71" path="/var/lib/kubelet/pods/bb392c81-3e44-4f5f-826f-95a6a4032f71/volumes"
Jul 07 17:13:47.840426 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:47.840394    2583 generic.go:356] "Generic (PLEG): container finished" podID="bb392c81-3e44-4f5f-826f-95a6a4032f71" containerID="eb14434ec6a79dc17a64df25efe6a2d2dea00f4483ce58b8eb4d46ed63555908" exitCode=143
Jul 07 17:13:47.840600 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:47.840439    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r4mdr/must-gather-6nn4m"
Jul 07 17:13:47.840600 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:47.840476    2583 scope.go:117] "RemoveContainer" containerID="eb14434ec6a79dc17a64df25efe6a2d2dea00f4483ce58b8eb4d46ed63555908"
Jul 07 17:13:47.848148 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:47.848114    2583 scope.go:117] "RemoveContainer" containerID="adb9300f64c40f47f1e6a636b0641810e7da81a840d4f68c7320777754a1847c"
Jul 07 17:13:47.860427 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:47.860408    2583 scope.go:117] "RemoveContainer" containerID="eb14434ec6a79dc17a64df25efe6a2d2dea00f4483ce58b8eb4d46ed63555908"
Jul 07 17:13:47.860688 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:13:47.860669    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb14434ec6a79dc17a64df25efe6a2d2dea00f4483ce58b8eb4d46ed63555908\": container with ID starting with eb14434ec6a79dc17a64df25efe6a2d2dea00f4483ce58b8eb4d46ed63555908 not found: ID does not exist" containerID="eb14434ec6a79dc17a64df25efe6a2d2dea00f4483ce58b8eb4d46ed63555908"
Jul 07 17:13:47.860751 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:47.860697    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb14434ec6a79dc17a64df25efe6a2d2dea00f4483ce58b8eb4d46ed63555908"} err="failed to get container status \"eb14434ec6a79dc17a64df25efe6a2d2dea00f4483ce58b8eb4d46ed63555908\": rpc error: code = NotFound desc = could not find container \"eb14434ec6a79dc17a64df25efe6a2d2dea00f4483ce58b8eb4d46ed63555908\": container with ID starting with eb14434ec6a79dc17a64df25efe6a2d2dea00f4483ce58b8eb4d46ed63555908 not found: ID does not exist"
Jul 07 17:13:47.860751 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:47.860718    2583 scope.go:117] "RemoveContainer" containerID="adb9300f64c40f47f1e6a636b0641810e7da81a840d4f68c7320777754a1847c"
Jul 07 17:13:47.860956 ip-10-0-141-189 kubenswrapper[2583]: E0707 17:13:47.860939    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adb9300f64c40f47f1e6a636b0641810e7da81a840d4f68c7320777754a1847c\": container with ID starting with adb9300f64c40f47f1e6a636b0641810e7da81a840d4f68c7320777754a1847c not found: ID does not exist" containerID="adb9300f64c40f47f1e6a636b0641810e7da81a840d4f68c7320777754a1847c"
Jul 07 17:13:47.861009 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:47.860963    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb9300f64c40f47f1e6a636b0641810e7da81a840d4f68c7320777754a1847c"} err="failed to get container status \"adb9300f64c40f47f1e6a636b0641810e7da81a840d4f68c7320777754a1847c\": rpc error: code = NotFound desc = could not find container \"adb9300f64c40f47f1e6a636b0641810e7da81a840d4f68c7320777754a1847c\": container with ID starting with adb9300f64c40f47f1e6a636b0641810e7da81a840d4f68c7320777754a1847c not found: ID does not exist"
Jul 07 17:13:49.018984 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:49.018959    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_118fa8c4-16ad-4526-88dc-57746af1a633/alertmanager/0.log"
Jul 07 17:13:49.053952 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:49.053928    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_118fa8c4-16ad-4526-88dc-57746af1a633/config-reloader/0.log"
Jul 07 17:13:49.140778 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:49.140751    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_118fa8c4-16ad-4526-88dc-57746af1a633/kube-rbac-proxy-web/0.log"
Jul 07 17:13:49.182687 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:49.182660    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_118fa8c4-16ad-4526-88dc-57746af1a633/kube-rbac-proxy/0.log"
Jul 07 17:13:49.229829 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:49.229788    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_118fa8c4-16ad-4526-88dc-57746af1a633/kube-rbac-proxy-metric/0.log"
Jul 07 17:13:49.266764 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:49.266736    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_118fa8c4-16ad-4526-88dc-57746af1a633/prom-label-proxy/0.log"
Jul 07 17:13:49.305732 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:49.305656    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_118fa8c4-16ad-4526-88dc-57746af1a633/init-config-reloader/0.log"
Jul 07 17:13:49.401758 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:49.401720    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-j4rm8_799dd927-af40-4e34-91a8-afabf2916b3c/kube-state-metrics/0.log"
Jul 07 17:13:49.433403 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:49.433377    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-j4rm8_799dd927-af40-4e34-91a8-afabf2916b3c/kube-rbac-proxy-main/0.log"
Jul 07 17:13:49.469377 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:49.469340    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-j4rm8_799dd927-af40-4e34-91a8-afabf2916b3c/kube-rbac-proxy-self/0.log"
Jul 07 17:13:49.508342 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:49.508314    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-78665f68c4-cj9jt_d86c8264-ee31-4fd3-a988-79bc42548e51/metrics-server/0.log"
Jul 07 17:13:49.551658 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:49.551622    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-5kgbd_2f3a1ce1-eb1e-43e3-a0e0-a7f28c63e227/monitoring-plugin/0.log"
Jul 07 17:13:49.594384 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:49.594314    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dthp7_98cd7ce4-4275-4530-8b3d-94cd6ee8f305/node-exporter/0.log"
Jul 07 17:13:49.622921 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:49.622895    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dthp7_98cd7ce4-4275-4530-8b3d-94cd6ee8f305/kube-rbac-proxy/0.log"
Jul 07 17:13:49.656798 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:49.656774    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dthp7_98cd7ce4-4275-4530-8b3d-94cd6ee8f305/init-textfile/0.log"
Jul 07 17:13:49.904279 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:49.904196    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-j4j5l_ef657466-f5a1-445b-8a06-b950868b9843/kube-rbac-proxy-main/0.log"
Jul 07 17:13:49.936827 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:49.936797    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-j4j5l_ef657466-f5a1-445b-8a06-b950868b9843/kube-rbac-proxy-self/0.log"
Jul 07 17:13:49.970983 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:49.970957    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-j4j5l_ef657466-f5a1-445b-8a06-b950868b9843/openshift-state-metrics/0.log"
Jul 07 17:13:50.028046 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:50.022454    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a237ad78-f074-4073-8732-7f7e4f8e8477/prometheus/0.log"
Jul 07 17:13:50.054195 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:50.054163    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a237ad78-f074-4073-8732-7f7e4f8e8477/config-reloader/0.log"
Jul 07 17:13:50.085837 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:50.085814    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a237ad78-f074-4073-8732-7f7e4f8e8477/thanos-sidecar/0.log"
Jul 07 17:13:50.118254 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:50.118224    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a237ad78-f074-4073-8732-7f7e4f8e8477/kube-rbac-proxy-web/0.log"
Jul 07 17:13:50.151524 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:50.151492    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a237ad78-f074-4073-8732-7f7e4f8e8477/kube-rbac-proxy/0.log"
Jul 07 17:13:50.184737 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:50.184659    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a237ad78-f074-4073-8732-7f7e4f8e8477/kube-rbac-proxy-thanos/0.log"
Jul 07 17:13:50.215189 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:50.215157    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a237ad78-f074-4073-8732-7f7e4f8e8477/init-config-reloader/0.log"
Jul 07 17:13:50.254300 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:50.254273    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-ctshv_f33e5bc9-45a5-4351-b72d-34c1c3c34a81/prometheus-operator/0.log"
Jul 07 17:13:50.287858 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:50.287825    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-ctshv_f33e5bc9-45a5-4351-b72d-34c1c3c34a81/kube-rbac-proxy/0.log"
Jul 07 17:13:50.318325 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:50.318301    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-bvlj2_e39c3317-fd9e-44b3-8eaf-242513bbf517/prometheus-operator-admission-webhook/0.log"
Jul 07 17:13:50.353691 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:50.353666    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f766df6b9-f899l_d2201672-aa8c-4b3b-9c55-49ea72d212db/telemeter-client/0.log"
Jul 07 17:13:50.388091 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:50.388065    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f766df6b9-f899l_d2201672-aa8c-4b3b-9c55-49ea72d212db/reload/0.log"
Jul 07 17:13:50.424776 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:50.424750    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f766df6b9-f899l_d2201672-aa8c-4b3b-9c55-49ea72d212db/kube-rbac-proxy/0.log"
Jul 07 17:13:50.468601 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:50.468573    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bb46bd4d-s5qzv_63a50a99-77c3-477d-9e21-cd1066fb698d/thanos-query/0.log"
Jul 07 17:13:50.495305 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:50.495268    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bb46bd4d-s5qzv_63a50a99-77c3-477d-9e21-cd1066fb698d/kube-rbac-proxy-web/0.log"
Jul 07 17:13:50.534667 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:50.534621    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bb46bd4d-s5qzv_63a50a99-77c3-477d-9e21-cd1066fb698d/kube-rbac-proxy/0.log"
Jul 07 17:13:50.566081 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:50.566051    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bb46bd4d-s5qzv_63a50a99-77c3-477d-9e21-cd1066fb698d/prom-label-proxy/0.log"
Jul 07 17:13:50.594710 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:50.594676    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bb46bd4d-s5qzv_63a50a99-77c3-477d-9e21-cd1066fb698d/kube-rbac-proxy-rules/0.log"
Jul 07 17:13:50.629879 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:50.629853    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bb46bd4d-s5qzv_63a50a99-77c3-477d-9e21-cd1066fb698d/kube-rbac-proxy-metrics/0.log"
Jul 07 17:13:53.114610 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.114577    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g455h/perf-node-gather-daemonset-9xz76"]
Jul 07 17:13:53.115118 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.114903    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb392c81-3e44-4f5f-826f-95a6a4032f71" containerName="gather"
Jul 07 17:13:53.115118 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.114913    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb392c81-3e44-4f5f-826f-95a6a4032f71" containerName="gather"
Jul 07 17:13:53.115118 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.114932    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb392c81-3e44-4f5f-826f-95a6a4032f71" containerName="copy"
Jul 07 17:13:53.115118 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.114937    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb392c81-3e44-4f5f-826f-95a6a4032f71" containerName="copy"
Jul 07 17:13:53.115118 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.114990    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb392c81-3e44-4f5f-826f-95a6a4032f71" containerName="copy"
Jul 07 17:13:53.115118 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.115001    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb392c81-3e44-4f5f-826f-95a6a4032f71" containerName="gather"
Jul 07 17:13:53.119429 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.119410    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g455h/perf-node-gather-daemonset-9xz76"
Jul 07 17:13:53.122095 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.122079    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g455h\"/\"kube-root-ca.crt\""
Jul 07 17:13:53.123205 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.123187    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g455h\"/\"default-dockercfg-4pbz6\""
Jul 07 17:13:53.123289 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.123273    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g455h\"/\"openshift-service-ca.crt\""
Jul 07 17:13:53.129436 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.129412    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g455h/perf-node-gather-daemonset-9xz76"]
Jul 07 17:13:53.148538 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.148506    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dff5220f-7710-48da-b7a0-2e22bba044b4-proc\") pod \"perf-node-gather-daemonset-9xz76\" (UID: \"dff5220f-7710-48da-b7a0-2e22bba044b4\") " pod="openshift-must-gather-g455h/perf-node-gather-daemonset-9xz76"
Jul 07 17:13:53.148699 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.148544    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dff5220f-7710-48da-b7a0-2e22bba044b4-lib-modules\") pod \"perf-node-gather-daemonset-9xz76\" (UID: \"dff5220f-7710-48da-b7a0-2e22bba044b4\") " pod="openshift-must-gather-g455h/perf-node-gather-daemonset-9xz76"
Jul 07 17:13:53.148699 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.148563    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dff5220f-7710-48da-b7a0-2e22bba044b4-sys\") pod \"perf-node-gather-daemonset-9xz76\" (UID: \"dff5220f-7710-48da-b7a0-2e22bba044b4\") " pod="openshift-must-gather-g455h/perf-node-gather-daemonset-9xz76"
Jul 07 17:13:53.148699 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.148620    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7hqf\" (UniqueName: \"kubernetes.io/projected/dff5220f-7710-48da-b7a0-2e22bba044b4-kube-api-access-s7hqf\") pod \"perf-node-gather-daemonset-9xz76\" (UID: \"dff5220f-7710-48da-b7a0-2e22bba044b4\") " pod="openshift-must-gather-g455h/perf-node-gather-daemonset-9xz76"
Jul 07 17:13:53.148834 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.148746    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dff5220f-7710-48da-b7a0-2e22bba044b4-podres\") pod \"perf-node-gather-daemonset-9xz76\" (UID: \"dff5220f-7710-48da-b7a0-2e22bba044b4\") " pod="openshift-must-gather-g455h/perf-node-gather-daemonset-9xz76"
Jul 07 17:13:53.249191 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.249159    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dff5220f-7710-48da-b7a0-2e22bba044b4-podres\") pod \"perf-node-gather-daemonset-9xz76\" (UID: \"dff5220f-7710-48da-b7a0-2e22bba044b4\") " pod="openshift-must-gather-g455h/perf-node-gather-daemonset-9xz76"
Jul 07 17:13:53.249381 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.249222    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dff5220f-7710-48da-b7a0-2e22bba044b4-proc\") pod \"perf-node-gather-daemonset-9xz76\" (UID: \"dff5220f-7710-48da-b7a0-2e22bba044b4\") " pod="openshift-must-gather-g455h/perf-node-gather-daemonset-9xz76"
Jul 07 17:13:53.249381 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.249247    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dff5220f-7710-48da-b7a0-2e22bba044b4-lib-modules\") pod \"perf-node-gather-daemonset-9xz76\" (UID: \"dff5220f-7710-48da-b7a0-2e22bba044b4\") " pod="openshift-must-gather-g455h/perf-node-gather-daemonset-9xz76"
Jul 07 17:13:53.249381 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.249263    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dff5220f-7710-48da-b7a0-2e22bba044b4-sys\") pod \"perf-node-gather-daemonset-9xz76\" (UID: \"dff5220f-7710-48da-b7a0-2e22bba044b4\") " pod="openshift-must-gather-g455h/perf-node-gather-daemonset-9xz76"
Jul 07 17:13:53.249381 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.249285    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7hqf\" (UniqueName: \"kubernetes.io/projected/dff5220f-7710-48da-b7a0-2e22bba044b4-kube-api-access-s7hqf\") pod \"perf-node-gather-daemonset-9xz76\" (UID: \"dff5220f-7710-48da-b7a0-2e22bba044b4\") " pod="openshift-must-gather-g455h/perf-node-gather-daemonset-9xz76"
Jul 07 17:13:53.249381 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.249350    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dff5220f-7710-48da-b7a0-2e22bba044b4-podres\") pod \"perf-node-gather-daemonset-9xz76\" (UID: \"dff5220f-7710-48da-b7a0-2e22bba044b4\") " pod="openshift-must-gather-g455h/perf-node-gather-daemonset-9xz76"
Jul 07 17:13:53.249381 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.249366    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dff5220f-7710-48da-b7a0-2e22bba044b4-sys\") pod \"perf-node-gather-daemonset-9xz76\" (UID: \"dff5220f-7710-48da-b7a0-2e22bba044b4\") " pod="openshift-must-gather-g455h/perf-node-gather-daemonset-9xz76"
Jul 07 17:13:53.249381 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.249367    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dff5220f-7710-48da-b7a0-2e22bba044b4-proc\") pod \"perf-node-gather-daemonset-9xz76\" (UID: \"dff5220f-7710-48da-b7a0-2e22bba044b4\") " pod="openshift-must-gather-g455h/perf-node-gather-daemonset-9xz76"
Jul 07 17:13:53.249644 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.249404    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dff5220f-7710-48da-b7a0-2e22bba044b4-lib-modules\") pod \"perf-node-gather-daemonset-9xz76\" (UID: \"dff5220f-7710-48da-b7a0-2e22bba044b4\") " pod="openshift-must-gather-g455h/perf-node-gather-daemonset-9xz76"
Jul 07 17:13:53.259708 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.259686    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7hqf\" (UniqueName: \"kubernetes.io/projected/dff5220f-7710-48da-b7a0-2e22bba044b4-kube-api-access-s7hqf\") pod \"perf-node-gather-daemonset-9xz76\" (UID: \"dff5220f-7710-48da-b7a0-2e22bba044b4\") " pod="openshift-must-gather-g455h/perf-node-gather-daemonset-9xz76"
Jul 07 17:13:53.429585 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.429500    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g455h/perf-node-gather-daemonset-9xz76"
Jul 07 17:13:53.574997 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.574967    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g455h/perf-node-gather-daemonset-9xz76"]
Jul 07 17:13:53.577405 ip-10-0-141-189 kubenswrapper[2583]: W0707 17:13:53.577378    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddff5220f_7710_48da_b7a0_2e22bba044b4.slice/crio-f31320788811b6ecd0a77009c8b886da05eb679749b2d4e21cf4ca181605f0e7 WatchSource:0}: Error finding container f31320788811b6ecd0a77009c8b886da05eb679749b2d4e21cf4ca181605f0e7: Status 404 returned error can't find the container with id f31320788811b6ecd0a77009c8b886da05eb679749b2d4e21cf4ca181605f0e7
Jul 07 17:13:53.868381 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.868348    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g455h/perf-node-gather-daemonset-9xz76" event={"ID":"dff5220f-7710-48da-b7a0-2e22bba044b4","Type":"ContainerStarted","Data":"1a9ff097a3dc052fb22d43a69b760697cc91fa2d46380e2a7dfc65735970786c"}
Jul 07 17:13:53.868571 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.868400    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g455h/perf-node-gather-daemonset-9xz76" event={"ID":"dff5220f-7710-48da-b7a0-2e22bba044b4","Type":"ContainerStarted","Data":"f31320788811b6ecd0a77009c8b886da05eb679749b2d4e21cf4ca181605f0e7"}
Jul 07 17:13:53.868571 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.868462    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-g455h/perf-node-gather-daemonset-9xz76"
Jul 07 17:13:53.896244 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:53.896189    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g455h/perf-node-gather-daemonset-9xz76" podStartSLOduration=0.89617484 podStartE2EDuration="896.17484ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:13:53.894112124 +0000 UTC m=+1626.907121969" watchObservedRunningTime="2026-07-07 17:13:53.89617484 +0000 UTC m=+1626.909184660"
Jul 07 17:13:55.043365 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:55.043341    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-grhmq_7c0e2b23-0949-4d6e-bd5e-11d1aff95701/dns/0.log"
Jul 07 17:13:55.075213 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:55.075190    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-grhmq_7c0e2b23-0949-4d6e-bd5e-11d1aff95701/kube-rbac-proxy/0.log"
Jul 07 17:13:55.149681 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:55.149647    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p6nfc_f7530471-b4ef-492d-b1bd-6393da0da34b/dns-node-resolver/0.log"
Jul 07 17:13:55.893810 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:55.893770    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mj2p8_20811d55-4cd6-4c95-9e50-e53a2b3bf2c7/node-ca/0.log"
Jul 07 17:13:57.288035 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:57.288006    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hjctb_9f753b07-d621-412c-8ffb-764e1a09ab1e/serve-healthcheck-canary/0.log"
Jul 07 17:13:57.826798 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:57.826770    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-58fvv_ad0977ee-f824-48f4-8f9f-e15ac2c968aa/kube-rbac-proxy/0.log"
Jul 07 17:13:57.856204 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:57.856174    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-58fvv_ad0977ee-f824-48f4-8f9f-e15ac2c968aa/exporter/0.log"
Jul 07 17:13:57.890415 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:57.890379    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-58fvv_ad0977ee-f824-48f4-8f9f-e15ac2c968aa/extractor/0.log"
Jul 07 17:13:59.882665 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:13:59.882639    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-g455h/perf-node-gather-daemonset-9xz76"
Jul 07 17:14:00.330902 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:14:00.330866    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-85768989b-rq7r5_84046f9b-fba7-4c61-abf9-f991d16bfb74/manager/0.log"
Jul 07 17:14:00.566708 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:14:00.566676    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-6slsd_d02288a1-c415-4487-9751-4b2b55a45360/s3-init/0.log"
Jul 07 17:14:00.613302 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:14:00.613220    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-6slsd_d02288a1-c415-4487-9751-4b2b55a45360/download-hf-model/0.log"
Jul 07 17:14:08.015337 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:14:08.015305    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gbzdh_99e5a676-4f78-4532-a5a4-87f1e55d2cba/kube-multus-additional-cni-plugins/0.log"
Jul 07 17:14:08.053739 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:14:08.053710    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gbzdh_99e5a676-4f78-4532-a5a4-87f1e55d2cba/egress-router-binary-copy/0.log"
Jul 07 17:14:08.082990 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:14:08.082959    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gbzdh_99e5a676-4f78-4532-a5a4-87f1e55d2cba/cni-plugins/0.log"
Jul 07 17:14:08.111727 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:14:08.111705    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gbzdh_99e5a676-4f78-4532-a5a4-87f1e55d2cba/bond-cni-plugin/0.log"
Jul 07 17:14:08.144778 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:14:08.144745    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gbzdh_99e5a676-4f78-4532-a5a4-87f1e55d2cba/routeoverride-cni/0.log"
Jul 07 17:14:08.177732 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:14:08.177706    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gbzdh_99e5a676-4f78-4532-a5a4-87f1e55d2cba/whereabouts-cni-bincopy/0.log"
Jul 07 17:14:08.208535 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:14:08.208503    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gbzdh_99e5a676-4f78-4532-a5a4-87f1e55d2cba/whereabouts-cni/0.log"
Jul 07 17:14:08.790815 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:14:08.790782    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gsc2d_f403e1aa-f12e-4df4-9626-79a45ab81549/kube-multus/0.log"
Jul 07 17:14:09.056505 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:14:09.056398    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vv52w_b399a96b-729b-484d-8edf-e97618163cfa/network-metrics-daemon/0.log"
Jul 07 17:14:09.081965 ip-10-0-141-189 kubenswrapper[2583]: I0707 17:14:09.081933    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vv52w_b399a96b-729b-484d-8edf-e97618163cfa/kube-rbac-proxy/0.log"