Jul 01 20:44:37.705290 ip-10-0-131-198 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 01 20:44:37.705303 ip-10-0-131-198 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 01 20:44:37.705312 ip-10-0-131-198 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 01 20:44:37.705611 ip-10-0-131-198 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 01 20:44:47.725978 ip-10-0-131-198 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 01 20:44:47.725999 ip-10-0-131-198 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 8734ce5039004e00a72902f6afc726eb --
Jul 01 20:46:40.213146 ip-10-0-131-198 systemd[1]: Starting Kubernetes Kubelet...
Jul 01 20:46:40.605604 ip-10-0-131-198 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 20:46:40.605604 ip-10-0-131-198 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 01 20:46:40.605604 ip-10-0-131-198 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 20:46:40.605604 ip-10-0-131-198 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 01 20:46:40.605604 ip-10-0-131-198 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 20:46:40.606543 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.606438    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 01 20:46:40.611686 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611663    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 20:46:40.611686 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611682    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 20:46:40.611686 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611686    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 20:46:40.611686 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611690    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 20:46:40.611686 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611693    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 20:46:40.611686 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611697    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 20:46:40.611909 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611700    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 20:46:40.611909 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611703    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 20:46:40.611909 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611706    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 20:46:40.611909 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611709    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 20:46:40.611909 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611712    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 20:46:40.611909 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611715    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 20:46:40.611909 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611717    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 20:46:40.611909 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611720    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 20:46:40.611909 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611723    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 20:46:40.611909 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611726    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 20:46:40.611909 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611728    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 20:46:40.611909 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611732    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 20:46:40.611909 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611735    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 20:46:40.611909 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611738    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 20:46:40.611909 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611741    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 20:46:40.611909 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611743    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 20:46:40.611909 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611747    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 20:46:40.611909 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611750    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 20:46:40.611909 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611753    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 20:46:40.611909 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611756    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 20:46:40.612465 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611759    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 20:46:40.612465 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611761    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 20:46:40.612465 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611764    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 20:46:40.612465 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611767    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 20:46:40.612465 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611770    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 20:46:40.612465 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611773    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 20:46:40.612465 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611776    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 20:46:40.612465 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611779    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 20:46:40.612465 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611781    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 20:46:40.612465 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611784    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 20:46:40.612465 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611787    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 20:46:40.612465 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611789    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 20:46:40.612465 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611792    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 20:46:40.612465 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611795    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 20:46:40.612465 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611798    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 20:46:40.612465 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611801    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 20:46:40.612465 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611804    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 20:46:40.612465 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611807    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 20:46:40.612465 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611809    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 20:46:40.612465 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611812    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 20:46:40.612987 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611815    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 20:46:40.612987 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611818    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 20:46:40.612987 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611821    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 20:46:40.612987 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611824    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 20:46:40.612987 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611827    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 20:46:40.612987 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611831    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 20:46:40.612987 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611834    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 20:46:40.612987 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611836    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 20:46:40.612987 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611839    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 20:46:40.612987 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611841    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 20:46:40.612987 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611845    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 20:46:40.612987 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611847    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 20:46:40.612987 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611850    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 20:46:40.612987 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611853    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 20:46:40.612987 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611856    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 20:46:40.612987 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611858    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 20:46:40.612987 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611861    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 20:46:40.612987 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611864    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 20:46:40.612987 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611866    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 20:46:40.612987 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611869    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 20:46:40.613520 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611872    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 20:46:40.613520 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611876    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 20:46:40.613520 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611881    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 20:46:40.613520 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611884    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 20:46:40.613520 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611887    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 20:46:40.613520 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611890    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 20:46:40.613520 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611896    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 20:46:40.613520 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611900    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 20:46:40.613520 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611903    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 20:46:40.613520 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611906    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 20:46:40.613520 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611909    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 20:46:40.613520 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611912    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 20:46:40.613520 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611916    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 20:46:40.613520 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611919    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 20:46:40.613520 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611922    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 20:46:40.613520 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611926    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 20:46:40.613520 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611949    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 20:46:40.613520 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611953    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 20:46:40.613520 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611957    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 20:46:40.614012 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611960    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 20:46:40.614012 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611963    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 20:46:40.614012 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611966    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 20:46:40.614012 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611970    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 20:46:40.614012 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611973    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 20:46:40.614012 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611976    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 20:46:40.614012 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611978    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 20:46:40.614012 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611981    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 20:46:40.614012 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611984    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 20:46:40.614012 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611987    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 20:46:40.614012 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611989    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 20:46:40.614012 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611992    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 20:46:40.614012 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611994    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 20:46:40.614012 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.611997    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 20:46:40.614012 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612743    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 20:46:40.614012 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612751    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 20:46:40.614012 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612754    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 20:46:40.614012 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612757    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 20:46:40.614012 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612760    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 20:46:40.614012 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612763    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 20:46:40.614536 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612766    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 20:46:40.614536 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612769    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 20:46:40.614536 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612772    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 20:46:40.614536 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612775    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 20:46:40.614536 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612778    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 20:46:40.614536 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612781    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 20:46:40.614536 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612784    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 20:46:40.614536 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612786    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 20:46:40.614536 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612789    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 20:46:40.614536 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612792    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 20:46:40.614536 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612795    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 20:46:40.614536 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612800    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 20:46:40.614536 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612805    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 20:46:40.614536 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612808    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 20:46:40.614536 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612811    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 20:46:40.614536 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612814    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 20:46:40.614536 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612817    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 20:46:40.614536 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612820    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 20:46:40.614536 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612823    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 20:46:40.615096 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612826    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 20:46:40.615096 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612829    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 20:46:40.615096 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612832    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 20:46:40.615096 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612835    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 20:46:40.615096 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612838    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 20:46:40.615096 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612841    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 20:46:40.615096 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612843    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 20:46:40.615096 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612846    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 20:46:40.615096 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612849    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 20:46:40.615096 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612852    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 20:46:40.615096 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612854    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 20:46:40.615096 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612858    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 20:46:40.615096 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612861    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 20:46:40.615096 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612864    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 20:46:40.615096 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612867    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 20:46:40.615096 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612870    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 20:46:40.615096 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612873    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 20:46:40.615096 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612876    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 20:46:40.615096 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612879    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 20:46:40.615096 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612882    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 20:46:40.615618 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612884    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 20:46:40.615618 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612887    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 20:46:40.615618 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612890    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 20:46:40.615618 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612893    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 20:46:40.615618 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612895    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 20:46:40.615618 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612899    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 20:46:40.615618 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612902    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 20:46:40.615618 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612905    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 20:46:40.615618 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612908    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 20:46:40.615618 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612911    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 20:46:40.615618 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612915    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 20:46:40.615618 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612918    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 20:46:40.615618 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612920    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 20:46:40.615618 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612923    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 20:46:40.615618 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612926    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 20:46:40.615618 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612929    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 20:46:40.615618 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612947    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 20:46:40.615618 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612951    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 20:46:40.615618 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612953    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 20:46:40.615618 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612956    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 20:46:40.616164 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612959    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 20:46:40.616164 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612962    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 20:46:40.616164 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612965    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 20:46:40.616164 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612967    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 20:46:40.616164 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612970    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 20:46:40.616164 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612973    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 20:46:40.616164 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612976    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 20:46:40.616164 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612979    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 20:46:40.616164 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612982    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 20:46:40.616164 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612984    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 20:46:40.616164 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612987    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 20:46:40.616164 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612990    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 20:46:40.616164 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612992    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 20:46:40.616164 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612995    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 20:46:40.616164 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.612999    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 20:46:40.616164 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613002    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 20:46:40.616164 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613005    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 20:46:40.616164 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613008    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 20:46:40.616164 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613012    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 20:46:40.616164 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613015    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 20:46:40.616684 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613018    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 20:46:40.616684 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613020    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 20:46:40.616684 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613023    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 20:46:40.616684 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613026    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 20:46:40.616684 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613028    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 20:46:40.616684 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613031    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 20:46:40.616684 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613034    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 20:46:40.616684 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613038    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 20:46:40.616684 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613042    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 20:46:40.616684 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613046    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 20:46:40.616684 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613050    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 20:46:40.616684 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613053    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 20:46:40.616684 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613056    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 20:46:40.616684 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613059    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 20:46:40.616684 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613455    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 20:46:40.616684 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613460    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 20:46:40.616684 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613463    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 20:46:40.616684 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613466    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 20:46:40.616684 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613469    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 20:46:40.617227 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613472    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 20:46:40.617227 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613474    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 20:46:40.617227 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613477    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 20:46:40.617227 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613480    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 20:46:40.617227 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613483    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 20:46:40.617227 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613488    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 20:46:40.617227 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613493    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 20:46:40.617227 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613497    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 20:46:40.617227 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613500    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 20:46:40.617227 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613503    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 20:46:40.617227 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613506    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 20:46:40.617227 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613510    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 20:46:40.617227 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613513    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 20:46:40.617227 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613517    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 20:46:40.617227 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613521    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 20:46:40.617227 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613525    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 20:46:40.617227 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613528    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 20:46:40.617227 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613532    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 20:46:40.617227 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613535    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 20:46:40.617722 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613538    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 20:46:40.617722 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613540    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 20:46:40.617722 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613543    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 20:46:40.617722 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613546    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 20:46:40.617722 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613549    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 20:46:40.617722 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613553    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 20:46:40.617722 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613556    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 20:46:40.617722 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613559    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 20:46:40.617722 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613562    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 20:46:40.617722 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613565    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 20:46:40.617722 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613568    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 20:46:40.617722 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613571    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 20:46:40.617722 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613574    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 20:46:40.617722 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613577    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 20:46:40.617722 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613580    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 20:46:40.617722 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613591    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 20:46:40.617722 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613594    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 20:46:40.617722 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613597    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 20:46:40.617722 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613600    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 20:46:40.618246 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613603    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 20:46:40.618246 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613606    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 20:46:40.618246 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613609    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 20:46:40.618246 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613612    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 20:46:40.618246 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613615    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 20:46:40.618246 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613617    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 20:46:40.618246 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613626    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 20:46:40.618246 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613629    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 20:46:40.618246 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613631    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 20:46:40.618246 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613635    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 20:46:40.618246 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613638    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 20:46:40.618246 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613640    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 20:46:40.618246 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613643    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 20:46:40.618246 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613646    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 20:46:40.618246 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613649    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 20:46:40.618246 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613652    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 20:46:40.618246 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613655    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 20:46:40.618246 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613658    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 20:46:40.618246 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613661    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 20:46:40.618829 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613664    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 20:46:40.618829 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613667    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 20:46:40.618829 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613670    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 20:46:40.618829 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613673    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 20:46:40.618829 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613676    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 20:46:40.618829 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613679    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 20:46:40.618829 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613681    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 20:46:40.618829 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613684    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 20:46:40.618829 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613687    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 20:46:40.618829 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613690    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 20:46:40.618829 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613693    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 20:46:40.618829 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613696    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 20:46:40.618829 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613698    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 20:46:40.618829 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613703    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 20:46:40.618829 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613706    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 20:46:40.618829 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613709    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 20:46:40.618829 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613711    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 20:46:40.618829 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613714    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 20:46:40.618829 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613717    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 20:46:40.618829 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613719    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 20:46:40.619366 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613723    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 20:46:40.619366 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613725    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 20:46:40.619366 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613728    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 20:46:40.619366 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613731    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 20:46:40.619366 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613734    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 20:46:40.619366 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613736    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 20:46:40.619366 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613739    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 20:46:40.619366 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613741    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 20:46:40.619366 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613744    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 20:46:40.619366 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613747    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 20:46:40.619366 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613750    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 20:46:40.619366 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613753    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 20:46:40.619366 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613755    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 20:46:40.619366 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613758    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 20:46:40.619366 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613761    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 20:46:40.619366 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613763    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 20:46:40.619366 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.613766    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 20:46:40.619366 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613837    2564 flags.go:64] FLAG: --address="0.0.0.0"
Jul 01 20:46:40.619366 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613844    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 01 20:46:40.619366 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613851    2564 flags.go:64] FLAG: --anonymous-auth="true"
Jul 01 20:46:40.619366 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613865    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 01 20:46:40.619909 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613869    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 01 20:46:40.619909 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613873    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 01 20:46:40.619909 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613877    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 01 20:46:40.619909 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613882    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 01 20:46:40.619909 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613885    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 01 20:46:40.619909 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613888    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 01 20:46:40.619909 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613892    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 01 20:46:40.619909 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613895    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 01 20:46:40.619909 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613898    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 01 20:46:40.619909 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613902    2564 flags.go:64] FLAG: --cgroup-root=""
Jul 01 20:46:40.619909 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613905    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 01 20:46:40.619909 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613908    2564 flags.go:64] FLAG: --client-ca-file=""
Jul 01 20:46:40.619909 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613911    2564 flags.go:64] FLAG: --cloud-provider="external"
Jul 01 20:46:40.619909 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613914    2564 flags.go:64] FLAG: --cluster-dns="[]"
Jul 01 20:46:40.619909 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613919    2564 flags.go:64] FLAG: --cluster-domain=""
Jul 01 20:46:40.619909 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613921    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 01 20:46:40.619909 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613925    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 01 20:46:40.619909 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613928    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 01 20:46:40.619909 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613945    2564 flags.go:64] FLAG: --container-log-max-files="5"
Jul 01 20:46:40.619909 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613949    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 01 20:46:40.619909 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613952    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 01 20:46:40.619909 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613955    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 01 20:46:40.619909 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613959    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 01 20:46:40.620503 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613963    2564 flags.go:64] FLAG: --contention-profiling="false"
Jul 01 20:46:40.620503 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613966    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 01 20:46:40.620503 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613969    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 01 20:46:40.620503 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613973    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 01 20:46:40.620503 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613975    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 01 20:46:40.620503 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613980    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 01 20:46:40.620503 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613983    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 01 20:46:40.620503 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613986    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 01 20:46:40.620503 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613989    2564 flags.go:64] FLAG: --enable-load-reader="false"
Jul 01 20:46:40.620503 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613993    2564 flags.go:64] FLAG: --enable-server="true"
Jul 01 20:46:40.620503 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.613996    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 01 20:46:40.620503 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614001    2564 flags.go:64] FLAG: --event-burst="100"
Jul 01 20:46:40.620503 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614004    2564 flags.go:64] FLAG: --event-qps="50"
Jul 01 20:46:40.620503 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614007    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 01 20:46:40.620503 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614011    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 01 20:46:40.620503 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614014    2564 flags.go:64] FLAG: --eviction-hard=""
Jul 01 20:46:40.620503 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614018    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 01 20:46:40.620503 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614021    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 01 20:46:40.620503 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614024    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 01 20:46:40.620503 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614028    2564 flags.go:64] FLAG: --eviction-soft=""
Jul 01 20:46:40.620503 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614030    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 01 20:46:40.620503 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614033    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 01 20:46:40.620503 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614037    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 01 20:46:40.620503 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614040    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 01 20:46:40.620503 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614044    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 01 20:46:40.621207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614047    2564 flags.go:64] FLAG: --fail-swap-on="true"
Jul 01 20:46:40.621207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614050    2564 flags.go:64] FLAG: --feature-gates=""
Jul 01 20:46:40.621207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614054    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 01 20:46:40.621207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614057    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 01 20:46:40.621207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614060    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 01 20:46:40.621207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614064    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 01 20:46:40.621207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614069    2564 flags.go:64] FLAG: --healthz-port="10248"
Jul 01 20:46:40.621207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614073    2564 flags.go:64] FLAG: --help="false"
Jul 01 20:46:40.621207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614076    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-131-198.ec2.internal"
Jul 01 20:46:40.621207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614079    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 01 20:46:40.621207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614082    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 01 20:46:40.621207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614085    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 01 20:46:40.621207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614089    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 01 20:46:40.621207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614092    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 01 20:46:40.621207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614096    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 01 20:46:40.621207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614099    2564 flags.go:64] FLAG: --image-service-endpoint=""
Jul 01 20:46:40.621207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614102    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 01 20:46:40.621207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614105    2564 flags.go:64] FLAG: --kube-api-burst="100"
Jul 01 20:46:40.621207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614108    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 01 20:46:40.621207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614111    2564 flags.go:64] FLAG: --kube-api-qps="50"
Jul 01 20:46:40.621207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614114    2564 flags.go:64] FLAG: --kube-reserved=""
Jul 01 20:46:40.621207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614117    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 01 20:46:40.621207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614120    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 01 20:46:40.621207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614123    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 01 20:46:40.621834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614126    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 01 20:46:40.621834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614129    2564 flags.go:64] FLAG: --lock-file=""
Jul 01 20:46:40.621834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614132    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 01 20:46:40.621834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614135    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 01 20:46:40.621834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614139    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 01 20:46:40.621834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614144    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 01 20:46:40.621834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614148    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 01 20:46:40.621834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614151    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 01 20:46:40.621834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614154    2564 flags.go:64] FLAG: --logging-format="text"
Jul 01 20:46:40.621834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614157    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 01 20:46:40.621834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614160    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 01 20:46:40.621834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614163    2564 flags.go:64] FLAG: --manifest-url=""
Jul 01 20:46:40.621834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614166    2564 flags.go:64] FLAG: --manifest-url-header=""
Jul 01 20:46:40.621834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614171    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 01 20:46:40.621834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614175    2564 flags.go:64] FLAG: --max-open-files="1000000"
Jul 01 20:46:40.621834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614180    2564 flags.go:64] FLAG: --max-pods="110"
Jul 01 20:46:40.621834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614183    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 01 20:46:40.621834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614186    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 01 20:46:40.621834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614189    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 01 20:46:40.621834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614192    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 01 20:46:40.621834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614195    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 01 20:46:40.621834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614198    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 01 20:46:40.621834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614201    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 01 20:46:40.621834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614209    2564 flags.go:64] FLAG: --node-status-max-images="50"
Jul 01 20:46:40.621834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614212    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 01 20:46:40.622515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614216    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 01 20:46:40.622515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614219    2564 flags.go:64] FLAG: --pod-cidr=""
Jul 01 20:46:40.622515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614222    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 01 20:46:40.622515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614227    2564 flags.go:64] FLAG: --pod-manifest-path=""
Jul 01 20:46:40.622515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614230    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 01 20:46:40.622515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614234    2564 flags.go:64] FLAG: --pods-per-core="0"
Jul 01 20:46:40.622515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614236    2564 flags.go:64] FLAG: --port="10250"
Jul 01 20:46:40.622515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614240    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 01 20:46:40.622515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614243    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ef09ccee90063039"
Jul 01 20:46:40.622515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614246    2564 flags.go:64] FLAG: --qos-reserved=""
Jul 01 20:46:40.622515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614253    2564 flags.go:64] FLAG: --read-only-port="10255"
Jul 01 20:46:40.622515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614256    2564 flags.go:64] FLAG: --register-node="true"
Jul 01 20:46:40.622515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614259    2564 flags.go:64] FLAG: --register-with-taints=""
Jul 01 20:46:40.622515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614262    2564 flags.go:64] FLAG: --registry-burst="10"
Jul 01 20:46:40.622515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614266    2564 flags.go:64] FLAG: --registry-qps="5"
Jul 01 20:46:40.622515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614269    2564 flags.go:64] FLAG: --reserved-cpus=""
Jul 01 20:46:40.622515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614272    2564 flags.go:64] FLAG: --reserved-memory=""
Jul 01 20:46:40.622515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614276    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 01 20:46:40.622515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614279    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 01 20:46:40.622515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614283    2564 flags.go:64] FLAG: --rotate-certificates="false"
Jul 01 20:46:40.622515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614286    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 01 20:46:40.622515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614290    2564 flags.go:64] FLAG: --runonce="false"
Jul 01 20:46:40.622515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614293    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 01 20:46:40.622515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614296    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 01 20:46:40.622515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614299    2564 flags.go:64] FLAG: --seccomp-default="false"
Jul 01 20:46:40.623191 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614302    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 01 20:46:40.623191 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614305    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 01 20:46:40.623191 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614309    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 01 20:46:40.623191 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614312    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 01 20:46:40.623191 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614315    2564 flags.go:64] FLAG: --storage-driver-password="root"
Jul 01 20:46:40.623191 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614318    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 01 20:46:40.623191 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614321    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 01 20:46:40.623191 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614324    2564 flags.go:64] FLAG: --storage-driver-user="root"
Jul 01 20:46:40.623191 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614327    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 01 20:46:40.623191 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614331    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 01 20:46:40.623191 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614334    2564 flags.go:64] FLAG: --system-cgroups=""
Jul 01 20:46:40.623191 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614337    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 01 20:46:40.623191 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614342    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 01 20:46:40.623191 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614346    2564 flags.go:64] FLAG: --tls-cert-file=""
Jul 01 20:46:40.623191 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614349    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 01 20:46:40.623191 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614353    2564 flags.go:64] FLAG: --tls-min-version=""
Jul 01 20:46:40.623191 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614356    2564 flags.go:64] FLAG: --tls-private-key-file=""
Jul 01 20:46:40.623191 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614361    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 01 20:46:40.623191 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614364    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 01 20:46:40.623191 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614367    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 01 20:46:40.623191 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614371    2564 flags.go:64] FLAG: --v="2"
Jul 01 20:46:40.623191 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614375    2564 flags.go:64] FLAG: --version="false"
Jul 01 20:46:40.623191 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614379    2564 flags.go:64] FLAG: --vmodule=""
Jul 01 20:46:40.623191 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614383    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 01 20:46:40.623191 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.614386    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 01 20:46:40.623839 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614502    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 20:46:40.623839 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614506    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 20:46:40.623839 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614509    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 20:46:40.623839 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614514    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 20:46:40.623839 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614517    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 20:46:40.623839 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614520    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 20:46:40.623839 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614523    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 20:46:40.623839 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614526    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 20:46:40.623839 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614530    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 20:46:40.623839 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614534    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 20:46:40.623839 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614537    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 20:46:40.623839 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614540    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 20:46:40.623839 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614543    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 20:46:40.623839 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614548    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 20:46:40.623839 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614552    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 20:46:40.623839 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614555    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 20:46:40.623839 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614559    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 20:46:40.623839 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614562    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 20:46:40.623839 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614565    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 20:46:40.624345 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614568    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 20:46:40.624345 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614570    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 20:46:40.624345 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614573    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 20:46:40.624345 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614577    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 20:46:40.624345 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614579    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 20:46:40.624345 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614583    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 20:46:40.624345 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614586    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 20:46:40.624345 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614589    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 20:46:40.624345 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614592    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 20:46:40.624345 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614595    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 20:46:40.624345 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614598    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 20:46:40.624345 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614607    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 20:46:40.624345 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614610    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 20:46:40.624345 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614613    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 20:46:40.624345 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614616    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 20:46:40.624345 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614618    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 20:46:40.624345 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614623    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 20:46:40.624345 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614625    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 20:46:40.624345 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614628    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 20:46:40.624345 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614632    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 20:46:40.624903 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614634    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 20:46:40.624903 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614637    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 20:46:40.624903 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614640    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 20:46:40.624903 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614643    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 20:46:40.624903 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614646    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 20:46:40.624903 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614648    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 20:46:40.624903 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614651    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 20:46:40.624903 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614654    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 20:46:40.624903 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614657    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 20:46:40.624903 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614660    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 20:46:40.624903 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614663    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 20:46:40.624903 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614665    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 20:46:40.624903 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614668    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 20:46:40.624903 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614671    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 20:46:40.624903 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614674    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 20:46:40.624903 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614677    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 20:46:40.624903 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614679    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 20:46:40.624903 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614683    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 20:46:40.624903 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614686    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 20:46:40.624903 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614689    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 20:46:40.625437 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614692    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 20:46:40.625437 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614695    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 20:46:40.625437 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614698    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 20:46:40.625437 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614701    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 20:46:40.625437 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614704    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 20:46:40.625437 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614707    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 20:46:40.625437 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614710    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 20:46:40.625437 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614713    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 20:46:40.625437 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614717    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 20:46:40.625437 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614720    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 20:46:40.625437 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614723    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 20:46:40.625437 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614726    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 20:46:40.625437 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614728    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 20:46:40.625437 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614731    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 20:46:40.625437 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614734    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 20:46:40.625437 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614736    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 20:46:40.625437 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614739    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 20:46:40.625437 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614742    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 20:46:40.625437 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614745    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 20:46:40.625437 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614748    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 20:46:40.626051 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614751    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 20:46:40.626051 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614754    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 20:46:40.626051 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614756    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 20:46:40.626051 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614759    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 20:46:40.626051 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614762    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 20:46:40.626051 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614765    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 20:46:40.626051 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614768    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 20:46:40.626051 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614771    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 20:46:40.626051 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614773    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 20:46:40.626051 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614779    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 20:46:40.626051 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614782    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 20:46:40.626051 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614786    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 20:46:40.626051 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614789    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 20:46:40.626051 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614792    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 20:46:40.626051 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614794    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 20:46:40.626051 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614798    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 20:46:40.626051 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614800    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 20:46:40.626051 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614804    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 20:46:40.626051 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614807    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 20:46:40.626051 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.614810    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 20:46:40.626578 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.615615    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 20:46:40.626578 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.622726    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 01 20:46:40.626578 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.622745    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 01 20:46:40.626578 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.622759    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 01 20:46:40.626578 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.622768    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 01 20:46:40.626578 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622824    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 20:46:40.626578 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622830    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 20:46:40.626578 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622834    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 20:46:40.626578 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622837    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 20:46:40.626578 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622841    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 20:46:40.626578 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622843    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 20:46:40.626578 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622846    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 20:46:40.626578 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622850    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 20:46:40.626578 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622853    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 20:46:40.626978 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622856    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 20:46:40.626978 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622859    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 20:46:40.626978 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622862    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 20:46:40.626978 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622865    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 20:46:40.626978 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622867    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 20:46:40.626978 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622870    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 20:46:40.626978 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622873    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 20:46:40.626978 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622876    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 20:46:40.626978 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622878    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 20:46:40.626978 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622881    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 20:46:40.626978 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622884    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 20:46:40.626978 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622886    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 20:46:40.626978 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622889    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 20:46:40.626978 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622892    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 20:46:40.626978 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622895    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 20:46:40.626978 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622897    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 20:46:40.626978 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622900    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 20:46:40.626978 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622903    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 20:46:40.626978 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622906    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 20:46:40.626978 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622908    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 20:46:40.627499 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622913    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 20:46:40.627499 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622916    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 20:46:40.627499 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622918    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 20:46:40.627499 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622921    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 20:46:40.627499 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622924    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 20:46:40.627499 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622927    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 20:46:40.627499 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622929    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 20:46:40.627499 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622948    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 20:46:40.627499 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622952    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 20:46:40.627499 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622955    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 20:46:40.627499 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622957    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 20:46:40.627499 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622960    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 20:46:40.627499 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622963    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 20:46:40.627499 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622966    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 20:46:40.627499 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622968    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 20:46:40.627499 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622971    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 20:46:40.627499 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622974    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 20:46:40.627499 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622976    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 20:46:40.627499 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622979    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 20:46:40.627499 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622981    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 20:46:40.628035 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622984    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 20:46:40.628035 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622988    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 20:46:40.628035 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622992    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 20:46:40.628035 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622996    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 20:46:40.628035 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.622999    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 20:46:40.628035 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623003    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 20:46:40.628035 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623008    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 20:46:40.628035 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623012    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 20:46:40.628035 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623015    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 20:46:40.628035 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623018    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 20:46:40.628035 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623022    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 20:46:40.628035 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623025    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 20:46:40.628035 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623028    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 20:46:40.628035 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623031    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 20:46:40.628035 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623035    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 20:46:40.628035 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623038    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 20:46:40.628035 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623041    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 20:46:40.628035 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623043    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 20:46:40.628035 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623046    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 20:46:40.628555 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623049    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 20:46:40.628555 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623051    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 20:46:40.628555 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623054    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 20:46:40.628555 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623057    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 20:46:40.628555 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623059    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 20:46:40.628555 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623062    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 20:46:40.628555 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623064    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 20:46:40.628555 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623067    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 20:46:40.628555 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623070    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 20:46:40.628555 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623072    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 20:46:40.628555 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623075    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 20:46:40.628555 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623078    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 20:46:40.628555 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623080    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 20:46:40.628555 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623083    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 20:46:40.628555 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623086    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 20:46:40.628555 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623088    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 20:46:40.628555 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623091    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 20:46:40.628555 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623094    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 20:46:40.628555 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623096    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 20:46:40.629067 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623099    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 20:46:40.629067 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623101    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 20:46:40.629067 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623104    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 20:46:40.629067 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623107    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 20:46:40.629067 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623110    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 20:46:40.629067 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623112    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 20:46:40.629067 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623115    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 20:46:40.629067 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623118    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 20:46:40.629067 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623121    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 20:46:40.629067 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623123    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 20:46:40.629067 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623126    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 20:46:40.629067 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623129    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 20:46:40.629067 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.623134    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 20:46:40.629067 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623247    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 20:46:40.629067 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623251    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 20:46:40.629455 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623255    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 20:46:40.629455 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623259    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 20:46:40.629455 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623262    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 20:46:40.629455 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623265    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 20:46:40.629455 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623268    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 20:46:40.629455 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623271    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 20:46:40.629455 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623275    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 20:46:40.629455 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623277    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 20:46:40.629455 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623280    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 20:46:40.629455 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623283    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 20:46:40.629455 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623285    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 20:46:40.629455 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623288    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 20:46:40.629455 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623290    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 20:46:40.629455 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623293    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 20:46:40.629455 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623296    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 20:46:40.629455 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623298    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 20:46:40.629455 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623301    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 20:46:40.629455 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623304    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 20:46:40.629455 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623307    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 20:46:40.629455 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623309    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 20:46:40.630082 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623312    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 20:46:40.630082 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623315    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 20:46:40.630082 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623317    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 20:46:40.630082 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623320    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 20:46:40.630082 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623323    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 20:46:40.630082 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623326    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 20:46:40.630082 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623329    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 20:46:40.630082 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623332    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 20:46:40.630082 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623336    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 20:46:40.630082 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623339    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 20:46:40.630082 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623341    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 20:46:40.630082 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623344    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 20:46:40.630082 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623347    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 20:46:40.630082 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623349    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 20:46:40.630082 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623352    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 20:46:40.630082 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623355    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 20:46:40.630082 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623357    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 20:46:40.630082 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623360    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 20:46:40.630082 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623362    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 20:46:40.630591 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623365    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 20:46:40.630591 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623368    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 20:46:40.630591 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623370    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 20:46:40.630591 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623373    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 20:46:40.630591 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623375    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 20:46:40.630591 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623378    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 20:46:40.630591 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623381    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 20:46:40.630591 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623384    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 20:46:40.630591 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623386    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 20:46:40.630591 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623389    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 20:46:40.630591 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623392    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 20:46:40.630591 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623394    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 20:46:40.630591 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623397    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 20:46:40.630591 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623400    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 20:46:40.630591 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623403    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 20:46:40.630591 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623405    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 20:46:40.630591 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623408    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 20:46:40.630591 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623411    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 20:46:40.630591 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623414    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 20:46:40.630591 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623417    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 20:46:40.631129 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623420    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 20:46:40.631129 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623422    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 20:46:40.631129 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623425    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 20:46:40.631129 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623428    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 20:46:40.631129 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623430    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 20:46:40.631129 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623433    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 20:46:40.631129 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623435    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 20:46:40.631129 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623441    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 20:46:40.631129 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623444    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 20:46:40.631129 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623446    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 20:46:40.631129 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623449    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 20:46:40.631129 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623452    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 20:46:40.631129 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623454    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 20:46:40.631129 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623457    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 20:46:40.631129 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623460    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 20:46:40.631129 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623462    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 20:46:40.631129 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623465    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 20:46:40.631129 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623467    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 20:46:40.631129 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623470    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 20:46:40.631627 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623474    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 20:46:40.631627 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623477    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 20:46:40.631627 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623480    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 20:46:40.631627 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623483    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 20:46:40.631627 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623487    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 20:46:40.631627 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623489    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 20:46:40.631627 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623492    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 20:46:40.631627 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623495    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 20:46:40.631627 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623497    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 20:46:40.631627 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623500    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 20:46:40.631627 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623503    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 20:46:40.631627 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623505    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 20:46:40.631627 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623508    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 20:46:40.631627 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623511    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 20:46:40.631627 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623514    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 20:46:40.631627 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623516    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 20:46:40.631627 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623519    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 20:46:40.631627 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623522    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 20:46:40.631627 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:40.623524    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 20:46:40.632176 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.623530    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 20:46:40.632176 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.623637    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 01 20:46:40.632176 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.625872    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 01 20:46:40.632176 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.627495    2564 server.go:1019] "Starting client certificate rotation"
Jul 01 20:46:40.632176 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.627601    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 20:46:40.632176 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.627630    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 20:46:40.632176 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.630061    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 20:46:40.632176 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.631705    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 20:46:40.640536 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.640518    2564 log.go:25] "Validated CRI v1 runtime API"
Jul 01 20:46:40.643386 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.643371    2564 log.go:25] "Validated CRI v1 image API"
Jul 01 20:46:40.644235 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.644219    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 01 20:46:40.647278 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.647259    2564 fs.go:135] Filesystem UUIDs: map[57216b84-e7f4-4eb3-b58d-ae2d22d003a1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9645af77-90c1-41e8-a54d-59d2f4907cea:/dev/nvme0n1p4]
Jul 01 20:46:40.647351 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.647277    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 01 20:46:40.653809 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.653702    2564 manager.go:216] Machine: {Timestamp:2026-07-01 20:46:40.65132885 +0000 UTC m=+0.327344411 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098900 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25b1cbc2b99d50e16c22a32c28b77f SystemUUID:ec25b1cb-c2b9-9d50-e16c-22a32c28b77f BootID:8734ce50-3900-4e00-a729-02f6afc726eb Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:bf:99:ea:41:ad Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:bf:99:ea:41:ad Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:fb:42:7e:93:9e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 01 20:46:40.653809 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.653807    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 01 20:46:40.653924 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.653882    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 01 20:46:40.655668 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.655646    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 01 20:46:40.655819 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.655670    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-198.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 01 20:46:40.657021 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.657009    2564 topology_manager.go:138] "Creating topology manager with none policy"
Jul 01 20:46:40.657063 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.657026    2564 container_manager_linux.go:309] "Creating device plugin manager"
Jul 01 20:46:40.657063 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.657043    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 20:46:40.657799 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.657788    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 20:46:40.658601 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.658591    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 01 20:46:40.661539 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.661505    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 20:46:40.661652 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.661643    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 01 20:46:40.661684 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.661673    2564 kubelet.go:486] "Attempting to sync node with API server"
Jul 01 20:46:40.661684 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.661683    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 01 20:46:40.661778 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.661702    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 01 20:46:40.661778 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.661711    2564 kubelet.go:398] "Adding apiserver pod source"
Jul 01 20:46:40.661778 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.661721    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 01 20:46:40.662117 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.662105    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 20:46:40.662155 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.662130    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 20:46:40.663073 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.663044    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 20:46:40.664383 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.664365    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 01 20:46:40.665944 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.665926    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 01 20:46:40.666006 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.665976    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 01 20:46:40.666700 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.666688    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 01 20:46:40.666780 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.666704    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 01 20:46:40.666780 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.666710    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 01 20:46:40.666780 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.666718    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 01 20:46:40.666780 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.666729    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 01 20:46:40.666780 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.666737    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 01 20:46:40.666780 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.666748    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 01 20:46:40.666780 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.666754    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 01 20:46:40.666780 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.666769    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 01 20:46:40.666780 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.666775    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 01 20:46:40.667071 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.666790    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 01 20:46:40.667071 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.666798    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 01 20:46:40.667547 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.667536    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 01 20:46:40.667547 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.667547    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 01 20:46:40.671221 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.671202    2564 server.go:1268] "Started kubelet"
Jul 01 20:46:40.671305 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.671252    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 01 20:46:40.672212 ip-10-0-131-198 systemd[1]: Started Kubernetes Kubelet.
Jul 01 20:46:40.672844 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.672497    2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-198.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 01 20:46:40.672844 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.671853    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 01 20:46:40.672844 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.672580    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 01 20:46:40.672844 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:40.672744    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 01 20:46:40.673182 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:40.672908    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-198.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 01 20:46:40.673431 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.673368    2564 server.go:310] "Adding debug handlers to kubelet server"
Jul 01 20:46:40.677446 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.673695    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 01 20:46:40.679081 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:40.679060    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 01 20:46:40.680806 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:40.679756    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-198.ec2.internal.18be45648e42a150  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-198.ec2.internal,UID:ip-10-0-131-198.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-198.ec2.internal,},FirstTimestamp:2026-07-01 20:46:40.671072592 +0000 UTC m=+0.347088153,LastTimestamp:2026-07-01 20:46:40.671072592 +0000 UTC m=+0.347088153,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-198.ec2.internal,}"
Jul 01 20:46:40.683382 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.683364    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 01 20:46:40.683459 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.683382    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 01 20:46:40.683533 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:40.683457    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-198.ec2.internal\" not found"
Jul 01 20:46:40.684061 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.684039    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 01 20:46:40.684160 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.684096    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 01 20:46:40.684160 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.684112    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 01 20:46:40.684160 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.684143    2564 factory.go:55] Registering systemd factory
Jul 01 20:46:40.684160 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.684161    2564 factory.go:223] Registration of the systemd container factory successfully
Jul 01 20:46:40.684369 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.684206    2564 reconstruct.go:97] "Volume reconstruction finished"
Jul 01 20:46:40.684369 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.684227    2564 reconciler.go:29] "Reconciler: start to sync state"
Jul 01 20:46:40.684369 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.684365    2564 factory.go:153] Registering CRI-O factory
Jul 01 20:46:40.684860 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.684379    2564 factory.go:223] Registration of the crio container factory successfully
Jul 01 20:46:40.684860 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.684434    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 01 20:46:40.684860 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.684461    2564 factory.go:103] Registering Raw factory
Jul 01 20:46:40.684860 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.684477    2564 manager.go:1196] Started watching for new ooms in manager
Jul 01 20:46:40.685083 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.684987    2564 manager.go:349] Starting recovery of all containers
Jul 01 20:46:40.685613 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:40.685583    2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-198.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 01 20:46:40.685757 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:40.685733    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 01 20:46:40.692650 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.692603    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 01 20:46:40.693503 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.693481    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nnnlw"
Jul 01 20:46:40.696989 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.696972    2564 manager.go:354] Recovery completed
Jul 01 20:46:40.701171 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.701158    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:40.703134 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.703119    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-198.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:40.703218 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.703148    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-198.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:40.703218 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.703162    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-198.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:40.703324 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.703285    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nnnlw"
Jul 01 20:46:40.703728 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.703710    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 01 20:46:40.703728 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.703724    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 01 20:46:40.703874 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.703741    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 20:46:40.703874 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.703743    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:40.704507 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.704486    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-198.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:40.704575 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.704524    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-198.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:40.704575 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.704541    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-198.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:40.705278 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.705254    2564 policy_none.go:49] "None policy: Start"
Jul 01 20:46:40.705278 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.705279    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 01 20:46:40.705485 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.705289    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 01 20:46:40.706664 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.706649    2564 policy_none.go:47] "Start"
Jul 01 20:46:40.745644 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.745485    2564 manager.go:337] "Starting Device Plugin manager"
Jul 01 20:46:40.757416 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:40.745680    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 01 20:46:40.757416 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.745696    2564 server.go:87] "Starting device plugin registration server"
Jul 01 20:46:40.757416 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.745948    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 01 20:46:40.757416 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.745960    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 01 20:46:40.757416 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.746137    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 01 20:46:40.757416 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.746710    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 01 20:46:40.757416 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.746729    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 01 20:46:40.757416 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:40.747313    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 01 20:46:40.757416 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:40.747389    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-198.ec2.internal\" not found"
Jul 01 20:46:40.757416 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.752237    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 01 20:46:40.757416 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.752270    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 01 20:46:40.757416 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.752289    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 01 20:46:40.757416 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:40.752322    2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 01 20:46:40.757416 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.754713    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 20:46:40.849946 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.849913    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:40.851187 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.851172    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-198.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:40.851257 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.851205    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-198.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:40.851257 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.851219    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-198.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:40.851257 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.851243    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-198.ec2.internal"
Jul 01 20:46:40.853507 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.853488    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-198.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-198.ec2.internal"]
Jul 01 20:46:40.853569 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.853558    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:40.854488 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.854468    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-198.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:40.854595 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.854504    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-198.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:40.854595 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.854520    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-198.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:40.855881 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.855855    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:40.855999 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.855982    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-198.ec2.internal"
Jul 01 20:46:40.856053 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.856016    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:40.856800 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.856771    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-198.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:40.856800 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.856771    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-198.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:40.856941 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.856804    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-198.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:40.856941 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.856818    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-198.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:40.856941 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.856827    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-198.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:40.856941 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.856839    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-198.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:40.858041 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.858026    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-198.ec2.internal"
Jul 01 20:46:40.858089 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.858059    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 20:46:40.858946 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.858919    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-198.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 20:46:40.858946 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.858947    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-198.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 20:46:40.859085 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.858962    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-198.ec2.internal" event="NodeHasSufficientPID"
Jul 01 20:46:40.860835 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.860819    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-198.ec2.internal"
Jul 01 20:46:40.860923 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:40.860843    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-198.ec2.internal\": node \"ip-10-0-131-198.ec2.internal\" not found"
Jul 01 20:46:40.887744 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:40.887723    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-198.ec2.internal\" not found" node="ip-10-0-131-198.ec2.internal"
Jul 01 20:46:40.887854 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:40.887840    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-198.ec2.internal\" not found"
Jul 01 20:46:40.891156 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:40.891140    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-198.ec2.internal\" not found" node="ip-10-0-131-198.ec2.internal"
Jul 01 20:46:40.986182 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.986148    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7bae69993a9f5e773a5205c5e78d2b81-config\") pod \"kube-apiserver-proxy-ip-10-0-131-198.ec2.internal\" (UID: \"7bae69993a9f5e773a5205c5e78d2b81\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-198.ec2.internal"
Jul 01 20:46:40.986348 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.986187    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/775ac52276a7c20fbf0b139e46e0a4bf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-198.ec2.internal\" (UID: \"775ac52276a7c20fbf0b139e46e0a4bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-198.ec2.internal"
Jul 01 20:46:40.986348 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:40.986220    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/775ac52276a7c20fbf0b139e46e0a4bf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-198.ec2.internal\" (UID: \"775ac52276a7c20fbf0b139e46e0a4bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-198.ec2.internal"
Jul 01 20:46:40.988360 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:40.988343    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-198.ec2.internal\" not found"
Jul 01 20:46:41.087421 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:41.087352    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7bae69993a9f5e773a5205c5e78d2b81-config\") pod \"kube-apiserver-proxy-ip-10-0-131-198.ec2.internal\" (UID: \"7bae69993a9f5e773a5205c5e78d2b81\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-198.ec2.internal"
Jul 01 20:46:41.087421 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:41.087294    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7bae69993a9f5e773a5205c5e78d2b81-config\") pod \"kube-apiserver-proxy-ip-10-0-131-198.ec2.internal\" (UID: \"7bae69993a9f5e773a5205c5e78d2b81\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-198.ec2.internal"
Jul 01 20:46:41.087421 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:41.087416    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/775ac52276a7c20fbf0b139e46e0a4bf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-198.ec2.internal\" (UID: \"775ac52276a7c20fbf0b139e46e0a4bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-198.ec2.internal"
Jul 01 20:46:41.087598 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:41.087434    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/775ac52276a7c20fbf0b139e46e0a4bf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-198.ec2.internal\" (UID: \"775ac52276a7c20fbf0b139e46e0a4bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-198.ec2.internal"
Jul 01 20:46:41.087598 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:41.087468    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/775ac52276a7c20fbf0b139e46e0a4bf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-198.ec2.internal\" (UID: \"775ac52276a7c20fbf0b139e46e0a4bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-198.ec2.internal"
Jul 01 20:46:41.087598 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:41.087477    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/775ac52276a7c20fbf0b139e46e0a4bf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-198.ec2.internal\" (UID: \"775ac52276a7c20fbf0b139e46e0a4bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-198.ec2.internal"
Jul 01 20:46:41.088424 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:41.088397    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-198.ec2.internal\" not found"
Jul 01 20:46:41.189055 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:41.189024    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-198.ec2.internal\" not found"
Jul 01 20:46:41.191358 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:41.191332    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-198.ec2.internal"
Jul 01 20:46:41.194743 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:41.194571    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-198.ec2.internal"
Jul 01 20:46:41.289578 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:41.289549    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-198.ec2.internal\" not found"
Jul 01 20:46:41.389813 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:41.389716    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-198.ec2.internal\" not found"
Jul 01 20:46:41.490116 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:41.490086    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-198.ec2.internal\" not found"
Jul 01 20:46:41.580743 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:41.580709    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 20:46:41.590706 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:41.590683    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-198.ec2.internal\" not found"
Jul 01 20:46:41.627113 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:41.627085    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 01 20:46:41.627537 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:41.627209    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 01 20:46:41.627537 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:41.627248    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 01 20:46:41.672379 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:41.672327    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 20:46:41.684032 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:41.684011    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-198.ec2.internal"
Jul 01 20:46:41.684148 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:41.684031    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 01 20:46:41.698165 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:41.698142    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 20:46:41.699106 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:41.699091    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-198.ec2.internal"
Jul 01 20:46:41.700612 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:41.700594    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 20:46:41.707396 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:41.707370    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-30 20:41:41 +0000 UTC" deadline="2028-02-23 22:56:41.07949211 +0000 UTC"
Jul 01 20:46:41.707396 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:41.707392    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14450h9m59.372102112s"
Jul 01 20:46:41.708646 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:41.708627    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 20:46:41.726749 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:41.726727    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-q4sxb"
Jul 01 20:46:41.735672 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:41.735654    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-q4sxb"
Jul 01 20:46:41.893321 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:41.893283    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bae69993a9f5e773a5205c5e78d2b81.slice/crio-7aa877fc9ae8ed055d30c8e5d6c38478f31f5c571c10418506970df693535f99 WatchSource:0}: Error finding container 7aa877fc9ae8ed055d30c8e5d6c38478f31f5c571c10418506970df693535f99: Status 404 returned error can't find the container with id 7aa877fc9ae8ed055d30c8e5d6c38478f31f5c571c10418506970df693535f99
Jul 01 20:46:41.893933 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:41.893896    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775ac52276a7c20fbf0b139e46e0a4bf.slice/crio-de3154144a05f798c239267c5007694447190f81aa4c631f9683d819c0169305 WatchSource:0}: Error finding container de3154144a05f798c239267c5007694447190f81aa4c631f9683d819c0169305: Status 404 returned error can't find the container with id de3154144a05f798c239267c5007694447190f81aa4c631f9683d819c0169305
Jul 01 20:46:41.898255 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:41.898239    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 20:46:42.091998 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.091939    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 20:46:42.663497 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.663463    2564 apiserver.go:52] "Watching apiserver"
Jul 01 20:46:42.675553 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.675339    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 01 20:46:42.675819 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.675790    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-vfs6k","kube-system/global-pull-secret-syncer-64pmt","kube-system/kube-apiserver-proxy-ip-10-0-131-198.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq","openshift-cluster-node-tuning-operator/tuned-ljqg5","openshift-dns/node-resolver-wmgbq","openshift-image-registry/node-ca-fr7j6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-198.ec2.internal","openshift-multus/multus-additional-cni-plugins-mfxlf","kube-system/konnectivity-agent-mkqr5","openshift-multus/multus-ftdgt","openshift-multus/network-metrics-daemon-9xhz4","openshift-network-diagnostics/network-check-target-9bl8d","openshift-network-operator/iptables-alerter-lmpqc"]
Jul 01 20:46:42.677360 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.677332    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fr7j6"
Jul 01 20:46:42.679256 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.679038    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:46:42.679256 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:42.679122    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bl8d" podUID="f4699b63-6ba0-4401-a43e-57c15fea25b3"
Jul 01 20:46:42.680986 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.680684    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lmpqc"
Jul 01 20:46:42.680986 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.680832    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4stpj\""
Jul 01 20:46:42.681234 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.681209    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 01 20:46:42.684225 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.683512    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 01 20:46:42.684225 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.683754    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq"
Jul 01 20:46:42.684225 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.684008    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 01 20:46:42.684225 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.683753    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 01 20:46:42.684501 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.684469    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-56hw5\""
Jul 01 20:46:42.684802 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.684689    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 01 20:46:42.685283 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.685263    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 01 20:46:42.686250 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.686062    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.687389 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.687300    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 01 20:46:42.687389 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.687316    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 01 20:46:42.687389 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.687303    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 01 20:46:42.687607 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.687483    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h4xlf\""
Jul 01 20:46:42.688215 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.687883    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wmgbq"
Jul 01 20:46:42.688762 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.688737    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 01 20:46:42.689779 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.689759    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mfxlf"
Jul 01 20:46:42.690800 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.690765    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 01 20:46:42.690965 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.690928    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zpvvw\""
Jul 01 20:46:42.691921 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.691610    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 01 20:46:42.691921 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.691673    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 01 20:46:42.691921 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.691778    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5drx9\""
Jul 01 20:46:42.691921 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.691810    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 20:46:42.693190 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.692264    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-c5knz\""
Jul 01 20:46:42.693190 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.692345    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.693190 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.693037    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 01 20:46:42.693408 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.693284    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 01 20:46:42.693548 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.693520    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 01 20:46:42.694306 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.693701    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:46:42.694306 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.693754    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 01 20:46:42.694306 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:42.693778    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:46:42.694306 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.694166    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 01 20:46:42.695047 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.695026    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mkqr5"
Jul 01 20:46:42.695141 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.695029    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 01 20:46:42.696960 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.696923    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 01 20:46:42.697328 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.697265    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mjhtd\""
Jul 01 20:46:42.697605 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.697586    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 01 20:46:42.697840 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.697813    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n8hk\" (UniqueName: \"kubernetes.io/projected/48225b8f-cd34-4fc1-b18e-7b219a899f86-kube-api-access-6n8hk\") pod \"iptables-alerter-lmpqc\" (UID: \"48225b8f-cd34-4fc1-b18e-7b219a899f86\") " pod="openshift-network-operator/iptables-alerter-lmpqc"
Jul 01 20:46:42.697931 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.697855    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ggnc\" (UniqueName: \"kubernetes.io/projected/47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2-kube-api-access-5ggnc\") pod \"aws-ebs-csi-driver-node-xsscq\" (UID: \"47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq"
Jul 01 20:46:42.697931 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.697882    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-etc-sysctl-conf\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.698076 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.697986    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb8effda-0dd6-437e-86f5-ac9c97c08193-tmp\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.698076 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.698014    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-etc-systemd\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.698076 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.698039    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d0e7e2a9-4cad-47c6-8a6b-46c461461be1-serviceca\") pod \"node-ca-fr7j6\" (UID: \"d0e7e2a9-4cad-47c6-8a6b-46c461461be1\") " pod="openshift-image-registry/node-ca-fr7j6"
Jul 01 20:46:42.698076 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.698064    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/48225b8f-cd34-4fc1-b18e-7b219a899f86-iptables-alerter-script\") pod \"iptables-alerter-lmpqc\" (UID: \"48225b8f-cd34-4fc1-b18e-7b219a899f86\") " pod="openshift-network-operator/iptables-alerter-lmpqc"
Jul 01 20:46:42.698299 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.698087    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/48225b8f-cd34-4fc1-b18e-7b219a899f86-host-slash\") pod \"iptables-alerter-lmpqc\" (UID: \"48225b8f-cd34-4fc1-b18e-7b219a899f86\") " pod="openshift-network-operator/iptables-alerter-lmpqc"
Jul 01 20:46:42.698299 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.698156    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2-sys-fs\") pod \"aws-ebs-csi-driver-node-xsscq\" (UID: \"47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq"
Jul 01 20:46:42.698299 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.698180    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-etc-sysctl-d\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.698299 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.698190    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:46:42.698299 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.698205    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-run\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.698299 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.698230    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2-registration-dir\") pod \"aws-ebs-csi-driver-node-xsscq\" (UID: \"47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq"
Jul 01 20:46:42.698299 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:42.698247    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:46:42.698299 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.698257    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0e7e2a9-4cad-47c6-8a6b-46c461461be1-host\") pod \"node-ca-fr7j6\" (UID: \"d0e7e2a9-4cad-47c6-8a6b-46c461461be1\") " pod="openshift-image-registry/node-ca-fr7j6"
Jul 01 20:46:42.698706 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.698345    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.698706 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.698342    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l29j\" (UniqueName: \"kubernetes.io/projected/d0e7e2a9-4cad-47c6-8a6b-46c461461be1-kube-api-access-6l29j\") pod \"node-ca-fr7j6\" (UID: \"d0e7e2a9-4cad-47c6-8a6b-46c461461be1\") " pod="openshift-image-registry/node-ca-fr7j6"
Jul 01 20:46:42.698706 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.698381    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-etc-modprobe-d\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.698706 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.698406    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crpxq\" (UniqueName: \"kubernetes.io/projected/eb8effda-0dd6-437e-86f5-ac9c97c08193-kube-api-access-crpxq\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.698706 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.698437    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xsscq\" (UID: \"47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq"
Jul 01 20:46:42.698706 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.698519    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-lib-modules\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.698706 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.698542    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-host\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.698706 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.698583    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2-socket-dir\") pod \"aws-ebs-csi-driver-node-xsscq\" (UID: \"47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq"
Jul 01 20:46:42.698706 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.698614    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2rm6\" (UniqueName: \"kubernetes.io/projected/f4699b63-6ba0-4401-a43e-57c15fea25b3-kube-api-access-p2rm6\") pod \"network-check-target-9bl8d\" (UID: \"f4699b63-6ba0-4401-a43e-57c15fea25b3\") " pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:46:42.698706 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.698681    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2-device-dir\") pod \"aws-ebs-csi-driver-node-xsscq\" (UID: \"47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq"
Jul 01 20:46:42.698706 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.698706    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2-etc-selinux\") pod \"aws-ebs-csi-driver-node-xsscq\" (UID: \"47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq"
Jul 01 20:46:42.699289 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.698730    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-etc-kubernetes\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.699289 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.698755    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-sys\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.699289 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.698779    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-var-lib-kubelet\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.699289 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.698806    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-etc-sysconfig\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.699289 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.698873    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb8effda-0dd6-437e-86f5-ac9c97c08193-etc-tuned\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.699289 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.699074    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 01 20:46:42.699289 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.699128    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-svsdd\""
Jul 01 20:46:42.699749 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.699307    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 01 20:46:42.699749 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.699534    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 01 20:46:42.700098 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.699954    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 01 20:46:42.700098 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.699981    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 01 20:46:42.702205 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.702183    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 01 20:46:42.702855 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.702837    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ts7ck\""
Jul 01 20:46:42.737478 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.737379    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 20:41:42 +0000 UTC" deadline="2027-12-28 18:54:51.230718253 +0000 UTC"
Jul 01 20:46:42.737478 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.737408    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13078h8m8.493315109s"
Jul 01 20:46:42.760063 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.759993    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-198.ec2.internal" event={"ID":"775ac52276a7c20fbf0b139e46e0a4bf","Type":"ContainerStarted","Data":"de3154144a05f798c239267c5007694447190f81aa4c631f9683d819c0169305"}
Jul 01 20:46:42.761984 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.761955    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-198.ec2.internal" event={"ID":"7bae69993a9f5e773a5205c5e78d2b81","Type":"ContainerStarted","Data":"7aa877fc9ae8ed055d30c8e5d6c38478f31f5c571c10418506970df693535f99"}
Jul 01 20:46:42.785294 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.785273    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 01 20:46:42.799411 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.799373    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-etc-sysctl-conf\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.799411 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.799408    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb8effda-0dd6-437e-86f5-ac9c97c08193-tmp\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.799592 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.799442    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb41c3f0-f0dc-4340-b524-bb23e5426370-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mfxlf\" (UID: \"fb41c3f0-f0dc-4340-b524-bb23e5426370\") " pod="openshift-multus/multus-additional-cni-plugins-mfxlf"
Jul 01 20:46:42.799592 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.799467    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqvds\" (UniqueName: \"kubernetes.io/projected/fb41c3f0-f0dc-4340-b524-bb23e5426370-kube-api-access-jqvds\") pod \"multus-additional-cni-plugins-mfxlf\" (UID: \"fb41c3f0-f0dc-4340-b524-bb23e5426370\") " pod="openshift-multus/multus-additional-cni-plugins-mfxlf"
Jul 01 20:46:42.799592 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.799489    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-host-var-lib-kubelet\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.799592 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.799516    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d0e7e2a9-4cad-47c6-8a6b-46c461461be1-serviceca\") pod \"node-ca-fr7j6\" (UID: \"d0e7e2a9-4cad-47c6-8a6b-46c461461be1\") " pod="openshift-image-registry/node-ca-fr7j6"
Jul 01 20:46:42.799592 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.799551    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/48225b8f-cd34-4fc1-b18e-7b219a899f86-iptables-alerter-script\") pod \"iptables-alerter-lmpqc\" (UID: \"48225b8f-cd34-4fc1-b18e-7b219a899f86\") " pod="openshift-network-operator/iptables-alerter-lmpqc"
Jul 01 20:46:42.799592 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.799578    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/48225b8f-cd34-4fc1-b18e-7b219a899f86-host-slash\") pod \"iptables-alerter-lmpqc\" (UID: \"48225b8f-cd34-4fc1-b18e-7b219a899f86\") " pod="openshift-network-operator/iptables-alerter-lmpqc"
Jul 01 20:46:42.799914 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.799604    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-etc-sysctl-conf\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.799914 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.799603    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2-sys-fs\") pod \"aws-ebs-csi-driver-node-xsscq\" (UID: \"47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq"
Jul 01 20:46:42.799914 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.799669    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-etc-sysctl-d\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.799914 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.799674    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2-sys-fs\") pod \"aws-ebs-csi-driver-node-xsscq\" (UID: \"47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq"
Jul 01 20:46:42.799914 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.799700    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-system-cni-dir\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.799914 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.799762    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-multus-conf-dir\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.799914 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.799790    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-run\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.799914 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.799793    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/48225b8f-cd34-4fc1-b18e-7b219a899f86-host-slash\") pod \"iptables-alerter-lmpqc\" (UID: \"48225b8f-cd34-4fc1-b18e-7b219a899f86\") " pod="openshift-network-operator/iptables-alerter-lmpqc"
Jul 01 20:46:42.799914 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.799808    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fb41c3f0-f0dc-4340-b524-bb23e5426370-os-release\") pod \"multus-additional-cni-plugins-mfxlf\" (UID: \"fb41c3f0-f0dc-4340-b524-bb23e5426370\") " pod="openshift-multus/multus-additional-cni-plugins-mfxlf"
Jul 01 20:46:42.799914 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.799842    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-cnibin\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.799914 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.799868    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-cni-binary-copy\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.799914 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.799893    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-host-run-multus-certs\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.799914 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.799921    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-run\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.799914 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.799923    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2-registration-dir\") pod \"aws-ebs-csi-driver-node-xsscq\" (UID: \"47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq"
Jul 01 20:46:42.800609 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.799963    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0e7e2a9-4cad-47c6-8a6b-46c461461be1-host\") pod \"node-ca-fr7j6\" (UID: \"d0e7e2a9-4cad-47c6-8a6b-46c461461be1\") " pod="openshift-image-registry/node-ca-fr7j6"
Jul 01 20:46:42.800609 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.799991    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l29j\" (UniqueName: \"kubernetes.io/projected/d0e7e2a9-4cad-47c6-8a6b-46c461461be1-kube-api-access-6l29j\") pod \"node-ca-fr7j6\" (UID: \"d0e7e2a9-4cad-47c6-8a6b-46c461461be1\") " pod="openshift-image-registry/node-ca-fr7j6"
Jul 01 20:46:42.800609 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800018    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-ovnkube-config\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.800609 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800024    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0e7e2a9-4cad-47c6-8a6b-46c461461be1-host\") pod \"node-ca-fr7j6\" (UID: \"d0e7e2a9-4cad-47c6-8a6b-46c461461be1\") " pod="openshift-image-registry/node-ca-fr7j6"
Jul 01 20:46:42.800609 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800009    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 01 20:46:42.800609 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800047    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xsscq\" (UID: \"47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq"
Jul 01 20:46:42.800609 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800198    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-host-run-netns\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.800609 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800225    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-host-cni-bin\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.800609 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800249    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb41c3f0-f0dc-4340-b524-bb23e5426370-cnibin\") pod \"multus-additional-cni-plugins-mfxlf\" (UID: \"fb41c3f0-f0dc-4340-b524-bb23e5426370\") " pod="openshift-multus/multus-additional-cni-plugins-mfxlf"
Jul 01 20:46:42.800609 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800280    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-lib-modules\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.800609 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800299    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xsscq\" (UID: \"47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq"
Jul 01 20:46:42.800609 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800343    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2rm6\" (UniqueName: \"kubernetes.io/projected/f4699b63-6ba0-4401-a43e-57c15fea25b3-kube-api-access-p2rm6\") pod \"network-check-target-9bl8d\" (UID: \"f4699b63-6ba0-4401-a43e-57c15fea25b3\") " pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:46:42.800609 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800353    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2-registration-dir\") pod \"aws-ebs-csi-driver-node-xsscq\" (UID: \"47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq"
Jul 01 20:46:42.800609 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800373    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-host-run-ovn-kubernetes\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.800609 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800410    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/48225b8f-cd34-4fc1-b18e-7b219a899f86-iptables-alerter-script\") pod \"iptables-alerter-lmpqc\" (UID: \"48225b8f-cd34-4fc1-b18e-7b219a899f86\") " pod="openshift-network-operator/iptables-alerter-lmpqc"
Jul 01 20:46:42.800609 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800476    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-ovn-node-metrics-cert\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.800609 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800503    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c606e570-2fd6-43cb-9a84-9cdeab442815-metrics-certs\") pod \"network-metrics-daemon-9xhz4\" (UID: \"c606e570-2fd6-43cb-9a84-9cdeab442815\") " pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:46:42.801634 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800530    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-etc-kubernetes\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.801634 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800554    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-sys\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.801634 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800580    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.801634 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800588    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-lib-modules\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.801634 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800607    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-ovnkube-script-lib\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.801634 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800641    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-etc-kubernetes\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.801634 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800635    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-dbus\") pod \"global-pull-secret-syncer-64pmt\" (UID: \"ed4e44b6-7fbe-4e18-974e-edbc64ce0398\") " pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:46:42.801634 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800677    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54d8b430-fceb-4c6b-9f65-233650a04f7f-tmp-dir\") pod \"node-resolver-wmgbq\" (UID: \"54d8b430-fceb-4c6b-9f65-233650a04f7f\") " pod="openshift-dns/node-resolver-wmgbq"
Jul 01 20:46:42.801634 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800738    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-sys\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.801634 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800739    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtf2j\" (UniqueName: \"kubernetes.io/projected/54d8b430-fceb-4c6b-9f65-233650a04f7f-kube-api-access-mtf2j\") pod \"node-resolver-wmgbq\" (UID: \"54d8b430-fceb-4c6b-9f65-233650a04f7f\") " pod="openshift-dns/node-resolver-wmgbq"
Jul 01 20:46:42.801634 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800774    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-host-run-k8s-cni-cncf-io\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.801634 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800811    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb8effda-0dd6-437e-86f5-ac9c97c08193-etc-tuned\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.801634 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800827    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-etc-sysctl-d\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.801634 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800875    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-etc-openvswitch\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.801634 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800913    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-hostroot\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.801634 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800956    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pgwg\" (UniqueName: \"kubernetes.io/projected/c606e570-2fd6-43cb-9a84-9cdeab442815-kube-api-access-9pgwg\") pod \"network-metrics-daemon-9xhz4\" (UID: \"c606e570-2fd6-43cb-9a84-9cdeab442815\") " pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:46:42.801634 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.800997    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-host-kubelet\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.802558 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801020    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-run-openvswitch\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.802558 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801046    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-kubelet-config\") pod \"global-pull-secret-syncer-64pmt\" (UID: \"ed4e44b6-7fbe-4e18-974e-edbc64ce0398\") " pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:46:42.802558 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801084    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-etc-systemd\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.802558 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801112    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckctq\" (UniqueName: \"kubernetes.io/projected/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-kube-api-access-ckctq\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.802558 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801139    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-host-var-lib-cni-bin\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.802558 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801164    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-etc-kubernetes\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.802558 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801204    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-node-log\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.802558 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801230    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb41c3f0-f0dc-4340-b524-bb23e5426370-system-cni-dir\") pod \"multus-additional-cni-plugins-mfxlf\" (UID: \"fb41c3f0-f0dc-4340-b524-bb23e5426370\") " pod="openshift-multus/multus-additional-cni-plugins-mfxlf"
Jul 01 20:46:42.802558 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801237    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-etc-systemd\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.802558 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801273    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-etc-modprobe-d\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.802558 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801310    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d0e7e2a9-4cad-47c6-8a6b-46c461461be1-serviceca\") pod \"node-ca-fr7j6\" (UID: \"d0e7e2a9-4cad-47c6-8a6b-46c461461be1\") " pod="openshift-image-registry/node-ca-fr7j6"
Jul 01 20:46:42.802558 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801357    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-etc-modprobe-d\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.802558 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801386    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crpxq\" (UniqueName: \"kubernetes.io/projected/eb8effda-0dd6-437e-86f5-ac9c97c08193-kube-api-access-crpxq\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.802558 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801424    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-host-cni-netd\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.802558 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801471    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb41c3f0-f0dc-4340-b524-bb23e5426370-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mfxlf\" (UID: \"fb41c3f0-f0dc-4340-b524-bb23e5426370\") " pod="openshift-multus/multus-additional-cni-plugins-mfxlf"
Jul 01 20:46:42.802558 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801498    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-host-run-netns\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.802558 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801552    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-multus-daemon-config\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.803506 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801578    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-original-pull-secret\") pod \"global-pull-secret-syncer-64pmt\" (UID: \"ed4e44b6-7fbe-4e18-974e-edbc64ce0398\") " pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:46:42.803506 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801604    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8634b68f-65d5-4747-b19c-2b2e1d44ba76-konnectivity-ca\") pod \"konnectivity-agent-mkqr5\" (UID: \"8634b68f-65d5-4747-b19c-2b2e1d44ba76\") " pod="kube-system/konnectivity-agent-mkqr5"
Jul 01 20:46:42.803506 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801630    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb41c3f0-f0dc-4340-b524-bb23e5426370-cni-binary-copy\") pod \"multus-additional-cni-plugins-mfxlf\" (UID: \"fb41c3f0-f0dc-4340-b524-bb23e5426370\") " pod="openshift-multus/multus-additional-cni-plugins-mfxlf"
Jul 01 20:46:42.803506 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801655    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-multus-cni-dir\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.803506 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801697    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7ww7\" (UniqueName: \"kubernetes.io/projected/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-kube-api-access-w7ww7\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.803506 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801729    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-host\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.803506 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801763    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2-socket-dir\") pod \"aws-ebs-csi-driver-node-xsscq\" (UID: \"47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq"
Jul 01 20:46:42.803506 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801798    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-run-systemd\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.803506 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801809    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-host\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.803506 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801822    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-run-ovn\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.803506 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801852    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-env-overrides\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.803506 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801896    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb41c3f0-f0dc-4340-b524-bb23e5426370-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mfxlf\" (UID: \"fb41c3f0-f0dc-4340-b524-bb23e5426370\") " pod="openshift-multus/multus-additional-cni-plugins-mfxlf"
Jul 01 20:46:42.803506 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801925    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-os-release\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.803506 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801948    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2-socket-dir\") pod \"aws-ebs-csi-driver-node-xsscq\" (UID: \"47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq"
Jul 01 20:46:42.803506 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801949    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-multus-socket-dir-parent\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.803506 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.801991    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2-device-dir\") pod \"aws-ebs-csi-driver-node-xsscq\" (UID: \"47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq"
Jul 01 20:46:42.803506 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.802015    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2-etc-selinux\") pod \"aws-ebs-csi-driver-node-xsscq\" (UID: \"47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq"
Jul 01 20:46:42.804449 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.802032    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-var-lib-kubelet\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.804449 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.802070    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-systemd-units\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.804449 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.802077    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-var-lib-kubelet\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.804449 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.802082    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2-device-dir\") pod \"aws-ebs-csi-driver-node-xsscq\" (UID: \"47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq"
Jul 01 20:46:42.804449 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.802100    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-var-lib-openvswitch\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.804449 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.802132    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2-etc-selinux\") pod \"aws-ebs-csi-driver-node-xsscq\" (UID: \"47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq"
Jul 01 20:46:42.804449 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.802133    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-log-socket\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.804449 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.802170    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/54d8b430-fceb-4c6b-9f65-233650a04f7f-hosts-file\") pod \"node-resolver-wmgbq\" (UID: \"54d8b430-fceb-4c6b-9f65-233650a04f7f\") " pod="openshift-dns/node-resolver-wmgbq"
Jul 01 20:46:42.804449 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.802195    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-host-var-lib-cni-multus\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.804449 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.802219    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8634b68f-65d5-4747-b19c-2b2e1d44ba76-agent-certs\") pod \"konnectivity-agent-mkqr5\" (UID: \"8634b68f-65d5-4747-b19c-2b2e1d44ba76\") " pod="kube-system/konnectivity-agent-mkqr5"
Jul 01 20:46:42.804449 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.802246    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-etc-sysconfig\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.804449 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.802270    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-host-slash\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.804449 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.802298    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n8hk\" (UniqueName: \"kubernetes.io/projected/48225b8f-cd34-4fc1-b18e-7b219a899f86-kube-api-access-6n8hk\") pod \"iptables-alerter-lmpqc\" (UID: \"48225b8f-cd34-4fc1-b18e-7b219a899f86\") " pod="openshift-network-operator/iptables-alerter-lmpqc"
Jul 01 20:46:42.804449 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.802308    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb8effda-0dd6-437e-86f5-ac9c97c08193-etc-sysconfig\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.804449 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.802323    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ggnc\" (UniqueName: \"kubernetes.io/projected/47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2-kube-api-access-5ggnc\") pod \"aws-ebs-csi-driver-node-xsscq\" (UID: \"47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq"
Jul 01 20:46:42.804449 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.804443    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb8effda-0dd6-437e-86f5-ac9c97c08193-etc-tuned\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.805289 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.804527    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb8effda-0dd6-437e-86f5-ac9c97c08193-tmp\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.812783 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.812759    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l29j\" (UniqueName: \"kubernetes.io/projected/d0e7e2a9-4cad-47c6-8a6b-46c461461be1-kube-api-access-6l29j\") pod \"node-ca-fr7j6\" (UID: \"d0e7e2a9-4cad-47c6-8a6b-46c461461be1\") " pod="openshift-image-registry/node-ca-fr7j6"
Jul 01 20:46:42.820013 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:42.819949    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:46:42.820013 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:42.819973    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:46:42.820013 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:42.819987    2564 projected.go:196] Error preparing data for projected volume kube-api-access-p2rm6 for pod openshift-network-diagnostics/network-check-target-9bl8d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:42.820256 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:42.820053    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4699b63-6ba0-4401-a43e-57c15fea25b3-kube-api-access-p2rm6 podName:f4699b63-6ba0-4401-a43e-57c15fea25b3 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:43.32003004 +0000 UTC m=+2.996045599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-p2rm6" (UniqueName: "kubernetes.io/projected/f4699b63-6ba0-4401-a43e-57c15fea25b3-kube-api-access-p2rm6") pod "network-check-target-9bl8d" (UID: "f4699b63-6ba0-4401-a43e-57c15fea25b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:42.821847 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.821806    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ggnc\" (UniqueName: \"kubernetes.io/projected/47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2-kube-api-access-5ggnc\") pod \"aws-ebs-csi-driver-node-xsscq\" (UID: \"47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq"
Jul 01 20:46:42.823175 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.822993    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n8hk\" (UniqueName: \"kubernetes.io/projected/48225b8f-cd34-4fc1-b18e-7b219a899f86-kube-api-access-6n8hk\") pod \"iptables-alerter-lmpqc\" (UID: \"48225b8f-cd34-4fc1-b18e-7b219a899f86\") " pod="openshift-network-operator/iptables-alerter-lmpqc"
Jul 01 20:46:42.823175 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.823058    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crpxq\" (UniqueName: \"kubernetes.io/projected/eb8effda-0dd6-437e-86f5-ac9c97c08193-kube-api-access-crpxq\") pod \"tuned-ljqg5\" (UID: \"eb8effda-0dd6-437e-86f5-ac9c97c08193\") " pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:42.903144 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903110    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb41c3f0-f0dc-4340-b524-bb23e5426370-cnibin\") pod \"multus-additional-cni-plugins-mfxlf\" (UID: \"fb41c3f0-f0dc-4340-b524-bb23e5426370\") " pod="openshift-multus/multus-additional-cni-plugins-mfxlf"
Jul 01 20:46:42.903340 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903166    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-host-run-ovn-kubernetes\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.903340 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903190    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-ovn-node-metrics-cert\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.903340 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903214    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c606e570-2fd6-43cb-9a84-9cdeab442815-metrics-certs\") pod \"network-metrics-daemon-9xhz4\" (UID: \"c606e570-2fd6-43cb-9a84-9cdeab442815\") " pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:46:42.903340 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903239    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.903340 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903248    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-host-run-ovn-kubernetes\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.903340 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903265    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-ovnkube-script-lib\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.903340 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903289    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-dbus\") pod \"global-pull-secret-syncer-64pmt\" (UID: \"ed4e44b6-7fbe-4e18-974e-edbc64ce0398\") " pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:46:42.903340 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903295    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb41c3f0-f0dc-4340-b524-bb23e5426370-cnibin\") pod \"multus-additional-cni-plugins-mfxlf\" (UID: \"fb41c3f0-f0dc-4340-b524-bb23e5426370\") " pod="openshift-multus/multus-additional-cni-plugins-mfxlf"
Jul 01 20:46:42.903340 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903329    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.903830 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903335    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54d8b430-fceb-4c6b-9f65-233650a04f7f-tmp-dir\") pod \"node-resolver-wmgbq\" (UID: \"54d8b430-fceb-4c6b-9f65-233650a04f7f\") " pod="openshift-dns/node-resolver-wmgbq"
Jul 01 20:46:42.903830 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903396    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtf2j\" (UniqueName: \"kubernetes.io/projected/54d8b430-fceb-4c6b-9f65-233650a04f7f-kube-api-access-mtf2j\") pod \"node-resolver-wmgbq\" (UID: \"54d8b430-fceb-4c6b-9f65-233650a04f7f\") " pod="openshift-dns/node-resolver-wmgbq"
Jul 01 20:46:42.903830 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903426    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-host-run-k8s-cni-cncf-io\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.903830 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:42.903440    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:42.903830 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903455    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-etc-openvswitch\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.903830 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903473    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-dbus\") pod \"global-pull-secret-syncer-64pmt\" (UID: \"ed4e44b6-7fbe-4e18-974e-edbc64ce0398\") " pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:46:42.903830 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903487    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-hostroot\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.903830 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:42.903510    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c606e570-2fd6-43cb-9a84-9cdeab442815-metrics-certs podName:c606e570-2fd6-43cb-9a84-9cdeab442815 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:43.403489517 +0000 UTC m=+3.079505086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c606e570-2fd6-43cb-9a84-9cdeab442815-metrics-certs") pod "network-metrics-daemon-9xhz4" (UID: "c606e570-2fd6-43cb-9a84-9cdeab442815") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:42.903830 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903523    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-host-run-k8s-cni-cncf-io\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.903830 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903530    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-etc-openvswitch\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.903830 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903578    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-hostroot\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.903830 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903620    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pgwg\" (UniqueName: \"kubernetes.io/projected/c606e570-2fd6-43cb-9a84-9cdeab442815-kube-api-access-9pgwg\") pod \"network-metrics-daemon-9xhz4\" (UID: \"c606e570-2fd6-43cb-9a84-9cdeab442815\") " pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:46:42.903830 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903629    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54d8b430-fceb-4c6b-9f65-233650a04f7f-tmp-dir\") pod \"node-resolver-wmgbq\" (UID: \"54d8b430-fceb-4c6b-9f65-233650a04f7f\") " pod="openshift-dns/node-resolver-wmgbq"
Jul 01 20:46:42.903830 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903648    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-host-kubelet\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.903830 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903673    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-run-openvswitch\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.903830 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903699    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-kubelet-config\") pod \"global-pull-secret-syncer-64pmt\" (UID: \"ed4e44b6-7fbe-4e18-974e-edbc64ce0398\") " pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:46:42.903830 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903718    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-host-kubelet\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.904763 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903728    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckctq\" (UniqueName: \"kubernetes.io/projected/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-kube-api-access-ckctq\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.904763 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903772    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-run-openvswitch\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.904763 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903775    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-kubelet-config\") pod \"global-pull-secret-syncer-64pmt\" (UID: \"ed4e44b6-7fbe-4e18-974e-edbc64ce0398\") " pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:46:42.904763 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903796    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-host-var-lib-cni-bin\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.904763 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903823    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-etc-kubernetes\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.904763 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903848    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-node-log\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.904763 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903856    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-host-var-lib-cni-bin\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.904763 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903872    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb41c3f0-f0dc-4340-b524-bb23e5426370-system-cni-dir\") pod \"multus-additional-cni-plugins-mfxlf\" (UID: \"fb41c3f0-f0dc-4340-b524-bb23e5426370\") " pod="openshift-multus/multus-additional-cni-plugins-mfxlf"
Jul 01 20:46:42.904763 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903895    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-etc-kubernetes\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.904763 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903905    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-node-log\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.904763 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903903    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-host-cni-netd\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.904763 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903952    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb41c3f0-f0dc-4340-b524-bb23e5426370-system-cni-dir\") pod \"multus-additional-cni-plugins-mfxlf\" (UID: \"fb41c3f0-f0dc-4340-b524-bb23e5426370\") " pod="openshift-multus/multus-additional-cni-plugins-mfxlf"
Jul 01 20:46:42.904763 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903956    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb41c3f0-f0dc-4340-b524-bb23e5426370-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mfxlf\" (UID: \"fb41c3f0-f0dc-4340-b524-bb23e5426370\") " pod="openshift-multus/multus-additional-cni-plugins-mfxlf"
Jul 01 20:46:42.904763 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903992    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-host-run-netns\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.904763 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.903998    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-host-cni-netd\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.904763 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904019    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-multus-daemon-config\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.904763 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904045    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-original-pull-secret\") pod \"global-pull-secret-syncer-64pmt\" (UID: \"ed4e44b6-7fbe-4e18-974e-edbc64ce0398\") " pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:46:42.905726 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904044    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-host-run-netns\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.905726 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904073    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8634b68f-65d5-4747-b19c-2b2e1d44ba76-konnectivity-ca\") pod \"konnectivity-agent-mkqr5\" (UID: \"8634b68f-65d5-4747-b19c-2b2e1d44ba76\") " pod="kube-system/konnectivity-agent-mkqr5"
Jul 01 20:46:42.905726 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904104    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb41c3f0-f0dc-4340-b524-bb23e5426370-cni-binary-copy\") pod \"multus-additional-cni-plugins-mfxlf\" (UID: \"fb41c3f0-f0dc-4340-b524-bb23e5426370\") " pod="openshift-multus/multus-additional-cni-plugins-mfxlf"
Jul 01 20:46:42.905726 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904129    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-multus-cni-dir\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.905726 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904154    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7ww7\" (UniqueName: \"kubernetes.io/projected/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-kube-api-access-w7ww7\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.905726 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904181    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-run-systemd\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.905726 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904205    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-run-ovn\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.905726 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904237    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-env-overrides\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.905726 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904264    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb41c3f0-f0dc-4340-b524-bb23e5426370-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mfxlf\" (UID: \"fb41c3f0-f0dc-4340-b524-bb23e5426370\") " pod="openshift-multus/multus-additional-cni-plugins-mfxlf"
Jul 01 20:46:42.905726 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904288    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-os-release\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.905726 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904332    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-multus-socket-dir-parent\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.905726 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904365    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-systemd-units\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.905726 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904389    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-var-lib-openvswitch\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.905726 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904421    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-log-socket\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.905726 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904446    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/54d8b430-fceb-4c6b-9f65-233650a04f7f-hosts-file\") pod \"node-resolver-wmgbq\" (UID: \"54d8b430-fceb-4c6b-9f65-233650a04f7f\") " pod="openshift-dns/node-resolver-wmgbq"
Jul 01 20:46:42.905726 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904472    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-host-var-lib-cni-multus\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.905726 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904498    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8634b68f-65d5-4747-b19c-2b2e1d44ba76-agent-certs\") pod \"konnectivity-agent-mkqr5\" (UID: \"8634b68f-65d5-4747-b19c-2b2e1d44ba76\") " pod="kube-system/konnectivity-agent-mkqr5"
Jul 01 20:46:42.905726 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904526    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-host-slash\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.906717 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904543    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-ovnkube-script-lib\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.906717 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904560    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb41c3f0-f0dc-4340-b524-bb23e5426370-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mfxlf\" (UID: \"fb41c3f0-f0dc-4340-b524-bb23e5426370\") " pod="openshift-multus/multus-additional-cni-plugins-mfxlf"
Jul 01 20:46:42.906717 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904592    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb41c3f0-f0dc-4340-b524-bb23e5426370-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mfxlf\" (UID: \"fb41c3f0-f0dc-4340-b524-bb23e5426370\") " pod="openshift-multus/multus-additional-cni-plugins-mfxlf"
Jul 01 20:46:42.906717 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904606    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvds\" (UniqueName: \"kubernetes.io/projected/fb41c3f0-f0dc-4340-b524-bb23e5426370-kube-api-access-jqvds\") pod \"multus-additional-cni-plugins-mfxlf\" (UID: \"fb41c3f0-f0dc-4340-b524-bb23e5426370\") " pod="openshift-multus/multus-additional-cni-plugins-mfxlf"
Jul 01 20:46:42.906717 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904616    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-multus-daemon-config\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.906717 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904652    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-host-var-lib-kubelet\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.906717 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904694    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-system-cni-dir\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.906717 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904721    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-multus-conf-dir\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.906717 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904752    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fb41c3f0-f0dc-4340-b524-bb23e5426370-os-release\") pod \"multus-additional-cni-plugins-mfxlf\" (UID: \"fb41c3f0-f0dc-4340-b524-bb23e5426370\") " pod="openshift-multus/multus-additional-cni-plugins-mfxlf"
Jul 01 20:46:42.906717 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904763    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb41c3f0-f0dc-4340-b524-bb23e5426370-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mfxlf\" (UID: \"fb41c3f0-f0dc-4340-b524-bb23e5426370\") " pod="openshift-multus/multus-additional-cni-plugins-mfxlf"
Jul 01 20:46:42.906717 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904777    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-cnibin\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.906717 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904816    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-cni-binary-copy\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.906717 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904844    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-run-systemd\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.906717 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904847    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-host-run-multus-certs\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.906717 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904895    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-host-run-multus-certs\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.906717 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904893    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-ovnkube-config\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.906717 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904952    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-host-run-netns\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.907610 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.904979    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-host-cni-bin\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.907610 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.905042    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb41c3f0-f0dc-4340-b524-bb23e5426370-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mfxlf\" (UID: \"fb41c3f0-f0dc-4340-b524-bb23e5426370\") " pod="openshift-multus/multus-additional-cni-plugins-mfxlf"
Jul 01 20:46:42.907610 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.905056    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-host-cni-bin\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.907610 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:42.905129    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:42.907610 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.905166    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-systemd-units\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.907610 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:42.905210    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-original-pull-secret podName:ed4e44b6-7fbe-4e18-974e-edbc64ce0398 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:43.40516382 +0000 UTC m=+3.081179385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-original-pull-secret") pod "global-pull-secret-syncer-64pmt" (UID: "ed4e44b6-7fbe-4e18-974e-edbc64ce0398") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:42.907610 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.905213    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-var-lib-openvswitch\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.907610 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.905262    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-log-socket\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.907610 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.905268    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-cnibin\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.907610 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.905342    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/54d8b430-fceb-4c6b-9f65-233650a04f7f-hosts-file\") pod \"node-resolver-wmgbq\" (UID: \"54d8b430-fceb-4c6b-9f65-233650a04f7f\") " pod="openshift-dns/node-resolver-wmgbq"
Jul 01 20:46:42.907610 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.905372    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-ovnkube-config\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.907610 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.905398    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8634b68f-65d5-4747-b19c-2b2e1d44ba76-konnectivity-ca\") pod \"konnectivity-agent-mkqr5\" (UID: \"8634b68f-65d5-4747-b19c-2b2e1d44ba76\") " pod="kube-system/konnectivity-agent-mkqr5"
Jul 01 20:46:42.907610 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.905402    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fb41c3f0-f0dc-4340-b524-bb23e5426370-os-release\") pod \"multus-additional-cni-plugins-mfxlf\" (UID: \"fb41c3f0-f0dc-4340-b524-bb23e5426370\") " pod="openshift-multus/multus-additional-cni-plugins-mfxlf"
Jul 01 20:46:42.907610 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.905425    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-run-ovn\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.907610 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.905444    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-os-release\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.907610 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.905483    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-host-var-lib-cni-multus\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.907610 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.905840    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-cni-binary-copy\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.907610 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.905128    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-multus-socket-dir-parent\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.908590 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.905890    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-system-cni-dir\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.908590 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.905922    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-host-run-netns\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.908590 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.905923    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb41c3f0-f0dc-4340-b524-bb23e5426370-cni-binary-copy\") pod \"multus-additional-cni-plugins-mfxlf\" (UID: \"fb41c3f0-f0dc-4340-b524-bb23e5426370\") " pod="openshift-multus/multus-additional-cni-plugins-mfxlf"
Jul 01 20:46:42.908590 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.905978    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-env-overrides\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.908590 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.905989    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-multus-cni-dir\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.908590 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.905993    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-host-var-lib-kubelet\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.908590 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.906030    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-multus-conf-dir\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.908590 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.906037    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-host-slash\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.908590 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.906268    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-ovn-node-metrics-cert\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.908590 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.908549    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8634b68f-65d5-4747-b19c-2b2e1d44ba76-agent-certs\") pod \"konnectivity-agent-mkqr5\" (UID: \"8634b68f-65d5-4747-b19c-2b2e1d44ba76\") " pod="kube-system/konnectivity-agent-mkqr5"
Jul 01 20:46:42.915886 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.915825    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqvds\" (UniqueName: \"kubernetes.io/projected/fb41c3f0-f0dc-4340-b524-bb23e5426370-kube-api-access-jqvds\") pod \"multus-additional-cni-plugins-mfxlf\" (UID: \"fb41c3f0-f0dc-4340-b524-bb23e5426370\") " pod="openshift-multus/multus-additional-cni-plugins-mfxlf"
Jul 01 20:46:42.916488 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.916408    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pgwg\" (UniqueName: \"kubernetes.io/projected/c606e570-2fd6-43cb-9a84-9cdeab442815-kube-api-access-9pgwg\") pod \"network-metrics-daemon-9xhz4\" (UID: \"c606e570-2fd6-43cb-9a84-9cdeab442815\") " pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:46:42.916805 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.916747    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckctq\" (UniqueName: \"kubernetes.io/projected/ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e-kube-api-access-ckctq\") pod \"ovnkube-node-vfs6k\" (UID: \"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e\") " pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:42.917108 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.917084    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtf2j\" (UniqueName: \"kubernetes.io/projected/54d8b430-fceb-4c6b-9f65-233650a04f7f-kube-api-access-mtf2j\") pod \"node-resolver-wmgbq\" (UID: \"54d8b430-fceb-4c6b-9f65-233650a04f7f\") " pod="openshift-dns/node-resolver-wmgbq"
Jul 01 20:46:42.917878 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.917836    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7ww7\" (UniqueName: \"kubernetes.io/projected/c2ef23e3-d38f-47a1-94ea-aa4f8fc40725-kube-api-access-w7ww7\") pod \"multus-ftdgt\" (UID: \"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725\") " pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:42.995517 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:42.995482    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fr7j6"
Jul 01 20:46:43.005339 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:43.005313    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lmpqc"
Jul 01 20:46:43.015445 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:43.015419    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq"
Jul 01 20:46:43.023584 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:43.023562    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ljqg5"
Jul 01 20:46:43.032584 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:43.032558    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wmgbq"
Jul 01 20:46:43.040327 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:43.040310    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mfxlf"
Jul 01 20:46:43.048205 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:43.048182    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:46:43.057168 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:43.057146    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mkqr5"
Jul 01 20:46:43.063971 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:43.063949    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ftdgt"
Jul 01 20:46:43.408918 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:43.408806    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-original-pull-secret\") pod \"global-pull-secret-syncer-64pmt\" (UID: \"ed4e44b6-7fbe-4e18-974e-edbc64ce0398\") " pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:46:43.408918 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:43.408905    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2rm6\" (UniqueName: \"kubernetes.io/projected/f4699b63-6ba0-4401-a43e-57c15fea25b3-kube-api-access-p2rm6\") pod \"network-check-target-9bl8d\" (UID: \"f4699b63-6ba0-4401-a43e-57c15fea25b3\") " pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:46:43.409149 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:43.408983    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:43.409149 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:43.409016    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:46:43.409149 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:43.409038    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:46:43.409149 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:43.409053    2564 projected.go:196] Error preparing data for projected volume kube-api-access-p2rm6 for pod openshift-network-diagnostics/network-check-target-9bl8d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:43.409149 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:43.409061    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-original-pull-secret podName:ed4e44b6-7fbe-4e18-974e-edbc64ce0398 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:44.409037478 +0000 UTC m=+4.085053024 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-original-pull-secret") pod "global-pull-secret-syncer-64pmt" (UID: "ed4e44b6-7fbe-4e18-974e-edbc64ce0398") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:43.409149 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:43.409104    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c606e570-2fd6-43cb-9a84-9cdeab442815-metrics-certs\") pod \"network-metrics-daemon-9xhz4\" (UID: \"c606e570-2fd6-43cb-9a84-9cdeab442815\") " pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:46:43.409149 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:43.409111    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4699b63-6ba0-4401-a43e-57c15fea25b3-kube-api-access-p2rm6 podName:f4699b63-6ba0-4401-a43e-57c15fea25b3 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:44.409094507 +0000 UTC m=+4.085110075 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2rm6" (UniqueName: "kubernetes.io/projected/f4699b63-6ba0-4401-a43e-57c15fea25b3-kube-api-access-p2rm6") pod "network-check-target-9bl8d" (UID: "f4699b63-6ba0-4401-a43e-57c15fea25b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:43.409533 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:43.409177    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:43.409533 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:43.409222    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c606e570-2fd6-43cb-9a84-9cdeab442815-metrics-certs podName:c606e570-2fd6-43cb-9a84-9cdeab442815 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:44.409211248 +0000 UTC m=+4.085226794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c606e570-2fd6-43cb-9a84-9cdeab442815-metrics-certs") pod "network-metrics-daemon-9xhz4" (UID: "c606e570-2fd6-43cb-9a84-9cdeab442815") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:43.738241 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:43.738135    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 20:41:42 +0000 UTC" deadline="2027-12-03 04:34:34.65919084 +0000 UTC"
Jul 01 20:46:43.738241 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:43.738188    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12463h47m50.921008589s"
Jul 01 20:46:43.947446 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:43.947419    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa69aa3_5a0f_4fb7_b2aa_a44b5754e78e.slice/crio-dd72d80bbe8904521716372dbf529201db0c7db531715e1c69c4049849bdc5d7 WatchSource:0}: Error finding container dd72d80bbe8904521716372dbf529201db0c7db531715e1c69c4049849bdc5d7: Status 404 returned error can't find the container with id dd72d80bbe8904521716372dbf529201db0c7db531715e1c69c4049849bdc5d7
Jul 01 20:46:43.948707 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:43.948625    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb41c3f0_f0dc_4340_b524_bb23e5426370.slice/crio-1f7e9b57a049fcbb5e69d77301f7ccac16a7b6e12fe35a19f06da5acecf11093 WatchSource:0}: Error finding container 1f7e9b57a049fcbb5e69d77301f7ccac16a7b6e12fe35a19f06da5acecf11093: Status 404 returned error can't find the container with id 1f7e9b57a049fcbb5e69d77301f7ccac16a7b6e12fe35a19f06da5acecf11093
Jul 01 20:46:43.949908 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:43.949881    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d8b430_fceb_4c6b_9f65_233650a04f7f.slice/crio-dba62b362821465103ffc7ec2fc5ad051e4caaed60475cb97b0bbdbe0fd3e422 WatchSource:0}: Error finding container dba62b362821465103ffc7ec2fc5ad051e4caaed60475cb97b0bbdbe0fd3e422: Status 404 returned error can't find the container with id dba62b362821465103ffc7ec2fc5ad051e4caaed60475cb97b0bbdbe0fd3e422
Jul 01 20:46:43.951427 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:43.950970    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ef23e3_d38f_47a1_94ea_aa4f8fc40725.slice/crio-f7846f8d357268cc40c1f3a870730ed41f6a8ea0ac3fdfde04217dde07a1b119 WatchSource:0}: Error finding container f7846f8d357268cc40c1f3a870730ed41f6a8ea0ac3fdfde04217dde07a1b119: Status 404 returned error can't find the container with id f7846f8d357268cc40c1f3a870730ed41f6a8ea0ac3fdfde04217dde07a1b119
Jul 01 20:46:43.954099 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:43.954067    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a26d5f_fa90_4d3e_95c3_d8ff0afbd4c2.slice/crio-34200368124d99bb86fbe68e22c3e6e15f64cd8143956f46946abd9493493eb1 WatchSource:0}: Error finding container 34200368124d99bb86fbe68e22c3e6e15f64cd8143956f46946abd9493493eb1: Status 404 returned error can't find the container with id 34200368124d99bb86fbe68e22c3e6e15f64cd8143956f46946abd9493493eb1
Jul 01 20:46:43.955733 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:43.955557    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48225b8f_cd34_4fc1_b18e_7b219a899f86.slice/crio-e8a2dc80a335aba8aff0acc477f37d14d42138d70bedc939df88d6e777525279 WatchSource:0}: Error finding container e8a2dc80a335aba8aff0acc477f37d14d42138d70bedc939df88d6e777525279: Status 404 returned error can't find the container with id e8a2dc80a335aba8aff0acc477f37d14d42138d70bedc939df88d6e777525279
Jul 01 20:46:43.956421 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:46:43.956401    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8effda_0dd6_437e_86f5_ac9c97c08193.slice/crio-4e94c3b83c320b8db09ba503100d1790ed70f695574784d825e663f265c7baea WatchSource:0}: Error finding container 4e94c3b83c320b8db09ba503100d1790ed70f695574784d825e663f265c7baea: Status 404 returned error can't find the container with id 4e94c3b83c320b8db09ba503100d1790ed70f695574784d825e663f265c7baea
Jul 01 20:46:44.220975 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:44.219388    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 20:46:44.417704 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:44.417582    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-original-pull-secret\") pod \"global-pull-secret-syncer-64pmt\" (UID: \"ed4e44b6-7fbe-4e18-974e-edbc64ce0398\") " pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:46:44.417704 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:44.417668    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2rm6\" (UniqueName: \"kubernetes.io/projected/f4699b63-6ba0-4401-a43e-57c15fea25b3-kube-api-access-p2rm6\") pod \"network-check-target-9bl8d\" (UID: \"f4699b63-6ba0-4401-a43e-57c15fea25b3\") " pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:46:44.417704 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:44.417701    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c606e570-2fd6-43cb-9a84-9cdeab442815-metrics-certs\") pod \"network-metrics-daemon-9xhz4\" (UID: \"c606e570-2fd6-43cb-9a84-9cdeab442815\") " pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:46:44.418035 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:44.417844    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:44.418035 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:44.417930    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c606e570-2fd6-43cb-9a84-9cdeab442815-metrics-certs podName:c606e570-2fd6-43cb-9a84-9cdeab442815 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:46.417889761 +0000 UTC m=+6.093905324 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c606e570-2fd6-43cb-9a84-9cdeab442815-metrics-certs") pod "network-metrics-daemon-9xhz4" (UID: "c606e570-2fd6-43cb-9a84-9cdeab442815") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:44.418407 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:44.418385    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:44.418501 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:44.418449    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-original-pull-secret podName:ed4e44b6-7fbe-4e18-974e-edbc64ce0398 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:46.418432532 +0000 UTC m=+6.094448083 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-original-pull-secret") pod "global-pull-secret-syncer-64pmt" (UID: "ed4e44b6-7fbe-4e18-974e-edbc64ce0398") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:44.418572 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:44.418536    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:46:44.418572 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:44.418558    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:46:44.418680 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:44.418572    2564 projected.go:196] Error preparing data for projected volume kube-api-access-p2rm6 for pod openshift-network-diagnostics/network-check-target-9bl8d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:44.418680 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:44.418629    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4699b63-6ba0-4401-a43e-57c15fea25b3-kube-api-access-p2rm6 podName:f4699b63-6ba0-4401-a43e-57c15fea25b3 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:46.418613857 +0000 UTC m=+6.094629420 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2rm6" (UniqueName: "kubernetes.io/projected/f4699b63-6ba0-4401-a43e-57c15fea25b3-kube-api-access-p2rm6") pod "network-check-target-9bl8d" (UID: "f4699b63-6ba0-4401-a43e-57c15fea25b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:44.761698 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:44.754062    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:46:44.761698 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:44.754202    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:46:44.761698 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:44.754597    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:46:44.761698 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:44.754681    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:46:44.761698 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:44.754739    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:46:44.761698 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:44.754812    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bl8d" podUID="f4699b63-6ba0-4401-a43e-57c15fea25b3"
Jul 01 20:46:44.769640 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:44.769596    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fr7j6" event={"ID":"d0e7e2a9-4cad-47c6-8a6b-46c461461be1","Type":"ContainerStarted","Data":"5d52c08298cb6ed47842013b734b598cb4e03b8084e0a1a316d97f94c0852262"}
Jul 01 20:46:44.775932 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:44.775833    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mkqr5" event={"ID":"8634b68f-65d5-4747-b19c-2b2e1d44ba76","Type":"ContainerStarted","Data":"78732daa8155f0fc7ccdf29660f4e5e9b15018b00e10bbb2e15596ed13296754"}
Jul 01 20:46:44.778825 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:44.778750    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lmpqc" event={"ID":"48225b8f-cd34-4fc1-b18e-7b219a899f86","Type":"ContainerStarted","Data":"e8a2dc80a335aba8aff0acc477f37d14d42138d70bedc939df88d6e777525279"}
Jul 01 20:46:44.783821 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:44.783745    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq" event={"ID":"47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2","Type":"ContainerStarted","Data":"34200368124d99bb86fbe68e22c3e6e15f64cd8143956f46946abd9493493eb1"}
Jul 01 20:46:44.792576 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:44.792549    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wmgbq" event={"ID":"54d8b430-fceb-4c6b-9f65-233650a04f7f","Type":"ContainerStarted","Data":"dba62b362821465103ffc7ec2fc5ad051e4caaed60475cb97b0bbdbe0fd3e422"}
Jul 01 20:46:44.797209 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:44.797183    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mfxlf" event={"ID":"fb41c3f0-f0dc-4340-b524-bb23e5426370","Type":"ContainerStarted","Data":"1f7e9b57a049fcbb5e69d77301f7ccac16a7b6e12fe35a19f06da5acecf11093"}
Jul 01 20:46:44.819967 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:44.815020    2564 generic.go:356] "Generic (PLEG): container finished" podID="775ac52276a7c20fbf0b139e46e0a4bf" containerID="9afa0866724cc4317842bc244cd32b542ba4fd6bf3659337a376b5d1d113b226" exitCode=0
Jul 01 20:46:44.819967 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:44.815371    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-198.ec2.internal" event={"ID":"775ac52276a7c20fbf0b139e46e0a4bf","Type":"ContainerDied","Data":"9afa0866724cc4317842bc244cd32b542ba4fd6bf3659337a376b5d1d113b226"}
Jul 01 20:46:44.824001 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:44.823518    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-198.ec2.internal" event={"ID":"7bae69993a9f5e773a5205c5e78d2b81","Type":"ContainerStarted","Data":"00e7e645913e4b98d2cc810996c6fd26985a781a50aea899257179e4a5bd505f"}
Jul 01 20:46:44.830213 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:44.829995    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ljqg5" event={"ID":"eb8effda-0dd6-437e-86f5-ac9c97c08193","Type":"ContainerStarted","Data":"4e94c3b83c320b8db09ba503100d1790ed70f695574784d825e663f265c7baea"}
Jul 01 20:46:44.858818 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:44.858755    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ftdgt" event={"ID":"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725","Type":"ContainerStarted","Data":"f7846f8d357268cc40c1f3a870730ed41f6a8ea0ac3fdfde04217dde07a1b119"}
Jul 01 20:46:44.862457 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:44.862346    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k" event={"ID":"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e","Type":"ContainerStarted","Data":"dd72d80bbe8904521716372dbf529201db0c7db531715e1c69c4049849bdc5d7"}
Jul 01 20:46:45.892025 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:45.891991    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-198.ec2.internal" event={"ID":"775ac52276a7c20fbf0b139e46e0a4bf","Type":"ContainerStarted","Data":"2b17da61ee830dda34bfaf8f69039bb441b090ba0fa7986964c6dbdfc245d07f"}
Jul 01 20:46:45.911458 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:45.911405    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-198.ec2.internal" podStartSLOduration=3.911382722 podStartE2EDuration="3.911382722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:45.911054158 +0000 UTC m=+5.587069735" watchObservedRunningTime="2026-07-01 20:46:45.911382722 +0000 UTC m=+5.587398294"
Jul 01 20:46:45.912241 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:45.912201    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-198.ec2.internal" podStartSLOduration=3.912185806 podStartE2EDuration="3.912185806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:46:44.851673578 +0000 UTC m=+4.527689148" watchObservedRunningTime="2026-07-01 20:46:45.912185806 +0000 UTC m=+5.588201380"
Jul 01 20:46:46.436843 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:46.436794    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2rm6\" (UniqueName: \"kubernetes.io/projected/f4699b63-6ba0-4401-a43e-57c15fea25b3-kube-api-access-p2rm6\") pod \"network-check-target-9bl8d\" (UID: \"f4699b63-6ba0-4401-a43e-57c15fea25b3\") " pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:46:46.437020 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:46.436859    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c606e570-2fd6-43cb-9a84-9cdeab442815-metrics-certs\") pod \"network-metrics-daemon-9xhz4\" (UID: \"c606e570-2fd6-43cb-9a84-9cdeab442815\") " pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:46:46.437020 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:46.436934    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-original-pull-secret\") pod \"global-pull-secret-syncer-64pmt\" (UID: \"ed4e44b6-7fbe-4e18-974e-edbc64ce0398\") " pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:46:46.437146 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:46.437063    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:46.437146 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:46.437129    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-original-pull-secret podName:ed4e44b6-7fbe-4e18-974e-edbc64ce0398 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:50.437109778 +0000 UTC m=+10.113125340 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-original-pull-secret") pod "global-pull-secret-syncer-64pmt" (UID: "ed4e44b6-7fbe-4e18-974e-edbc64ce0398") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:46.437861 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:46.437568    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:46:46.437861 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:46.437590    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:46:46.437861 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:46.437604    2564 projected.go:196] Error preparing data for projected volume kube-api-access-p2rm6 for pod openshift-network-diagnostics/network-check-target-9bl8d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:46.437861 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:46.437653    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4699b63-6ba0-4401-a43e-57c15fea25b3-kube-api-access-p2rm6 podName:f4699b63-6ba0-4401-a43e-57c15fea25b3 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:50.437637787 +0000 UTC m=+10.113653339 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2rm6" (UniqueName: "kubernetes.io/projected/f4699b63-6ba0-4401-a43e-57c15fea25b3-kube-api-access-p2rm6") pod "network-check-target-9bl8d" (UID: "f4699b63-6ba0-4401-a43e-57c15fea25b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:46.437861 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:46.437714    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:46.437861 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:46.437749    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c606e570-2fd6-43cb-9a84-9cdeab442815-metrics-certs podName:c606e570-2fd6-43cb-9a84-9cdeab442815 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:50.437735763 +0000 UTC m=+10.113751311 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c606e570-2fd6-43cb-9a84-9cdeab442815-metrics-certs") pod "network-metrics-daemon-9xhz4" (UID: "c606e570-2fd6-43cb-9a84-9cdeab442815") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:46.755468 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:46.754905    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:46:46.755468 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:46.754957    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:46:46.755468 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:46.754904    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:46:46.755468 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:46.755063    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:46:46.755468 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:46.755159    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bl8d" podUID="f4699b63-6ba0-4401-a43e-57c15fea25b3"
Jul 01 20:46:46.755468 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:46.755248    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:46:48.752946 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:48.752911    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:46:48.753372 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:48.752954    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:46:48.753372 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:48.752920    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:46:48.753372 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:48.753055    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:46:48.753372 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:48.753132    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bl8d" podUID="f4699b63-6ba0-4401-a43e-57c15fea25b3"
Jul 01 20:46:48.753372 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:48.753217    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:46:50.470546 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:50.470486    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2rm6\" (UniqueName: \"kubernetes.io/projected/f4699b63-6ba0-4401-a43e-57c15fea25b3-kube-api-access-p2rm6\") pod \"network-check-target-9bl8d\" (UID: \"f4699b63-6ba0-4401-a43e-57c15fea25b3\") " pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:46:50.470546 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:50.470536    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c606e570-2fd6-43cb-9a84-9cdeab442815-metrics-certs\") pod \"network-metrics-daemon-9xhz4\" (UID: \"c606e570-2fd6-43cb-9a84-9cdeab442815\") " pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:46:50.471072 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:50.470609    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:46:50.471072 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:50.470628    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-original-pull-secret\") pod \"global-pull-secret-syncer-64pmt\" (UID: \"ed4e44b6-7fbe-4e18-974e-edbc64ce0398\") " pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:46:50.471072 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:50.470634    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:46:50.471072 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:50.470699    2564 projected.go:196] Error preparing data for projected volume kube-api-access-p2rm6 for pod openshift-network-diagnostics/network-check-target-9bl8d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:50.471072 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:50.470704    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:50.471072 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:50.470763    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c606e570-2fd6-43cb-9a84-9cdeab442815-metrics-certs podName:c606e570-2fd6-43cb-9a84-9cdeab442815 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:58.470743077 +0000 UTC m=+18.146758639 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c606e570-2fd6-43cb-9a84-9cdeab442815-metrics-certs") pod "network-metrics-daemon-9xhz4" (UID: "c606e570-2fd6-43cb-9a84-9cdeab442815") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:50.471072 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:50.470784    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4699b63-6ba0-4401-a43e-57c15fea25b3-kube-api-access-p2rm6 podName:f4699b63-6ba0-4401-a43e-57c15fea25b3 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:58.470772538 +0000 UTC m=+18.146788084 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2rm6" (UniqueName: "kubernetes.io/projected/f4699b63-6ba0-4401-a43e-57c15fea25b3-kube-api-access-p2rm6") pod "network-check-target-9bl8d" (UID: "f4699b63-6ba0-4401-a43e-57c15fea25b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:50.471072 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:50.470842    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:50.471072 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:50.470899    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-original-pull-secret podName:ed4e44b6-7fbe-4e18-974e-edbc64ce0398 nodeName:}" failed. No retries permitted until 2026-07-01 20:46:58.47088405 +0000 UTC m=+18.146899599 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-original-pull-secret") pod "global-pull-secret-syncer-64pmt" (UID: "ed4e44b6-7fbe-4e18-974e-edbc64ce0398") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:50.753653 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:50.753569    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:46:50.753653 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:50.753594    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:46:50.753891 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:50.753701    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:46:50.753891 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:50.753773    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:46:50.753891 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:50.753854    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:46:50.754063 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:50.753943    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bl8d" podUID="f4699b63-6ba0-4401-a43e-57c15fea25b3"
Jul 01 20:46:52.753412 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:52.753364    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:46:52.753412 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:52.753392    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:46:52.753412 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:52.753412    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:46:52.754050 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:52.753507    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:46:52.754050 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:52.753646    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:46:52.754050 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:52.753729    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bl8d" podUID="f4699b63-6ba0-4401-a43e-57c15fea25b3"
Jul 01 20:46:54.753580 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:54.753477    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:46:54.753580 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:54.753499    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:46:54.753580 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:54.753478    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:46:54.754120 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:54.753605    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bl8d" podUID="f4699b63-6ba0-4401-a43e-57c15fea25b3"
Jul 01 20:46:54.754120 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:54.753716    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:46:54.754120 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:54.753809    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:46:56.752691 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:56.752652    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:46:56.752691 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:56.752666    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:46:56.753253 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:56.752652    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:46:56.753253 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:56.752788    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:46:56.753253 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:56.752870    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bl8d" podUID="f4699b63-6ba0-4401-a43e-57c15fea25b3"
Jul 01 20:46:56.753253 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:56.752963    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:46:58.530326 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:58.530279    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-original-pull-secret\") pod \"global-pull-secret-syncer-64pmt\" (UID: \"ed4e44b6-7fbe-4e18-974e-edbc64ce0398\") " pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:46:58.530884 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:58.530372    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2rm6\" (UniqueName: \"kubernetes.io/projected/f4699b63-6ba0-4401-a43e-57c15fea25b3-kube-api-access-p2rm6\") pod \"network-check-target-9bl8d\" (UID: \"f4699b63-6ba0-4401-a43e-57c15fea25b3\") " pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:46:58.530884 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:58.530401    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c606e570-2fd6-43cb-9a84-9cdeab442815-metrics-certs\") pod \"network-metrics-daemon-9xhz4\" (UID: \"c606e570-2fd6-43cb-9a84-9cdeab442815\") " pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:46:58.530884 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:58.530442    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:58.530884 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:58.530502    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:58.530884 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:58.530514    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-original-pull-secret podName:ed4e44b6-7fbe-4e18-974e-edbc64ce0398 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:14.530492531 +0000 UTC m=+34.206508076 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-original-pull-secret") pod "global-pull-secret-syncer-64pmt" (UID: "ed4e44b6-7fbe-4e18-974e-edbc64ce0398") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:46:58.530884 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:58.530549    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:46:58.530884 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:58.530572    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:46:58.530884 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:58.530598    2564 projected.go:196] Error preparing data for projected volume kube-api-access-p2rm6 for pod openshift-network-diagnostics/network-check-target-9bl8d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:58.530884 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:58.530553    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c606e570-2fd6-43cb-9a84-9cdeab442815-metrics-certs podName:c606e570-2fd6-43cb-9a84-9cdeab442815 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:14.530536258 +0000 UTC m=+34.206551810 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c606e570-2fd6-43cb-9a84-9cdeab442815-metrics-certs") pod "network-metrics-daemon-9xhz4" (UID: "c606e570-2fd6-43cb-9a84-9cdeab442815") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:46:58.530884 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:58.530649    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4699b63-6ba0-4401-a43e-57c15fea25b3-kube-api-access-p2rm6 podName:f4699b63-6ba0-4401-a43e-57c15fea25b3 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:14.53063687 +0000 UTC m=+34.206652429 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2rm6" (UniqueName: "kubernetes.io/projected/f4699b63-6ba0-4401-a43e-57c15fea25b3-kube-api-access-p2rm6") pod "network-check-target-9bl8d" (UID: "f4699b63-6ba0-4401-a43e-57c15fea25b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:46:58.752928 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:58.752895    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:46:58.753116 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:58.752895    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:46:58.753116 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:58.753023    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bl8d" podUID="f4699b63-6ba0-4401-a43e-57c15fea25b3"
Jul 01 20:46:58.753116 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:46:58.752908    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:46:58.753251 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:58.753116    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:46:58.753251 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:46:58.753192    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:47:00.753379 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:00.753344    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:47:00.753841 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:00.753450    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:47:00.753841 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:00.753450    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:47:00.753841 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:00.753578    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:47:00.753841 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:00.753598    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:47:00.753841 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:00.753702    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bl8d" podUID="f4699b63-6ba0-4401-a43e-57c15fea25b3"
Jul 01 20:47:02.752640 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:02.752611    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:47:02.752640 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:02.752640    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:47:02.753097 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:02.752611    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:47:02.753097 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:02.752702    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:47:02.753097 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:02.752805    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bl8d" podUID="f4699b63-6ba0-4401-a43e-57c15fea25b3"
Jul 01 20:47:02.753097 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:02.753025    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:47:03.929673 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:03.929280    2564 generic.go:356] "Generic (PLEG): container finished" podID="fb41c3f0-f0dc-4340-b524-bb23e5426370" containerID="90c67d75764b6550f380b7ab92899d5f32d46f0ae9143733267ad80965a652eb" exitCode=0
Jul 01 20:47:03.930499 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:03.929374    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mfxlf" event={"ID":"fb41c3f0-f0dc-4340-b524-bb23e5426370","Type":"ContainerDied","Data":"90c67d75764b6550f380b7ab92899d5f32d46f0ae9143733267ad80965a652eb"}
Jul 01 20:47:03.931489 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:03.931463    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ljqg5" event={"ID":"eb8effda-0dd6-437e-86f5-ac9c97c08193","Type":"ContainerStarted","Data":"bdc880ad9740fdcd6e246dbc8e41ac68ae4f09bc4ddd0bbcb828f2727892f154"}
Jul 01 20:47:03.933215 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:03.933193    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ftdgt" event={"ID":"c2ef23e3-d38f-47a1-94ea-aa4f8fc40725","Type":"ContainerStarted","Data":"59bb1f06639f083d2dc3288a8ad1bca4ff6f88c7d9fd94f1c3bcb1ef7c6184e8"}
Jul 01 20:47:03.936599 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:03.936577    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/ovn-acl-logging/0.log"
Jul 01 20:47:03.937016 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:03.936994    2564 generic.go:356] "Generic (PLEG): container finished" podID="ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e" containerID="6cd8d23c9f63ffafe842671c5decf44f36d18aad2df2f85adb5ac0ea761fa58d" exitCode=1
Jul 01 20:47:03.937117 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:03.937049    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k" event={"ID":"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e","Type":"ContainerStarted","Data":"2e727d4853f18288c53a5a3f6702525d2f82a69d4cfeff9bbe819c774442e14c"}
Jul 01 20:47:03.937117 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:03.937112    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k" event={"ID":"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e","Type":"ContainerStarted","Data":"a35a03f6151bb4529e9d7b92ce2aadf185e1591ed03895a9a163af7fb2c8a5fc"}
Jul 01 20:47:03.937226 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:03.937149    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k" event={"ID":"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e","Type":"ContainerStarted","Data":"08274f8c110fda5e6dc9e68ee0a083718ee3fe7178e7c83bf0563b700dbd7918"}
Jul 01 20:47:03.937226 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:03.937189    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k" event={"ID":"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e","Type":"ContainerStarted","Data":"f8f139842f49592001cdb57ce26dc4c0bdea5edd53ee38a7e1b37bf8dc4a684c"}
Jul 01 20:47:03.937323 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:03.937240    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k" event={"ID":"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e","Type":"ContainerDied","Data":"6cd8d23c9f63ffafe842671c5decf44f36d18aad2df2f85adb5ac0ea761fa58d"}
Jul 01 20:47:03.937323 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:03.937287    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k" event={"ID":"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e","Type":"ContainerStarted","Data":"4ecf6b439f550e9d29cd4e99bf184ae3ee0fbc672348c950e4f14fa251b0ce0b"}
Jul 01 20:47:03.938759 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:03.938720    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fr7j6" event={"ID":"d0e7e2a9-4cad-47c6-8a6b-46c461461be1","Type":"ContainerStarted","Data":"a181e5133dc42ab1625ec5ffcf991fa395bdff650e771755bf98ec1bc3f72db7"}
Jul 01 20:47:03.940361 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:03.940333    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mkqr5" event={"ID":"8634b68f-65d5-4747-b19c-2b2e1d44ba76","Type":"ContainerStarted","Data":"11be4ec58f796bc905588475dc126fea7b32f847027b493c93f706af16d5639b"}
Jul 01 20:47:03.941910 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:03.941830    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq" event={"ID":"47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2","Type":"ContainerStarted","Data":"105e3f94a9d25dcc1361cc269f59c898a0a0fc150ab925c42245ab2af05925e8"}
Jul 01 20:47:03.943302 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:03.943269    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wmgbq" event={"ID":"54d8b430-fceb-4c6b-9f65-233650a04f7f","Type":"ContainerStarted","Data":"2dc06b2034f52f36fe3954c474fedc786b261523b9f0b4f6e671be6d3fa54175"}
Jul 01 20:47:03.980714 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:03.980653    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wmgbq" podStartSLOduration=4.140586862 podStartE2EDuration="22.980635631s" totalImagesPullingTime="18.840048769s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:03.980246971 +0000 UTC m=+23.656262537" watchObservedRunningTime="2026-07-01 20:47:03.980635631 +0000 UTC m=+23.656651198"
Jul 01 20:47:04.031598 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:04.031555    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ljqg5" podStartSLOduration=4.201023306 podStartE2EDuration="23.031537437s" totalImagesPullingTime="18.830514131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:04.0314382 +0000 UTC m=+23.707453768" watchObservedRunningTime="2026-07-01 20:47:04.031537437 +0000 UTC m=+23.707553047"
Jul 01 20:47:04.031793 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:04.031705    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fr7j6" podStartSLOduration=4.231253736 podStartE2EDuration="23.031695841s" totalImagesPullingTime="18.800442105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:04.006806294 +0000 UTC m=+23.682821861" watchObservedRunningTime="2026-07-01 20:47:04.031695841 +0000 UTC m=+23.707711408"
Jul 01 20:47:04.056658 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:04.055298    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mkqr5" podStartSLOduration=4.643777827 podStartE2EDuration="23.055271686s" totalImagesPullingTime="18.411493859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:04.0550793 +0000 UTC m=+23.731094868" watchObservedRunningTime="2026-07-01 20:47:04.055271686 +0000 UTC m=+23.731287255"
Jul 01 20:47:04.077508 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:04.077466    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ftdgt" podStartSLOduration=4.237045311 podStartE2EDuration="23.077451224s" totalImagesPullingTime="18.840405913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:04.076788993 +0000 UTC m=+23.752804559" watchObservedRunningTime="2026-07-01 20:47:04.077451224 +0000 UTC m=+23.753466790"
Jul 01 20:47:04.225682 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:04.225665    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fr7j6_d0e7e2a9-4cad-47c6-8a6b-46c461461be1/node-ca/0.log"
Jul 01 20:47:04.535520 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:04.535484    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 01 20:47:04.752992 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:04.752911    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:47:04.752992 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:04.752924    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:47:04.753240 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:04.752915    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:47:04.753240 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:04.753098    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bl8d" podUID="f4699b63-6ba0-4401-a43e-57c15fea25b3"
Jul 01 20:47:04.753240 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:04.753177    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:47:04.753365 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:04.753256    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:47:04.761159 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:04.761081    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-01T20:47:04.535502854Z","UUID":"06c47e02-c673-405c-bc3f-760c23525e1d","Handler":null,"Name":"","Endpoint":""}
Jul 01 20:47:04.762594 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:04.762573    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 01 20:47:04.762706 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:04.762609    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 01 20:47:04.947337 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:04.947301    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lmpqc" event={"ID":"48225b8f-cd34-4fc1-b18e-7b219a899f86","Type":"ContainerStarted","Data":"fc1514f95ddb9d810c9f8eda1de09ac04a313c618383cfd90c1c9947259fb2ff"}
Jul 01 20:47:04.949258 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:04.949229    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq" event={"ID":"47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2","Type":"ContainerStarted","Data":"51651f813cc623d9963052931a981e740ce6a68e95f964c87568548147c21f01"}
Jul 01 20:47:06.752716 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:06.752683    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:47:06.753196 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:06.752683    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:47:06.753196 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:06.752786    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:47:06.753196 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:06.752803    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:47:06.753196 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:06.752887    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bl8d" podUID="f4699b63-6ba0-4401-a43e-57c15fea25b3"
Jul 01 20:47:06.753196 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:06.752988    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:47:06.956043 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:06.956013    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/ovn-acl-logging/0.log"
Jul 01 20:47:06.956477 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:06.956442    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k" event={"ID":"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e","Type":"ContainerStarted","Data":"4a34ddef9fd9cd0796b089b3bf1a35355e6154e110c52db2f6b95ed401010b6f"}
Jul 01 20:47:06.958453 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:06.958431    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq" event={"ID":"47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2","Type":"ContainerStarted","Data":"ffc60a045b12fdb59edc70ffdd6ed0ad167fe401d30294028a60437cd7a56be8"}
Jul 01 20:47:06.987091 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:06.987035    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lmpqc" podStartSLOduration=7.183195345 podStartE2EDuration="25.987013552s" totalImagesPullingTime="18.803818207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:04.967427447 +0000 UTC m=+24.643443026" watchObservedRunningTime="2026-07-01 20:47:06.987013552 +0000 UTC m=+26.663029120"
Jul 01 20:47:06.987441 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:06.987397    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xsscq" podStartSLOduration=4.487746024 podStartE2EDuration="25.987388489s" totalImagesPullingTime="21.499642465s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:06.98599224 +0000 UTC m=+26.662007803" watchObservedRunningTime="2026-07-01 20:47:06.987388489 +0000 UTC m=+26.663404058"
Jul 01 20:47:08.753445 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:08.753246    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:47:08.754093 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:08.753254    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:47:08.754093 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:08.753475    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:47:08.754093 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:08.753285    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:47:08.754093 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:08.753561    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bl8d" podUID="f4699b63-6ba0-4401-a43e-57c15fea25b3"
Jul 01 20:47:08.754093 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:08.753657    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:47:08.830207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:08.830172    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mkqr5"
Jul 01 20:47:08.830843 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:08.830823    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mkqr5"
Jul 01 20:47:08.965035 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:08.965007    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/ovn-acl-logging/0.log"
Jul 01 20:47:08.965403 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:08.965381    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k" event={"ID":"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e","Type":"ContainerStarted","Data":"18e316387f637744b0d32cd239ff85b59fb3ab309b07658a0ff76bb51e07616d"}
Jul 01 20:47:08.965621 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:08.965584    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:47:08.968778 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:08.966490    2564 scope.go:117] "RemoveContainer" containerID="6cd8d23c9f63ffafe842671c5decf44f36d18aad2df2f85adb5ac0ea761fa58d"
Jul 01 20:47:08.970594 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:08.970567    2564 generic.go:356] "Generic (PLEG): container finished" podID="fb41c3f0-f0dc-4340-b524-bb23e5426370" containerID="eb5700a6334426bc75c1d33bc3a0d9bb2850004f5ffd8d49b0b3f27591bf0a48" exitCode=0
Jul 01 20:47:08.970695 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:08.970615    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mfxlf" event={"ID":"fb41c3f0-f0dc-4340-b524-bb23e5426370","Type":"ContainerDied","Data":"eb5700a6334426bc75c1d33bc3a0d9bb2850004f5ffd8d49b0b3f27591bf0a48"}
Jul 01 20:47:08.970754 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:08.970709    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mkqr5"
Jul 01 20:47:08.971247 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:08.971231    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mkqr5"
Jul 01 20:47:08.983155 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:08.982998    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:47:09.975644 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:09.975616    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/ovn-acl-logging/0.log"
Jul 01 20:47:09.976089 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:09.975956    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k" event={"ID":"ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e","Type":"ContainerStarted","Data":"d646321a3ff39f66bd8fe0546f1a00be681a639fa6d12a7de4245c018b808440"}
Jul 01 20:47:09.976258 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:09.976241    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:47:09.976390 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:09.976379    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:47:09.990991 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:09.990968    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:47:10.026025 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:10.025244    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k" podStartSLOduration=9.967910184 podStartE2EDuration="29.025223488s" totalImagesPullingTime="19.057313304s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:10.024120495 +0000 UTC m=+29.700136063" watchObservedRunningTime="2026-07-01 20:47:10.025223488 +0000 UTC m=+29.701239056"
Jul 01 20:47:10.754190 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:10.754102    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:47:10.754425 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:10.754236    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:47:10.754425 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:10.754248    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bl8d" podUID="f4699b63-6ba0-4401-a43e-57c15fea25b3"
Jul 01 20:47:10.754425 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:10.754361    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:47:10.754425 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:10.754371    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:47:10.754603 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:10.754446    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:47:10.980029 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:10.979995    2564 generic.go:356] "Generic (PLEG): container finished" podID="fb41c3f0-f0dc-4340-b524-bb23e5426370" containerID="8c5a9e55c4388444d850365179417df0ef5adc0c8d972496c72da94566446527" exitCode=0
Jul 01 20:47:10.980502 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:10.980062    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mfxlf" event={"ID":"fb41c3f0-f0dc-4340-b524-bb23e5426370","Type":"ContainerDied","Data":"8c5a9e55c4388444d850365179417df0ef5adc0c8d972496c72da94566446527"}
Jul 01 20:47:12.752754 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:12.752717    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:47:12.753185 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:12.752717    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:47:12.753185 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:12.752841    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:47:12.753185 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:12.752939    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:47:12.753185 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:12.752728    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:47:12.753185 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:12.753035    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bl8d" podUID="f4699b63-6ba0-4401-a43e-57c15fea25b3"
Jul 01 20:47:12.986513 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:12.986481    2564 generic.go:356] "Generic (PLEG): container finished" podID="fb41c3f0-f0dc-4340-b524-bb23e5426370" containerID="1c666ecac3d99b7a486cc28e5b85561f38ec85dfbdff7ec359715f6ee14901ed" exitCode=0
Jul 01 20:47:12.986678 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:12.986547    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mfxlf" event={"ID":"fb41c3f0-f0dc-4340-b524-bb23e5426370","Type":"ContainerDied","Data":"1c666ecac3d99b7a486cc28e5b85561f38ec85dfbdff7ec359715f6ee14901ed"}
Jul 01 20:47:14.544040 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:14.544005    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2rm6\" (UniqueName: \"kubernetes.io/projected/f4699b63-6ba0-4401-a43e-57c15fea25b3-kube-api-access-p2rm6\") pod \"network-check-target-9bl8d\" (UID: \"f4699b63-6ba0-4401-a43e-57c15fea25b3\") " pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:47:14.544040 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:14.544048    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c606e570-2fd6-43cb-9a84-9cdeab442815-metrics-certs\") pod \"network-metrics-daemon-9xhz4\" (UID: \"c606e570-2fd6-43cb-9a84-9cdeab442815\") " pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:47:14.544579 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:14.544088    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-original-pull-secret\") pod \"global-pull-secret-syncer-64pmt\" (UID: \"ed4e44b6-7fbe-4e18-974e-edbc64ce0398\") " pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:47:14.544579 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:14.544203    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 20:47:14.544579 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:14.544212    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 20:47:14.544579 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:14.544229    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 20:47:14.544579 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:14.544245    2564 projected.go:196] Error preparing data for projected volume kube-api-access-p2rm6 for pod openshift-network-diagnostics/network-check-target-9bl8d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:47:14.544579 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:14.544275    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-original-pull-secret podName:ed4e44b6-7fbe-4e18-974e-edbc64ce0398 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:46.544260891 +0000 UTC m=+66.220276450 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-original-pull-secret") pod "global-pull-secret-syncer-64pmt" (UID: "ed4e44b6-7fbe-4e18-974e-edbc64ce0398") : object "kube-system"/"original-pull-secret" not registered
Jul 01 20:47:14.544579 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:14.544201    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:47:14.544579 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:14.544322    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4699b63-6ba0-4401-a43e-57c15fea25b3-kube-api-access-p2rm6 podName:f4699b63-6ba0-4401-a43e-57c15fea25b3 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:46.544280631 +0000 UTC m=+66.220296194 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2rm6" (UniqueName: "kubernetes.io/projected/f4699b63-6ba0-4401-a43e-57c15fea25b3-kube-api-access-p2rm6") pod "network-check-target-9bl8d" (UID: "f4699b63-6ba0-4401-a43e-57c15fea25b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 20:47:14.544579 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:14.544424    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c606e570-2fd6-43cb-9a84-9cdeab442815-metrics-certs podName:c606e570-2fd6-43cb-9a84-9cdeab442815 nodeName:}" failed. No retries permitted until 2026-07-01 20:47:46.544406829 +0000 UTC m=+66.220422378 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c606e570-2fd6-43cb-9a84-9cdeab442815-metrics-certs") pod "network-metrics-daemon-9xhz4" (UID: "c606e570-2fd6-43cb-9a84-9cdeab442815") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 20:47:14.753297 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:14.753263    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:47:14.753513 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:14.753269    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:47:14.753513 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:14.753392    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:47:14.753513 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:14.753486    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:47:14.753513 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:14.753487    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bl8d" podUID="f4699b63-6ba0-4401-a43e-57c15fea25b3"
Jul 01 20:47:14.753715 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:14.753595    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:47:16.752947 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:16.752916    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:47:16.753418 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:16.752955    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:47:16.753418 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:16.752916    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:47:16.753418 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:16.753030    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bl8d" podUID="f4699b63-6ba0-4401-a43e-57c15fea25b3"
Jul 01 20:47:16.753418 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:16.753128    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:47:16.753418 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:16.753202    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:47:18.000513 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:18.000482    2564 generic.go:356] "Generic (PLEG): container finished" podID="fb41c3f0-f0dc-4340-b524-bb23e5426370" containerID="864dadc5d9c51917e1c93a0b5b5d611d1b081d131fe1caabcb8d52f7f7e5b2b5" exitCode=0
Jul 01 20:47:18.000889 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:18.000541    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mfxlf" event={"ID":"fb41c3f0-f0dc-4340-b524-bb23e5426370","Type":"ContainerDied","Data":"864dadc5d9c51917e1c93a0b5b5d611d1b081d131fe1caabcb8d52f7f7e5b2b5"}
Jul 01 20:47:18.753097 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:18.753060    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:47:18.753097 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:18.753070    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:47:18.753304 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:18.753194    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:47:18.753304 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:18.753213    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:47:18.753304 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:18.753287    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:47:18.753406 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:18.753361    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bl8d" podUID="f4699b63-6ba0-4401-a43e-57c15fea25b3"
Jul 01 20:47:19.005287 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:19.005207    2564 generic.go:356] "Generic (PLEG): container finished" podID="fb41c3f0-f0dc-4340-b524-bb23e5426370" containerID="44707f7e562d9c99a6edb90ab8465249894862bfd4a5e55565bc47b6d18fae49" exitCode=0
Jul 01 20:47:19.005287 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:19.005251    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mfxlf" event={"ID":"fb41c3f0-f0dc-4340-b524-bb23e5426370","Type":"ContainerDied","Data":"44707f7e562d9c99a6edb90ab8465249894862bfd4a5e55565bc47b6d18fae49"}
Jul 01 20:47:20.011162 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:20.011124    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mfxlf" event={"ID":"fb41c3f0-f0dc-4340-b524-bb23e5426370","Type":"ContainerStarted","Data":"01d214e7cb18fdc51b63afaf794b293f6fdead28d0d6630d1a26017c40f4a29e"}
Jul 01 20:47:20.043924 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:20.043882    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mfxlf" podStartSLOduration=9.334344082 podStartE2EDuration="39.043850959s" totalImagesPullingTime="29.709506877s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:20.043739331 +0000 UTC m=+39.719754898" watchObservedRunningTime="2026-07-01 20:47:20.043850959 +0000 UTC m=+39.719866525"
Jul 01 20:47:20.754090 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:20.754062    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:47:20.754291 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:20.754143    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:47:20.754291 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:20.754234    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:47:20.754415 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:20.754325    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bl8d" podUID="f4699b63-6ba0-4401-a43e-57c15fea25b3"
Jul 01 20:47:20.754415 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:20.754374    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:47:20.754519 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:20.754486    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:47:22.753313 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:22.753269    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:47:22.753313 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:22.753293    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:47:22.753888 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:22.753269    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:47:22.753888 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:22.753373    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:47:22.753888 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:22.753433    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:47:22.753888 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:22.753484    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bl8d" podUID="f4699b63-6ba0-4401-a43e-57c15fea25b3"
Jul 01 20:47:24.753063 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:24.753026    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:47:24.753537 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:24.753036    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:47:24.753537 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:24.753158    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:47:24.753537 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:24.753036    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:47:24.753537 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:24.753206    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bl8d" podUID="f4699b63-6ba0-4401-a43e-57c15fea25b3"
Jul 01 20:47:24.753537 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:24.753287    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:47:26.753110 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:26.753081    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:47:26.753515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:26.753186    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:47:26.753515 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:26.753199    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bl8d" podUID="f4699b63-6ba0-4401-a43e-57c15fea25b3"
Jul 01 20:47:26.753515 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:26.753282    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:47:26.753515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:26.753320    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:47:26.753515 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:26.753377    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:47:28.752997 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:28.752962    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:47:28.753444 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:28.752963    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:47:28.753444 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:28.753159    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bl8d" podUID="f4699b63-6ba0-4401-a43e-57c15fea25b3"
Jul 01 20:47:28.753444 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:28.753056    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:47:28.753444 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:28.752963    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:47:28.753444 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:28.753276    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:47:30.755541 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:30.755513    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:47:30.755541 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:30.755531    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:47:30.756057 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:30.755522    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:47:30.756057 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:30.755617    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:47:30.756057 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:30.755673    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bl8d" podUID="f4699b63-6ba0-4401-a43e-57c15fea25b3"
Jul 01 20:47:30.756057 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:30.755780    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:47:31.512041 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:31.511800    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9xhz4"]
Jul 01 20:47:31.512204 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:31.512067    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:47:31.512204 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:31.512161    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:47:31.515536 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:31.515510    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9bl8d"]
Jul 01 20:47:31.515666 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:31.515603    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:47:31.515726 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:31.515690    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bl8d" podUID="f4699b63-6ba0-4401-a43e-57c15fea25b3"
Jul 01 20:47:31.516161 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:31.516131    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-64pmt"]
Jul 01 20:47:31.516272 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:31.516216    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:47:31.516334 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:31.516306    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:47:32.755661 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:32.755632    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:47:32.756277 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:32.755739    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:47:32.756277 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:32.755787    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:47:32.756277 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:32.755884    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:47:33.752758 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:33.752673    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:47:33.752949 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:33.752786    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bl8d" podUID="f4699b63-6ba0-4401-a43e-57c15fea25b3"
Jul 01 20:47:34.756085 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:34.756054    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:47:34.756470 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:34.756054    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:47:34.756470 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:34.756169    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pmt" podUID="ed4e44b6-7fbe-4e18-974e-edbc64ce0398"
Jul 01 20:47:34.756470 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:34.756241    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xhz4" podUID="c606e570-2fd6-43cb-9a84-9cdeab442815"
Jul 01 20:47:35.626761 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:35.626687    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-198.ec2.internal" event="NodeReady"
Jul 01 20:47:35.626916 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:35.626811    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 01 20:47:35.752731 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:35.752705    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:47:35.767805 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:35.767785    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 20:47:35.768228 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:35.767829    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 20:47:35.768228 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:35.768055    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-r694q\""
Jul 01 20:47:35.856290 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:35.856260    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ls26v"]
Jul 01 20:47:35.877749 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:35.877701    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ls26v"
Jul 01 20:47:35.893077 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:35.890572    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 01 20:47:35.893077 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:35.892125    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ls26v"]
Jul 01 20:47:35.896513 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:35.896493    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 01 20:47:35.898195 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:35.898168    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 01 20:47:35.907985 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:35.907963    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/912ca207-8e8d-42bf-b1b3-9d8c062a40a9-cert\") pod \"ingress-canary-ls26v\" (UID: \"912ca207-8e8d-42bf-b1b3-9d8c062a40a9\") " pod="openshift-ingress-canary/ingress-canary-ls26v"
Jul 01 20:47:35.908072 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:35.907993    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js98k\" (UniqueName: \"kubernetes.io/projected/912ca207-8e8d-42bf-b1b3-9d8c062a40a9-kube-api-access-js98k\") pod \"ingress-canary-ls26v\" (UID: \"912ca207-8e8d-42bf-b1b3-9d8c062a40a9\") " pod="openshift-ingress-canary/ingress-canary-ls26v"
Jul 01 20:47:35.966803 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:35.966782    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vpvmm\""
Jul 01 20:47:35.975614 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:35.975592    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zt5qb"]
Jul 01 20:47:35.992796 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:35.992771    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zt5qb"
Jul 01 20:47:35.996007 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:35.995990    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 01 20:47:35.998698 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:35.998678    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tfxg4\""
Jul 01 20:47:35.999363 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:35.999344    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9chmn"]
Jul 01 20:47:36.008425 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.008407    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/912ca207-8e8d-42bf-b1b3-9d8c062a40a9-cert\") pod \"ingress-canary-ls26v\" (UID: \"912ca207-8e8d-42bf-b1b3-9d8c062a40a9\") " pod="openshift-ingress-canary/ingress-canary-ls26v"
Jul 01 20:47:36.008504 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.008434    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js98k\" (UniqueName: \"kubernetes.io/projected/912ca207-8e8d-42bf-b1b3-9d8c062a40a9-kube-api-access-js98k\") pod \"ingress-canary-ls26v\" (UID: \"912ca207-8e8d-42bf-b1b3-9d8c062a40a9\") " pod="openshift-ingress-canary/ingress-canary-ls26v"
Jul 01 20:47:36.012359 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.012340    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/912ca207-8e8d-42bf-b1b3-9d8c062a40a9-cert\") pod \"ingress-canary-ls26v\" (UID: \"912ca207-8e8d-42bf-b1b3-9d8c062a40a9\") " pod="openshift-ingress-canary/ingress-canary-ls26v"
Jul 01 20:47:36.012946 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.012933    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9chmn"
Jul 01 20:47:36.017016 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.016997    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 01 20:47:36.018001 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.017985    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 01 20:47:36.018451 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.018432    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 01 20:47:36.019253 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.019233    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zt5qb"]
Jul 01 20:47:36.027027 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.027009    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 01 20:47:36.027210 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.027198    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j9f8h\""
Jul 01 20:47:36.034730 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.034614    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js98k\" (UniqueName: \"kubernetes.io/projected/912ca207-8e8d-42bf-b1b3-9d8c062a40a9-kube-api-access-js98k\") pod \"ingress-canary-ls26v\" (UID: \"912ca207-8e8d-42bf-b1b3-9d8c062a40a9\") " pod="openshift-ingress-canary/ingress-canary-ls26v"
Jul 01 20:47:36.037363 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.037346    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 01 20:47:36.072274 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.072246    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9chmn"]
Jul 01 20:47:36.108632 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.108605    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2-crio-socket\") pod \"insights-runtime-extractor-9chmn\" (UID: \"f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2\") " pod="openshift-insights/insights-runtime-extractor-9chmn"
Jul 01 20:47:36.108632 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.108633    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ed73c7-bfbc-427c-b455-736a3c19ec2e-config-volume\") pod \"dns-default-zt5qb\" (UID: \"f9ed73c7-bfbc-427c-b455-736a3c19ec2e\") " pod="openshift-dns/dns-default-zt5qb"
Jul 01 20:47:36.108798 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.108650    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2-data-volume\") pod \"insights-runtime-extractor-9chmn\" (UID: \"f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2\") " pod="openshift-insights/insights-runtime-extractor-9chmn"
Jul 01 20:47:36.108798 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.108749    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njwlq\" (UniqueName: \"kubernetes.io/projected/f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2-kube-api-access-njwlq\") pod \"insights-runtime-extractor-9chmn\" (UID: \"f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2\") " pod="openshift-insights/insights-runtime-extractor-9chmn"
Jul 01 20:47:36.108798 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.108786    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9ed73c7-bfbc-427c-b455-736a3c19ec2e-tmp-dir\") pod \"dns-default-zt5qb\" (UID: \"f9ed73c7-bfbc-427c-b455-736a3c19ec2e\") " pod="openshift-dns/dns-default-zt5qb"
Jul 01 20:47:36.108976 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.108818    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9chmn\" (UID: \"f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2\") " pod="openshift-insights/insights-runtime-extractor-9chmn"
Jul 01 20:47:36.108976 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.108845    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9chmn\" (UID: \"f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2\") " pod="openshift-insights/insights-runtime-extractor-9chmn"
Jul 01 20:47:36.108976 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.108921    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt6rw\" (UniqueName: \"kubernetes.io/projected/f9ed73c7-bfbc-427c-b455-736a3c19ec2e-kube-api-access-wt6rw\") pod \"dns-default-zt5qb\" (UID: \"f9ed73c7-bfbc-427c-b455-736a3c19ec2e\") " pod="openshift-dns/dns-default-zt5qb"
Jul 01 20:47:36.108976 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.108957    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9ed73c7-bfbc-427c-b455-736a3c19ec2e-metrics-tls\") pod \"dns-default-zt5qb\" (UID: \"f9ed73c7-bfbc-427c-b455-736a3c19ec2e\") " pod="openshift-dns/dns-default-zt5qb"
Jul 01 20:47:36.185582 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.185558    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ls26v"
Jul 01 20:47:36.209695 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.209666    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt6rw\" (UniqueName: \"kubernetes.io/projected/f9ed73c7-bfbc-427c-b455-736a3c19ec2e-kube-api-access-wt6rw\") pod \"dns-default-zt5qb\" (UID: \"f9ed73c7-bfbc-427c-b455-736a3c19ec2e\") " pod="openshift-dns/dns-default-zt5qb"
Jul 01 20:47:36.209781 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.209697    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9ed73c7-bfbc-427c-b455-736a3c19ec2e-metrics-tls\") pod \"dns-default-zt5qb\" (UID: \"f9ed73c7-bfbc-427c-b455-736a3c19ec2e\") " pod="openshift-dns/dns-default-zt5qb"
Jul 01 20:47:36.209781 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.209722    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2-crio-socket\") pod \"insights-runtime-extractor-9chmn\" (UID: \"f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2\") " pod="openshift-insights/insights-runtime-extractor-9chmn"
Jul 01 20:47:36.209781 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.209745    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ed73c7-bfbc-427c-b455-736a3c19ec2e-config-volume\") pod \"dns-default-zt5qb\" (UID: \"f9ed73c7-bfbc-427c-b455-736a3c19ec2e\") " pod="openshift-dns/dns-default-zt5qb"
Jul 01 20:47:36.209781 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.209769    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2-data-volume\") pod \"insights-runtime-extractor-9chmn\" (UID: \"f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2\") " pod="openshift-insights/insights-runtime-extractor-9chmn"
Jul 01 20:47:36.210024 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.209824    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njwlq\" (UniqueName: \"kubernetes.io/projected/f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2-kube-api-access-njwlq\") pod \"insights-runtime-extractor-9chmn\" (UID: \"f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2\") " pod="openshift-insights/insights-runtime-extractor-9chmn"
Jul 01 20:47:36.210024 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.209851    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9ed73c7-bfbc-427c-b455-736a3c19ec2e-tmp-dir\") pod \"dns-default-zt5qb\" (UID: \"f9ed73c7-bfbc-427c-b455-736a3c19ec2e\") " pod="openshift-dns/dns-default-zt5qb"
Jul 01 20:47:36.210024 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.209897    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9chmn\" (UID: \"f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2\") " pod="openshift-insights/insights-runtime-extractor-9chmn"
Jul 01 20:47:36.210024 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.209903    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2-crio-socket\") pod \"insights-runtime-extractor-9chmn\" (UID: \"f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2\") " pod="openshift-insights/insights-runtime-extractor-9chmn"
Jul 01 20:47:36.210024 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.209920    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9chmn\" (UID: \"f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2\") " pod="openshift-insights/insights-runtime-extractor-9chmn"
Jul 01 20:47:36.210300 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.210281    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9ed73c7-bfbc-427c-b455-736a3c19ec2e-tmp-dir\") pod \"dns-default-zt5qb\" (UID: \"f9ed73c7-bfbc-427c-b455-736a3c19ec2e\") " pod="openshift-dns/dns-default-zt5qb"
Jul 01 20:47:36.210356 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.210296    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2-data-volume\") pod \"insights-runtime-extractor-9chmn\" (UID: \"f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2\") " pod="openshift-insights/insights-runtime-extractor-9chmn"
Jul 01 20:47:36.210409 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.210364    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ed73c7-bfbc-427c-b455-736a3c19ec2e-config-volume\") pod \"dns-default-zt5qb\" (UID: \"f9ed73c7-bfbc-427c-b455-736a3c19ec2e\") " pod="openshift-dns/dns-default-zt5qb"
Jul 01 20:47:36.210716 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.210694    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9chmn\" (UID: \"f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2\") " pod="openshift-insights/insights-runtime-extractor-9chmn"
Jul 01 20:47:36.212657 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.212637    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9ed73c7-bfbc-427c-b455-736a3c19ec2e-metrics-tls\") pod \"dns-default-zt5qb\" (UID: \"f9ed73c7-bfbc-427c-b455-736a3c19ec2e\") " pod="openshift-dns/dns-default-zt5qb"
Jul 01 20:47:36.212812 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.212792    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9chmn\" (UID: \"f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2\") " pod="openshift-insights/insights-runtime-extractor-9chmn"
Jul 01 20:47:36.225387 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.225365    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njwlq\" (UniqueName: \"kubernetes.io/projected/f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2-kube-api-access-njwlq\") pod \"insights-runtime-extractor-9chmn\" (UID: \"f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2\") " pod="openshift-insights/insights-runtime-extractor-9chmn"
Jul 01 20:47:36.225473 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.225418    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt6rw\" (UniqueName: \"kubernetes.io/projected/f9ed73c7-bfbc-427c-b455-736a3c19ec2e-kube-api-access-wt6rw\") pod \"dns-default-zt5qb\" (UID: \"f9ed73c7-bfbc-427c-b455-736a3c19ec2e\") " pod="openshift-dns/dns-default-zt5qb"
Jul 01 20:47:36.301706 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.301670    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zt5qb"
Jul 01 20:47:36.323087 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.323061    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9chmn"
Jul 01 20:47:36.377052 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.377017    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ls26v"]
Jul 01 20:47:36.383602 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:47:36.383556    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod912ca207_8e8d_42bf_b1b3_9d8c062a40a9.slice/crio-42c3bcca639b418f88ad497a0c66f200b95dbf1c1651ca7e9e420e954093f2a0 WatchSource:0}: Error finding container 42c3bcca639b418f88ad497a0c66f200b95dbf1c1651ca7e9e420e954093f2a0: Status 404 returned error can't find the container with id 42c3bcca639b418f88ad497a0c66f200b95dbf1c1651ca7e9e420e954093f2a0
Jul 01 20:47:36.463948 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.463845    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zt5qb"]
Jul 01 20:47:36.467449 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:47:36.467421    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ed73c7_bfbc_427c_b455_736a3c19ec2e.slice/crio-92e8d87dae9e46c7b638aebcf209ecb90918be3b5dd44063c7628e4faefe0e89 WatchSource:0}: Error finding container 92e8d87dae9e46c7b638aebcf209ecb90918be3b5dd44063c7628e4faefe0e89: Status 404 returned error can't find the container with id 92e8d87dae9e46c7b638aebcf209ecb90918be3b5dd44063c7628e4faefe0e89
Jul 01 20:47:36.478936 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.478913    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9chmn"]
Jul 01 20:47:36.481725 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:47:36.481704    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c1cd54_42a7_40c2_bb7f_1ac7ed095eb2.slice/crio-9f8b7e50f1c224de7a879edb118e469216036981465b96af7c217e4463a97745 WatchSource:0}: Error finding container 9f8b7e50f1c224de7a879edb118e469216036981465b96af7c217e4463a97745: Status 404 returned error can't find the container with id 9f8b7e50f1c224de7a879edb118e469216036981465b96af7c217e4463a97745
Jul 01 20:47:36.753332 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.753164    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:47:36.753481 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.753362    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:47:36.758355 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.758335    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 20:47:36.758722 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.758699    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 20:47:36.758963 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:36.758709    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7lgrj\""
Jul 01 20:47:37.054601 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:37.054411    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9chmn" event={"ID":"f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2","Type":"ContainerStarted","Data":"9f8b7e50f1c224de7a879edb118e469216036981465b96af7c217e4463a97745"}
Jul 01 20:47:37.058585 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:37.058516    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zt5qb" event={"ID":"f9ed73c7-bfbc-427c-b455-736a3c19ec2e","Type":"ContainerStarted","Data":"92e8d87dae9e46c7b638aebcf209ecb90918be3b5dd44063c7628e4faefe0e89"}
Jul 01 20:47:37.060020 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:37.059965    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ls26v" event={"ID":"912ca207-8e8d-42bf-b1b3-9d8c062a40a9","Type":"ContainerStarted","Data":"42c3bcca639b418f88ad497a0c66f200b95dbf1c1651ca7e9e420e954093f2a0"}
Jul 01 20:47:41.108775 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.108743    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-md7qm"]
Jul 01 20:47:41.110726 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.110705    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-md7qm"
Jul 01 20:47:41.113673 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.113647    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 01 20:47:41.118031 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.115046    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 01 20:47:41.118031 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.115309    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 01 20:47:41.118031 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.115595    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 01 20:47:41.118031 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.115924    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-nltf9\""
Jul 01 20:47:41.118031 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.116133    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 01 20:47:41.124337 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.124269    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-md7qm"]
Jul 01 20:47:41.149797 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.149764    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edb0e1f0-4d38-4af1-80a9-7a2e362797c6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-md7qm\" (UID: \"edb0e1f0-4d38-4af1-80a9-7a2e362797c6\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-md7qm"
Jul 01 20:47:41.149973 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.149835    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edb0e1f0-4d38-4af1-80a9-7a2e362797c6-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-md7qm\" (UID: \"edb0e1f0-4d38-4af1-80a9-7a2e362797c6\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-md7qm"
Jul 01 20:47:41.150045 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.149966    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr48p\" (UniqueName: \"kubernetes.io/projected/edb0e1f0-4d38-4af1-80a9-7a2e362797c6-kube-api-access-vr48p\") pod \"openshift-state-metrics-57f99c8b49-md7qm\" (UID: \"edb0e1f0-4d38-4af1-80a9-7a2e362797c6\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-md7qm"
Jul 01 20:47:41.150045 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.150012    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/edb0e1f0-4d38-4af1-80a9-7a2e362797c6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-md7qm\" (UID: \"edb0e1f0-4d38-4af1-80a9-7a2e362797c6\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-md7qm"
Jul 01 20:47:41.161659 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.161607    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-22nc7"]
Jul 01 20:47:41.163898 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.163870    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-x5xnh"]
Jul 01 20:47:41.164012 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.163987    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-22nc7"
Jul 01 20:47:41.165751 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.165721    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.167107 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.167082    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 01 20:47:41.167539 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.167520    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 01 20:47:41.167728 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.167708    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 01 20:47:41.167896 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.167874    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5l47j\""
Jul 01 20:47:41.168775 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.168757    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 01 20:47:41.168893 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.168793    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 01 20:47:41.169109 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.169089    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 01 20:47:41.169271 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.169253    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v8blg\""
Jul 01 20:47:41.181936 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.181916    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-22nc7"]
Jul 01 20:47:41.250896 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.250844    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/83c9de00-f066-4aa4-900c-4e0d6c5042ca-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-22nc7\" (UID: \"83c9de00-f066-4aa4-900c-4e0d6c5042ca\") " pod="openshift-monitoring/kube-state-metrics-845b954455-22nc7"
Jul 01 20:47:41.250896 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.250895    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/83c9de00-f066-4aa4-900c-4e0d6c5042ca-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-22nc7\" (UID: \"83c9de00-f066-4aa4-900c-4e0d6c5042ca\") " pod="openshift-monitoring/kube-state-metrics-845b954455-22nc7"
Jul 01 20:47:41.251136 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.250965    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/edb0e1f0-4d38-4af1-80a9-7a2e362797c6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-md7qm\" (UID: \"edb0e1f0-4d38-4af1-80a9-7a2e362797c6\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-md7qm"
Jul 01 20:47:41.251136 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.251002    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/866f6082-54b0-4b03-8d36-69dc77a6ca61-root\") pod \"node-exporter-x5xnh\" (UID: \"866f6082-54b0-4b03-8d36-69dc77a6ca61\") " pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.251136 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.251034    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edb0e1f0-4d38-4af1-80a9-7a2e362797c6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-md7qm\" (UID: \"edb0e1f0-4d38-4af1-80a9-7a2e362797c6\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-md7qm"
Jul 01 20:47:41.251136 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.251060    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/866f6082-54b0-4b03-8d36-69dc77a6ca61-sys\") pod \"node-exporter-x5xnh\" (UID: \"866f6082-54b0-4b03-8d36-69dc77a6ca61\") " pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.251136 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.251118    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/866f6082-54b0-4b03-8d36-69dc77a6ca61-node-exporter-accelerators-collector-config\") pod \"node-exporter-x5xnh\" (UID: \"866f6082-54b0-4b03-8d36-69dc77a6ca61\") " pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.251370 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.251154    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83c9de00-f066-4aa4-900c-4e0d6c5042ca-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-22nc7\" (UID: \"83c9de00-f066-4aa4-900c-4e0d6c5042ca\") " pod="openshift-monitoring/kube-state-metrics-845b954455-22nc7"
Jul 01 20:47:41.251370 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.251199    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/866f6082-54b0-4b03-8d36-69dc77a6ca61-metrics-client-ca\") pod \"node-exporter-x5xnh\" (UID: \"866f6082-54b0-4b03-8d36-69dc77a6ca61\") " pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.251370 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.251255    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/866f6082-54b0-4b03-8d36-69dc77a6ca61-node-exporter-tls\") pod \"node-exporter-x5xnh\" (UID: \"866f6082-54b0-4b03-8d36-69dc77a6ca61\") " pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.251370 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.251306    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83c9de00-f066-4aa4-900c-4e0d6c5042ca-metrics-client-ca\") pod \"kube-state-metrics-845b954455-22nc7\" (UID: \"83c9de00-f066-4aa4-900c-4e0d6c5042ca\") " pod="openshift-monitoring/kube-state-metrics-845b954455-22nc7"
Jul 01 20:47:41.251370 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.251346    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr48p\" (UniqueName: \"kubernetes.io/projected/edb0e1f0-4d38-4af1-80a9-7a2e362797c6-kube-api-access-vr48p\") pod \"openshift-state-metrics-57f99c8b49-md7qm\" (UID: \"edb0e1f0-4d38-4af1-80a9-7a2e362797c6\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-md7qm"
Jul 01 20:47:41.251612 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.251373    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/866f6082-54b0-4b03-8d36-69dc77a6ca61-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x5xnh\" (UID: \"866f6082-54b0-4b03-8d36-69dc77a6ca61\") " pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.251612 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.251400    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/866f6082-54b0-4b03-8d36-69dc77a6ca61-node-exporter-wtmp\") pod \"node-exporter-x5xnh\" (UID: \"866f6082-54b0-4b03-8d36-69dc77a6ca61\") " pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.251612 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.251429    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/83c9de00-f066-4aa4-900c-4e0d6c5042ca-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-22nc7\" (UID: \"83c9de00-f066-4aa4-900c-4e0d6c5042ca\") " pod="openshift-monitoring/kube-state-metrics-845b954455-22nc7"
Jul 01 20:47:41.251612 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.251488    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edb0e1f0-4d38-4af1-80a9-7a2e362797c6-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-md7qm\" (UID: \"edb0e1f0-4d38-4af1-80a9-7a2e362797c6\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-md7qm"
Jul 01 20:47:41.251767 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.251623    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsmt9\" (UniqueName: \"kubernetes.io/projected/866f6082-54b0-4b03-8d36-69dc77a6ca61-kube-api-access-tsmt9\") pod \"node-exporter-x5xnh\" (UID: \"866f6082-54b0-4b03-8d36-69dc77a6ca61\") " pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.251767 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.251647    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/866f6082-54b0-4b03-8d36-69dc77a6ca61-node-exporter-textfile\") pod \"node-exporter-x5xnh\" (UID: \"866f6082-54b0-4b03-8d36-69dc77a6ca61\") " pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.251767 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.251672    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9xmx\" (UniqueName: \"kubernetes.io/projected/83c9de00-f066-4aa4-900c-4e0d6c5042ca-kube-api-access-r9xmx\") pod \"kube-state-metrics-845b954455-22nc7\" (UID: \"83c9de00-f066-4aa4-900c-4e0d6c5042ca\") " pod="openshift-monitoring/kube-state-metrics-845b954455-22nc7"
Jul 01 20:47:41.252181 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.252159    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edb0e1f0-4d38-4af1-80a9-7a2e362797c6-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-md7qm\" (UID: \"edb0e1f0-4d38-4af1-80a9-7a2e362797c6\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-md7qm"
Jul 01 20:47:41.254496 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.254471    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edb0e1f0-4d38-4af1-80a9-7a2e362797c6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-md7qm\" (UID: \"edb0e1f0-4d38-4af1-80a9-7a2e362797c6\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-md7qm"
Jul 01 20:47:41.254603 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.254482    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/edb0e1f0-4d38-4af1-80a9-7a2e362797c6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-md7qm\" (UID: \"edb0e1f0-4d38-4af1-80a9-7a2e362797c6\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-md7qm"
Jul 01 20:47:41.283273 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.283248    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr48p\" (UniqueName: \"kubernetes.io/projected/edb0e1f0-4d38-4af1-80a9-7a2e362797c6-kube-api-access-vr48p\") pod \"openshift-state-metrics-57f99c8b49-md7qm\" (UID: \"edb0e1f0-4d38-4af1-80a9-7a2e362797c6\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-md7qm"
Jul 01 20:47:41.352578 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.352546    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/83c9de00-f066-4aa4-900c-4e0d6c5042ca-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-22nc7\" (UID: \"83c9de00-f066-4aa4-900c-4e0d6c5042ca\") " pod="openshift-monitoring/kube-state-metrics-845b954455-22nc7"
Jul 01 20:47:41.352578 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.352591    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/866f6082-54b0-4b03-8d36-69dc77a6ca61-root\") pod \"node-exporter-x5xnh\" (UID: \"866f6082-54b0-4b03-8d36-69dc77a6ca61\") " pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.352827 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.352624    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/866f6082-54b0-4b03-8d36-69dc77a6ca61-sys\") pod \"node-exporter-x5xnh\" (UID: \"866f6082-54b0-4b03-8d36-69dc77a6ca61\") " pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.352827 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.352658    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/866f6082-54b0-4b03-8d36-69dc77a6ca61-node-exporter-accelerators-collector-config\") pod \"node-exporter-x5xnh\" (UID: \"866f6082-54b0-4b03-8d36-69dc77a6ca61\") " pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.352827 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.352685    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/866f6082-54b0-4b03-8d36-69dc77a6ca61-root\") pod \"node-exporter-x5xnh\" (UID: \"866f6082-54b0-4b03-8d36-69dc77a6ca61\") " pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.352827 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.352700    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83c9de00-f066-4aa4-900c-4e0d6c5042ca-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-22nc7\" (UID: \"83c9de00-f066-4aa4-900c-4e0d6c5042ca\") " pod="openshift-monitoring/kube-state-metrics-845b954455-22nc7"
Jul 01 20:47:41.352827 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.352775    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/866f6082-54b0-4b03-8d36-69dc77a6ca61-metrics-client-ca\") pod \"node-exporter-x5xnh\" (UID: \"866f6082-54b0-4b03-8d36-69dc77a6ca61\") " pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.352827 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.352818    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/866f6082-54b0-4b03-8d36-69dc77a6ca61-node-exporter-tls\") pod \"node-exporter-x5xnh\" (UID: \"866f6082-54b0-4b03-8d36-69dc77a6ca61\") " pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.353154 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.352885    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83c9de00-f066-4aa4-900c-4e0d6c5042ca-metrics-client-ca\") pod \"kube-state-metrics-845b954455-22nc7\" (UID: \"83c9de00-f066-4aa4-900c-4e0d6c5042ca\") " pod="openshift-monitoring/kube-state-metrics-845b954455-22nc7"
Jul 01 20:47:41.353154 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.352920    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/866f6082-54b0-4b03-8d36-69dc77a6ca61-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x5xnh\" (UID: \"866f6082-54b0-4b03-8d36-69dc77a6ca61\") " pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.353154 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.352947    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/866f6082-54b0-4b03-8d36-69dc77a6ca61-node-exporter-wtmp\") pod \"node-exporter-x5xnh\" (UID: \"866f6082-54b0-4b03-8d36-69dc77a6ca61\") " pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.353154 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.352979    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/83c9de00-f066-4aa4-900c-4e0d6c5042ca-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-22nc7\" (UID: \"83c9de00-f066-4aa4-900c-4e0d6c5042ca\") " pod="openshift-monitoring/kube-state-metrics-845b954455-22nc7"
Jul 01 20:47:41.353154 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.353022    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/83c9de00-f066-4aa4-900c-4e0d6c5042ca-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-22nc7\" (UID: \"83c9de00-f066-4aa4-900c-4e0d6c5042ca\") " pod="openshift-monitoring/kube-state-metrics-845b954455-22nc7"
Jul 01 20:47:41.353154 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.353071    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsmt9\" (UniqueName: \"kubernetes.io/projected/866f6082-54b0-4b03-8d36-69dc77a6ca61-kube-api-access-tsmt9\") pod \"node-exporter-x5xnh\" (UID: \"866f6082-54b0-4b03-8d36-69dc77a6ca61\") " pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.353154 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.353095    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/866f6082-54b0-4b03-8d36-69dc77a6ca61-node-exporter-textfile\") pod \"node-exporter-x5xnh\" (UID: \"866f6082-54b0-4b03-8d36-69dc77a6ca61\") " pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.353154 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.353118    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9xmx\" (UniqueName: \"kubernetes.io/projected/83c9de00-f066-4aa4-900c-4e0d6c5042ca-kube-api-access-r9xmx\") pod \"kube-state-metrics-845b954455-22nc7\" (UID: \"83c9de00-f066-4aa4-900c-4e0d6c5042ca\") " pod="openshift-monitoring/kube-state-metrics-845b954455-22nc7"
Jul 01 20:47:41.353532 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.353156    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/83c9de00-f066-4aa4-900c-4e0d6c5042ca-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-22nc7\" (UID: \"83c9de00-f066-4aa4-900c-4e0d6c5042ca\") " pod="openshift-monitoring/kube-state-metrics-845b954455-22nc7"
Jul 01 20:47:41.353532 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.353333    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/866f6082-54b0-4b03-8d36-69dc77a6ca61-node-exporter-accelerators-collector-config\") pod \"node-exporter-x5xnh\" (UID: \"866f6082-54b0-4b03-8d36-69dc77a6ca61\") " pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.353532 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.353391    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/866f6082-54b0-4b03-8d36-69dc77a6ca61-sys\") pod \"node-exporter-x5xnh\" (UID: \"866f6082-54b0-4b03-8d36-69dc77a6ca61\") " pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.353532 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:41.353456    2564 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 01 20:47:41.353532 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:47:41.353509    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83c9de00-f066-4aa4-900c-4e0d6c5042ca-kube-state-metrics-tls podName:83c9de00-f066-4aa4-900c-4e0d6c5042ca nodeName:}" failed. No retries permitted until 2026-07-01 20:47:41.853491242 +0000 UTC m=+61.529506809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/83c9de00-f066-4aa4-900c-4e0d6c5042ca-kube-state-metrics-tls") pod "kube-state-metrics-845b954455-22nc7" (UID: "83c9de00-f066-4aa4-900c-4e0d6c5042ca") : secret "kube-state-metrics-tls" not found
Jul 01 20:47:41.353847 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.353628    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/866f6082-54b0-4b03-8d36-69dc77a6ca61-node-exporter-wtmp\") pod \"node-exporter-x5xnh\" (UID: \"866f6082-54b0-4b03-8d36-69dc77a6ca61\") " pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.353847 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.353717    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83c9de00-f066-4aa4-900c-4e0d6c5042ca-metrics-client-ca\") pod \"kube-state-metrics-845b954455-22nc7\" (UID: \"83c9de00-f066-4aa4-900c-4e0d6c5042ca\") " pod="openshift-monitoring/kube-state-metrics-845b954455-22nc7"
Jul 01 20:47:41.353847 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.353786    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/83c9de00-f066-4aa4-900c-4e0d6c5042ca-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-22nc7\" (UID: \"83c9de00-f066-4aa4-900c-4e0d6c5042ca\") " pod="openshift-monitoring/kube-state-metrics-845b954455-22nc7"
Jul 01 20:47:41.353847 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.353833    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/866f6082-54b0-4b03-8d36-69dc77a6ca61-metrics-client-ca\") pod \"node-exporter-x5xnh\" (UID: \"866f6082-54b0-4b03-8d36-69dc77a6ca61\") " pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.354075 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.353935    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/866f6082-54b0-4b03-8d36-69dc77a6ca61-node-exporter-textfile\") pod \"node-exporter-x5xnh\" (UID: \"866f6082-54b0-4b03-8d36-69dc77a6ca61\") " pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.355640 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.355592    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83c9de00-f066-4aa4-900c-4e0d6c5042ca-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-22nc7\" (UID: \"83c9de00-f066-4aa4-900c-4e0d6c5042ca\") " pod="openshift-monitoring/kube-state-metrics-845b954455-22nc7"
Jul 01 20:47:41.355908 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.355887    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/866f6082-54b0-4b03-8d36-69dc77a6ca61-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x5xnh\" (UID: \"866f6082-54b0-4b03-8d36-69dc77a6ca61\") " pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.356080 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.356055    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/866f6082-54b0-4b03-8d36-69dc77a6ca61-node-exporter-tls\") pod \"node-exporter-x5xnh\" (UID: \"866f6082-54b0-4b03-8d36-69dc77a6ca61\") " pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.375806 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.375752    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9xmx\" (UniqueName: \"kubernetes.io/projected/83c9de00-f066-4aa4-900c-4e0d6c5042ca-kube-api-access-r9xmx\") pod \"kube-state-metrics-845b954455-22nc7\" (UID: \"83c9de00-f066-4aa4-900c-4e0d6c5042ca\") " pod="openshift-monitoring/kube-state-metrics-845b954455-22nc7"
Jul 01 20:47:41.380746 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.380728    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsmt9\" (UniqueName: \"kubernetes.io/projected/866f6082-54b0-4b03-8d36-69dc77a6ca61-kube-api-access-tsmt9\") pod \"node-exporter-x5xnh\" (UID: \"866f6082-54b0-4b03-8d36-69dc77a6ca61\") " pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.420612 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.420563    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-md7qm"
Jul 01 20:47:41.480903 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.480842    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x5xnh"
Jul 01 20:47:41.505086 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:47:41.505053    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod866f6082_54b0_4b03_8d36_69dc77a6ca61.slice/crio-d721409919a2982cc391e37d4f05f63a17652e753bcb2e75ad6b07f8493b40ef WatchSource:0}: Error finding container d721409919a2982cc391e37d4f05f63a17652e753bcb2e75ad6b07f8493b40ef: Status 404 returned error can't find the container with id d721409919a2982cc391e37d4f05f63a17652e753bcb2e75ad6b07f8493b40ef
Jul 01 20:47:41.582162 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.582117    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-md7qm"]
Jul 01 20:47:41.590395 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:47:41.590366    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb0e1f0_4d38_4af1_80a9_7a2e362797c6.slice/crio-c3877e50e9fc3c52b298bc9894abaf98ea611adc01632e46b0f184c973700128 WatchSource:0}: Error finding container c3877e50e9fc3c52b298bc9894abaf98ea611adc01632e46b0f184c973700128: Status 404 returned error can't find the container with id c3877e50e9fc3c52b298bc9894abaf98ea611adc01632e46b0f184c973700128
Jul 01 20:47:41.856993 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.856954    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/83c9de00-f066-4aa4-900c-4e0d6c5042ca-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-22nc7\" (UID: \"83c9de00-f066-4aa4-900c-4e0d6c5042ca\") " pod="openshift-monitoring/kube-state-metrics-845b954455-22nc7"
Jul 01 20:47:41.859377 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.859348    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/83c9de00-f066-4aa4-900c-4e0d6c5042ca-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-22nc7\" (UID: \"83c9de00-f066-4aa4-900c-4e0d6c5042ca\") " pod="openshift-monitoring/kube-state-metrics-845b954455-22nc7"
Jul 01 20:47:41.995167 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:41.995137    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vfs6k"
Jul 01 20:47:42.074736 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.074547    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-22nc7"
Jul 01 20:47:42.081389 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.081349    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zt5qb" event={"ID":"f9ed73c7-bfbc-427c-b455-736a3c19ec2e","Type":"ContainerStarted","Data":"800df823aa12edff599bb4f5630f5e206bf42412dcdd0904a052c7d96800d94f"}
Jul 01 20:47:42.081557 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.081452    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zt5qb" event={"ID":"f9ed73c7-bfbc-427c-b455-736a3c19ec2e","Type":"ContainerStarted","Data":"b79bdaaed6963ee7918ca34c2cbca07dddf6bcfd4b487322c5a6595a5f6004bc"}
Jul 01 20:47:42.081628 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.081608    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zt5qb"
Jul 01 20:47:42.084964 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.084889    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-md7qm" event={"ID":"edb0e1f0-4d38-4af1-80a9-7a2e362797c6","Type":"ContainerStarted","Data":"c74a2aca68fe215d8c8cc8004b4da9018709baf00808e40a12d450a677f2d4bd"}
Jul 01 20:47:42.084964 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.084951    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-md7qm" event={"ID":"edb0e1f0-4d38-4af1-80a9-7a2e362797c6","Type":"ContainerStarted","Data":"46aaee1174d25d9d20cd0f57d5f073ef8f982201ad45abedc5a6f50daea8bd1c"}
Jul 01 20:47:42.085140 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.084982    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-md7qm" event={"ID":"edb0e1f0-4d38-4af1-80a9-7a2e362797c6","Type":"ContainerStarted","Data":"c3877e50e9fc3c52b298bc9894abaf98ea611adc01632e46b0f184c973700128"}
Jul 01 20:47:42.087088 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.087055    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x5xnh" event={"ID":"866f6082-54b0-4b03-8d36-69dc77a6ca61","Type":"ContainerStarted","Data":"d721409919a2982cc391e37d4f05f63a17652e753bcb2e75ad6b07f8493b40ef"}
Jul 01 20:47:42.088902 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.088877    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9chmn" event={"ID":"f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2","Type":"ContainerStarted","Data":"b9218031d2add83afd9a2289805eae510f91138c9a402f26eaf79cb111284b4f"}
Jul 01 20:47:42.090818 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.090792    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ls26v" event={"ID":"912ca207-8e8d-42bf-b1b3-9d8c062a40a9","Type":"ContainerStarted","Data":"363d0eb8afb9dac70bb0f27ddb12761e203e17fd7ea7c1152bc459690a365d64"}
Jul 01 20:47:42.103981 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.103937    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zt5qb" podStartSLOduration=2.138762968 podStartE2EDuration="7.103919173s" totalImagesPullingTime="4.965156205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:42.102740067 +0000 UTC m=+61.778755646" watchObservedRunningTime="2026-07-01 20:47:42.103919173 +0000 UTC m=+61.779934741"
Jul 01 20:47:42.121511 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.121461    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ls26v" podStartSLOduration=2.069189851 podStartE2EDuration="7.121440077s" totalImagesPullingTime="5.052250226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:42.120104907 +0000 UTC m=+61.796120475" watchObservedRunningTime="2026-07-01 20:47:42.121440077 +0000 UTC m=+61.797455645"
Jul 01 20:47:42.238843 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.238813    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-22nc7"]
Jul 01 20:47:42.253302 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.253238    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 20:47:42.278152 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.278117    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.282898 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.282872    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 01 20:47:42.283031 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.282904    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 01 20:47:42.283436 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.283379    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 01 20:47:42.283436 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.283389    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 01 20:47:42.283646 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.283608    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 01 20:47:42.283989 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.283956    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 01 20:47:42.284226 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.284204    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 01 20:47:42.284320 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.284223    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 01 20:47:42.284320 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.284240    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ldxkw\""
Jul 01 20:47:42.284320 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.284312    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 20:47:42.284464 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.284204    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 01 20:47:42.333141 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:47:42.333106    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c9de00_f066_4aa4_900c_4e0d6c5042ca.slice/crio-3f8c590e288b79428b937e908c11e4a70f431d7e8eb0349abd3c7c9894484c63 WatchSource:0}: Error finding container 3f8c590e288b79428b937e908c11e4a70f431d7e8eb0349abd3c7c9894484c63: Status 404 returned error can't find the container with id 3f8c590e288b79428b937e908c11e4a70f431d7e8eb0349abd3c7c9894484c63
Jul 01 20:47:42.362796 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.362762    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de41a7ac-ca29-46f1-8145-f294eb2cbd56-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.362988 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.362812    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.362988 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.362845    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/de41a7ac-ca29-46f1-8145-f294eb2cbd56-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.362988 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.362961    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.363154 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.362994    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de41a7ac-ca29-46f1-8145-f294eb2cbd56-config-out\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.363154 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.363045    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de41a7ac-ca29-46f1-8145-f294eb2cbd56-tls-assets\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.363154 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.363077    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-web-config\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.363154 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.363102    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-config-volume\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.363154 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.363124    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.363395 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.363207    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de41a7ac-ca29-46f1-8145-f294eb2cbd56-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.363395 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.363233    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r7dw\" (UniqueName: \"kubernetes.io/projected/de41a7ac-ca29-46f1-8145-f294eb2cbd56-kube-api-access-6r7dw\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.363395 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.363259    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.363395 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.363295    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.464108 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.464070    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de41a7ac-ca29-46f1-8145-f294eb2cbd56-config-out\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.464360 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.464134    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de41a7ac-ca29-46f1-8145-f294eb2cbd56-tls-assets\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.464360 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.464171    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-web-config\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.464484 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.464451    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-config-volume\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.464550 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.464494    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.464550 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.464543    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de41a7ac-ca29-46f1-8145-f294eb2cbd56-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.464640 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.464571    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r7dw\" (UniqueName: \"kubernetes.io/projected/de41a7ac-ca29-46f1-8145-f294eb2cbd56-kube-api-access-6r7dw\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.464640 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.464601    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.464727 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.464637    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.464727 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.464678    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de41a7ac-ca29-46f1-8145-f294eb2cbd56-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.464727 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.464712    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.464887 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.464736    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/de41a7ac-ca29-46f1-8145-f294eb2cbd56-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.464887 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.464808    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.466399 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.465591    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/de41a7ac-ca29-46f1-8145-f294eb2cbd56-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.466399 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.466085    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de41a7ac-ca29-46f1-8145-f294eb2cbd56-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.466399 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.466188    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de41a7ac-ca29-46f1-8145-f294eb2cbd56-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.466874 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.466819    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de41a7ac-ca29-46f1-8145-f294eb2cbd56-config-out\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.467812 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.467772    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de41a7ac-ca29-46f1-8145-f294eb2cbd56-tls-assets\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.468350 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.468296    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-web-config\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.469010 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.468986    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.469763 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.469659    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.470413 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.470287    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-config-volume\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.470413 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.470344    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.470693 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.470668    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.471118 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.471095    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.481714 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.481670    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r7dw\" (UniqueName: \"kubernetes.io/projected/de41a7ac-ca29-46f1-8145-f294eb2cbd56-kube-api-access-6r7dw\") pod \"alertmanager-main-0\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:42.588315 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:42.588245    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:47:43.095699 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:43.095657    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-22nc7" event={"ID":"83c9de00-f066-4aa4-900c-4e0d6c5042ca","Type":"ContainerStarted","Data":"3f8c590e288b79428b937e908c11e4a70f431d7e8eb0349abd3c7c9894484c63"}
Jul 01 20:47:43.683534 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:43.683508    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 20:47:43.755829 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:47:43.755736    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde41a7ac_ca29_46f1_8145_f294eb2cbd56.slice/crio-3d1e1c8c35169ea426b285c4c6b0e34b57d277683061ee1173b3ae250bf89db0 WatchSource:0}: Error finding container 3d1e1c8c35169ea426b285c4c6b0e34b57d277683061ee1173b3ae250bf89db0: Status 404 returned error can't find the container with id 3d1e1c8c35169ea426b285c4c6b0e34b57d277683061ee1173b3ae250bf89db0
Jul 01 20:47:44.101790 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:44.101694    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9chmn" event={"ID":"f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2","Type":"ContainerStarted","Data":"9671e79c29ada5b89c7495123a5b56e3eb0aa95100095558bd56ed7cf770b65e"}
Jul 01 20:47:44.103984 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:44.103955    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-md7qm" event={"ID":"edb0e1f0-4d38-4af1-80a9-7a2e362797c6","Type":"ContainerStarted","Data":"3268cc6c6f2793487329c11590ab5a1dd29016cf118a94d674c535a4c6ce93bc"}
Jul 01 20:47:44.105360 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:44.105328    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de41a7ac-ca29-46f1-8145-f294eb2cbd56","Type":"ContainerStarted","Data":"3d1e1c8c35169ea426b285c4c6b0e34b57d277683061ee1173b3ae250bf89db0"}
Jul 01 20:47:44.106996 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:44.106974    2564 generic.go:356] "Generic (PLEG): container finished" podID="866f6082-54b0-4b03-8d36-69dc77a6ca61" containerID="5f2285f972d989e045a4737e26e2f83ccddc708c69a1e2eda826d6716e406a3b" exitCode=0
Jul 01 20:47:44.107104 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:44.107004    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x5xnh" event={"ID":"866f6082-54b0-4b03-8d36-69dc77a6ca61","Type":"ContainerDied","Data":"5f2285f972d989e045a4737e26e2f83ccddc708c69a1e2eda826d6716e406a3b"}
Jul 01 20:47:44.130216 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:44.130168    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-md7qm" podStartSLOduration=1.341350281 podStartE2EDuration="3.130152476s" totalImagesPullingTime="1.788802195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:44.129183292 +0000 UTC m=+63.805198901" watchObservedRunningTime="2026-07-01 20:47:44.130152476 +0000 UTC m=+63.806168033"
Jul 01 20:47:45.113555 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:45.113508    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-22nc7" event={"ID":"83c9de00-f066-4aa4-900c-4e0d6c5042ca","Type":"ContainerStarted","Data":"0bd63768b752ad933eab7c4f4e6302eaf7cb0c1b0e221994c6497a355c5a808e"}
Jul 01 20:47:45.113990 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:45.113590    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-22nc7" event={"ID":"83c9de00-f066-4aa4-900c-4e0d6c5042ca","Type":"ContainerStarted","Data":"93c2afd3d05e78eb110c40a45abff442b7c23f455c67c1ae55540daf9259a50a"}
Jul 01 20:47:45.113990 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:45.113619    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-22nc7" event={"ID":"83c9de00-f066-4aa4-900c-4e0d6c5042ca","Type":"ContainerStarted","Data":"3df3f9e6ffbac933ff6cc6dec8bfa4433ca7b86e01e27db0a79c0450aca3faa6"}
Jul 01 20:47:45.115296 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:45.115268    2564 generic.go:356] "Generic (PLEG): container finished" podID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerID="97091edf3d8b2392f5f65b0373fc5b06ba273abcda5ffd7428b045c2fbc5ef12" exitCode=0
Jul 01 20:47:45.115422 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:45.115320    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de41a7ac-ca29-46f1-8145-f294eb2cbd56","Type":"ContainerDied","Data":"97091edf3d8b2392f5f65b0373fc5b06ba273abcda5ffd7428b045c2fbc5ef12"}
Jul 01 20:47:45.122195 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:45.122173    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x5xnh" event={"ID":"866f6082-54b0-4b03-8d36-69dc77a6ca61","Type":"ContainerStarted","Data":"c212bd72864b6109ecb5ca4dc85c764a549bdeb78c1f8fd5ad05276ecada5cf3"}
Jul 01 20:47:45.122357 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:45.122337    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x5xnh" event={"ID":"866f6082-54b0-4b03-8d36-69dc77a6ca61","Type":"ContainerStarted","Data":"652c0e4c20083c8c729cb6c8171f85f89edc796570e5664b0f3c54190ecd90ad"}
Jul 01 20:47:45.185233 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:45.185173    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-845b954455-22nc7" podStartSLOduration=2.289317477 podStartE2EDuration="4.185152279s" totalImagesPullingTime="1.895834802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:45.139594474 +0000 UTC m=+64.815610042" watchObservedRunningTime="2026-07-01 20:47:45.185152279 +0000 UTC m=+64.861167846"
Jul 01 20:47:45.208834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:45.208366    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-x5xnh" podStartSLOduration=2.15203214 podStartE2EDuration="4.208345624s" totalImagesPullingTime="1.978635024s" totalInitContainerRuntime="77.67846ms" isStatefulPod=true podCreationTimestamp="2026-07-01 20:47:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:45.2072729 +0000 UTC m=+64.883288492" watchObservedRunningTime="2026-07-01 20:47:45.208345624 +0000 UTC m=+64.884361192"
Jul 01 20:47:46.360955 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.360916    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"]
Jul 01 20:47:46.376295 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.376268    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"
Jul 01 20:47:46.379716 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.379544    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 01 20:47:46.379716 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.379568    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-fbgk5\""
Jul 01 20:47:46.379716 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.379544    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 01 20:47:46.379955 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.379807    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"]
Jul 01 20:47:46.380008 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.379988    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 01 20:47:46.380073 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.380027    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 01 20:47:46.380073 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.379987    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 01 20:47:46.387166 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.387145    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 01 20:47:46.499109 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.499081    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d584412e-45b7-45b6-8b32-158eb4a93739-serving-certs-ca-bundle\") pod \"telemeter-client-5fbb849ddd-7w6dx\" (UID: \"d584412e-45b7-45b6-8b32-158eb4a93739\") " pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"
Jul 01 20:47:46.499197 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.499115    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d584412e-45b7-45b6-8b32-158eb4a93739-federate-client-tls\") pod \"telemeter-client-5fbb849ddd-7w6dx\" (UID: \"d584412e-45b7-45b6-8b32-158eb4a93739\") " pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"
Jul 01 20:47:46.499197 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.499134    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d584412e-45b7-45b6-8b32-158eb4a93739-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5fbb849ddd-7w6dx\" (UID: \"d584412e-45b7-45b6-8b32-158eb4a93739\") " pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"
Jul 01 20:47:46.499197 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.499157    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8gb9\" (UniqueName: \"kubernetes.io/projected/d584412e-45b7-45b6-8b32-158eb4a93739-kube-api-access-j8gb9\") pod \"telemeter-client-5fbb849ddd-7w6dx\" (UID: \"d584412e-45b7-45b6-8b32-158eb4a93739\") " pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"
Jul 01 20:47:46.499355 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.499215    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d584412e-45b7-45b6-8b32-158eb4a93739-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5fbb849ddd-7w6dx\" (UID: \"d584412e-45b7-45b6-8b32-158eb4a93739\") " pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"
Jul 01 20:47:46.499355 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.499254    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d584412e-45b7-45b6-8b32-158eb4a93739-telemeter-client-tls\") pod \"telemeter-client-5fbb849ddd-7w6dx\" (UID: \"d584412e-45b7-45b6-8b32-158eb4a93739\") " pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"
Jul 01 20:47:46.499355 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.499298    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d584412e-45b7-45b6-8b32-158eb4a93739-secret-telemeter-client\") pod \"telemeter-client-5fbb849ddd-7w6dx\" (UID: \"d584412e-45b7-45b6-8b32-158eb4a93739\") " pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"
Jul 01 20:47:46.499355 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.499352    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d584412e-45b7-45b6-8b32-158eb4a93739-metrics-client-ca\") pod \"telemeter-client-5fbb849ddd-7w6dx\" (UID: \"d584412e-45b7-45b6-8b32-158eb4a93739\") " pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"
Jul 01 20:47:46.600212 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.600183    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d584412e-45b7-45b6-8b32-158eb4a93739-metrics-client-ca\") pod \"telemeter-client-5fbb849ddd-7w6dx\" (UID: \"d584412e-45b7-45b6-8b32-158eb4a93739\") " pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"
Jul 01 20:47:46.600390 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.600228    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2rm6\" (UniqueName: \"kubernetes.io/projected/f4699b63-6ba0-4401-a43e-57c15fea25b3-kube-api-access-p2rm6\") pod \"network-check-target-9bl8d\" (UID: \"f4699b63-6ba0-4401-a43e-57c15fea25b3\") " pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:47:46.600390 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.600264    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d584412e-45b7-45b6-8b32-158eb4a93739-serving-certs-ca-bundle\") pod \"telemeter-client-5fbb849ddd-7w6dx\" (UID: \"d584412e-45b7-45b6-8b32-158eb4a93739\") " pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"
Jul 01 20:47:46.600390 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.600297    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d584412e-45b7-45b6-8b32-158eb4a93739-federate-client-tls\") pod \"telemeter-client-5fbb849ddd-7w6dx\" (UID: \"d584412e-45b7-45b6-8b32-158eb4a93739\") " pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"
Jul 01 20:47:46.600390 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.600323    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d584412e-45b7-45b6-8b32-158eb4a93739-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5fbb849ddd-7w6dx\" (UID: \"d584412e-45b7-45b6-8b32-158eb4a93739\") " pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"
Jul 01 20:47:46.600588 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.600420    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8gb9\" (UniqueName: \"kubernetes.io/projected/d584412e-45b7-45b6-8b32-158eb4a93739-kube-api-access-j8gb9\") pod \"telemeter-client-5fbb849ddd-7w6dx\" (UID: \"d584412e-45b7-45b6-8b32-158eb4a93739\") " pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"
Jul 01 20:47:46.600588 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.600479    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d584412e-45b7-45b6-8b32-158eb4a93739-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5fbb849ddd-7w6dx\" (UID: \"d584412e-45b7-45b6-8b32-158eb4a93739\") " pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"
Jul 01 20:47:46.600588 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.600537    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d584412e-45b7-45b6-8b32-158eb4a93739-telemeter-client-tls\") pod \"telemeter-client-5fbb849ddd-7w6dx\" (UID: \"d584412e-45b7-45b6-8b32-158eb4a93739\") " pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"
Jul 01 20:47:46.600588 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.600575    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c606e570-2fd6-43cb-9a84-9cdeab442815-metrics-certs\") pod \"network-metrics-daemon-9xhz4\" (UID: \"c606e570-2fd6-43cb-9a84-9cdeab442815\") " pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:47:46.600770 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.600607    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d584412e-45b7-45b6-8b32-158eb4a93739-secret-telemeter-client\") pod \"telemeter-client-5fbb849ddd-7w6dx\" (UID: \"d584412e-45b7-45b6-8b32-158eb4a93739\") " pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"
Jul 01 20:47:46.600770 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.600666    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-original-pull-secret\") pod \"global-pull-secret-syncer-64pmt\" (UID: \"ed4e44b6-7fbe-4e18-974e-edbc64ce0398\") " pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:47:46.601749 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.601018    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d584412e-45b7-45b6-8b32-158eb4a93739-metrics-client-ca\") pod \"telemeter-client-5fbb849ddd-7w6dx\" (UID: \"d584412e-45b7-45b6-8b32-158eb4a93739\") " pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"
Jul 01 20:47:46.601749 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.601121    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d584412e-45b7-45b6-8b32-158eb4a93739-serving-certs-ca-bundle\") pod \"telemeter-client-5fbb849ddd-7w6dx\" (UID: \"d584412e-45b7-45b6-8b32-158eb4a93739\") " pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"
Jul 01 20:47:46.601749 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.601700    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d584412e-45b7-45b6-8b32-158eb4a93739-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5fbb849ddd-7w6dx\" (UID: \"d584412e-45b7-45b6-8b32-158eb4a93739\") " pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"
Jul 01 20:47:46.603473 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.603444    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d584412e-45b7-45b6-8b32-158eb4a93739-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5fbb849ddd-7w6dx\" (UID: \"d584412e-45b7-45b6-8b32-158eb4a93739\") " pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"
Jul 01 20:47:46.603581 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.603554    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d584412e-45b7-45b6-8b32-158eb4a93739-federate-client-tls\") pod \"telemeter-client-5fbb849ddd-7w6dx\" (UID: \"d584412e-45b7-45b6-8b32-158eb4a93739\") " pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"
Jul 01 20:47:46.603753 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.603712    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d584412e-45b7-45b6-8b32-158eb4a93739-secret-telemeter-client\") pod \"telemeter-client-5fbb849ddd-7w6dx\" (UID: \"d584412e-45b7-45b6-8b32-158eb4a93739\") " pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"
Jul 01 20:47:46.603846 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.603823    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d584412e-45b7-45b6-8b32-158eb4a93739-telemeter-client-tls\") pod \"telemeter-client-5fbb849ddd-7w6dx\" (UID: \"d584412e-45b7-45b6-8b32-158eb4a93739\") " pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"
Jul 01 20:47:46.603929 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.603875    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 20:47:46.604072 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.604052    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 20:47:46.604333 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.604319    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 20:47:46.610105 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.610086    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8gb9\" (UniqueName: \"kubernetes.io/projected/d584412e-45b7-45b6-8b32-158eb4a93739-kube-api-access-j8gb9\") pod \"telemeter-client-5fbb849ddd-7w6dx\" (UID: \"d584412e-45b7-45b6-8b32-158eb4a93739\") " pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"
Jul 01 20:47:46.613249 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.613194    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 20:47:46.614310 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.614265    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed4e44b6-7fbe-4e18-974e-edbc64ce0398-original-pull-secret\") pod \"global-pull-secret-syncer-64pmt\" (UID: \"ed4e44b6-7fbe-4e18-974e-edbc64ce0398\") " pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:47:46.614575 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.614551    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c606e570-2fd6-43cb-9a84-9cdeab442815-metrics-certs\") pod \"network-metrics-daemon-9xhz4\" (UID: \"c606e570-2fd6-43cb-9a84-9cdeab442815\") " pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:47:46.623982 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.623963    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2rm6\" (UniqueName: \"kubernetes.io/projected/f4699b63-6ba0-4401-a43e-57c15fea25b3-kube-api-access-p2rm6\") pod \"network-check-target-9bl8d\" (UID: \"f4699b63-6ba0-4401-a43e-57c15fea25b3\") " pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:47:46.665307 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.665274    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pmt"
Jul 01 20:47:46.673487 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.673467    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7lgrj\""
Jul 01 20:47:46.680914 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.680890    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xhz4"
Jul 01 20:47:46.688903 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.688880    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"
Jul 01 20:47:46.825438 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.822729    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-64pmt"]
Jul 01 20:47:46.830897 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:47:46.830824    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded4e44b6_7fbe_4e18_974e_edbc64ce0398.slice/crio-7dd9bae2330531e4af7549b51f403bab0eb96598a55727079c838d5bacc3a816 WatchSource:0}: Error finding container 7dd9bae2330531e4af7549b51f403bab0eb96598a55727079c838d5bacc3a816: Status 404 returned error can't find the container with id 7dd9bae2330531e4af7549b51f403bab0eb96598a55727079c838d5bacc3a816
Jul 01 20:47:46.846634 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.846606    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9xhz4"]
Jul 01 20:47:46.850217 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:47:46.850189    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc606e570_2fd6_43cb_9a84_9cdeab442815.slice/crio-0910c96fef7408a840b1270db7728eae688eb89e3e2fb5aff63a268fb798a74a WatchSource:0}: Error finding container 0910c96fef7408a840b1270db7728eae688eb89e3e2fb5aff63a268fb798a74a: Status 404 returned error can't find the container with id 0910c96fef7408a840b1270db7728eae688eb89e3e2fb5aff63a268fb798a74a
Jul 01 20:47:46.867544 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.867470    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-r694q\""
Jul 01 20:47:46.872495 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.872446    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:47:46.872634 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:46.872611    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx"]
Jul 01 20:47:46.874416 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:47:46.874395    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd584412e_45b7_45b6_8b32_158eb4a93739.slice/crio-1916625c3fbbd5b18f926e9e80394576366eec420b680e2cf0453ecf3ee6693d WatchSource:0}: Error finding container 1916625c3fbbd5b18f926e9e80394576366eec420b680e2cf0453ecf3ee6693d: Status 404 returned error can't find the container with id 1916625c3fbbd5b18f926e9e80394576366eec420b680e2cf0453ecf3ee6693d
Jul 01 20:47:47.036114 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.036085    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9bl8d"]
Jul 01 20:47:47.039556 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:47:47.039530    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4699b63_6ba0_4401_a43e_57c15fea25b3.slice/crio-313c77b5ffc275d7e62fdea6a8981c5b576b06c811b3f2e9d352f0a76e844749 WatchSource:0}: Error finding container 313c77b5ffc275d7e62fdea6a8981c5b576b06c811b3f2e9d352f0a76e844749: Status 404 returned error can't find the container with id 313c77b5ffc275d7e62fdea6a8981c5b576b06c811b3f2e9d352f0a76e844749
Jul 01 20:47:47.131082 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.130986    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx" event={"ID":"d584412e-45b7-45b6-8b32-158eb4a93739","Type":"ContainerStarted","Data":"1916625c3fbbd5b18f926e9e80394576366eec420b680e2cf0453ecf3ee6693d"}
Jul 01 20:47:47.132052 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.132027    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9bl8d" event={"ID":"f4699b63-6ba0-4401-a43e-57c15fea25b3","Type":"ContainerStarted","Data":"313c77b5ffc275d7e62fdea6a8981c5b576b06c811b3f2e9d352f0a76e844749"}
Jul 01 20:47:47.133131 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.133100    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9xhz4" event={"ID":"c606e570-2fd6-43cb-9a84-9cdeab442815","Type":"ContainerStarted","Data":"0910c96fef7408a840b1270db7728eae688eb89e3e2fb5aff63a268fb798a74a"}
Jul 01 20:47:47.134179 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.134156    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-64pmt" event={"ID":"ed4e44b6-7fbe-4e18-974e-edbc64ce0398","Type":"ContainerStarted","Data":"7dd9bae2330531e4af7549b51f403bab0eb96598a55727079c838d5bacc3a816"}
Jul 01 20:47:47.135993 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.135974    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9chmn" event={"ID":"f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2","Type":"ContainerStarted","Data":"9b391eb5de6bf9308f8345247a35665b1878d6ac8952dd3f0427f840c0600251"}
Jul 01 20:47:47.161238 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.161187    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9chmn" podStartSLOduration=2.352610849 podStartE2EDuration="12.161170903s" totalImagesPullingTime="9.808560054s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:47:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:47.159545677 +0000 UTC m=+66.835561244" watchObservedRunningTime="2026-07-01 20:47:47.161170903 +0000 UTC m=+66.837186489"
Jul 01 20:47:47.667787 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.667754    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 20:47:47.671569 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.671543    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.675854 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.675837    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 01 20:47:47.676236 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.676209    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 01 20:47:47.676411 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.676396    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 01 20:47:47.677276 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.677198    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 01 20:47:47.677276 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.677214    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dfsh0inbenim0\""
Jul 01 20:47:47.677276 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.677222    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 01 20:47:47.677478 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.677290    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 01 20:47:47.677478 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.677317    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 01 20:47:47.677675 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.677647    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 01 20:47:47.678418 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.677937    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 01 20:47:47.678418 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.677942    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 01 20:47:47.678418 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.677949    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 01 20:47:47.678418 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.677952    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 01 20:47:47.678418 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.678055    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-t2kph\""
Jul 01 20:47:47.678418 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.678276    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 01 20:47:47.681136 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.680843    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 01 20:47:47.684051 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.684031    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 01 20:47:47.699471 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.699382    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 20:47:47.711285 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.711249    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.711389 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.711289    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.711389 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.711322    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14ab9a24-1175-41f1-b736-93bc2ca01ae8-config-out\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.711389 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.711359    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.711594 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.711397    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-config\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.711594 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.711417    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14ab9a24-1175-41f1-b736-93bc2ca01ae8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.711594 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.711440    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.711594 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.711466    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-web-config\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.711594 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.711499    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.711594 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.711525    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.711594 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.711549    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.711594 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.711578    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.711991 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.711607    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.711991 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.711631    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.711991 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.711651    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.711991 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.711674    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.711991 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.711707    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.711991 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.711732    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-887wf\" (UniqueName: \"kubernetes.io/projected/14ab9a24-1175-41f1-b736-93bc2ca01ae8-kube-api-access-887wf\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.711991 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.711758    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.711991 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.711785    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.813065 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.813033    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14ab9a24-1175-41f1-b736-93bc2ca01ae8-config-out\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.813177 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.813069    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.813177 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.813113    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-config\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.813177 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.813135    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14ab9a24-1175-41f1-b736-93bc2ca01ae8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.813177 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.813158    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.813468 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.813186    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-web-config\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.813468 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.813219    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.813468 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.813246    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.813468 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.813270    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.813468 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.813301    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.813468 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.813334    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.813468 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.813361    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.813468 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.813383    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.813468 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.813420    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.813468 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.813457    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.814077 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.813488    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887wf\" (UniqueName: \"kubernetes.io/projected/14ab9a24-1175-41f1-b736-93bc2ca01ae8-kube-api-access-887wf\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.814077 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.813526    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.814077 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.813553    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.814077 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.813603    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.814077 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.813630    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.814729 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.814569    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.816594 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.816283    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14ab9a24-1175-41f1-b736-93bc2ca01ae8-config-out\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.819113 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.818789    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.819816 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.819476    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.824453 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.823185    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.824453 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.823236    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.824453 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.823646    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.824453 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.823646    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.824453 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.823873    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14ab9a24-1175-41f1-b736-93bc2ca01ae8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.824453 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.824154    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.824453 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.824273    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.824453 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.824309    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.824904 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.824770    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.825095 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.824956    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.825208 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.825180    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-config\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.825896 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.825509    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.825896 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.825812    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.826645 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.826617    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.828869 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.828137    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-web-config\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.834235 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.834169    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-887wf\" (UniqueName: \"kubernetes.io/projected/14ab9a24-1175-41f1-b736-93bc2ca01ae8-kube-api-access-887wf\") pod \"prometheus-k8s-0\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:47.984366 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:47.984330    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:47:48.151317 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:48.150949    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de41a7ac-ca29-46f1-8145-f294eb2cbd56","Type":"ContainerStarted","Data":"46f18a4f724dea3b4e7acef72117cf0b9e01fc33d20c8c104203036e8eb1a7da"}
Jul 01 20:47:48.151317 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:48.151063    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de41a7ac-ca29-46f1-8145-f294eb2cbd56","Type":"ContainerStarted","Data":"e9be90caef40461db05c2b9b966c06053e850ca373ebd1f5f8baca44f028913a"}
Jul 01 20:47:48.151317 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:48.151100    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de41a7ac-ca29-46f1-8145-f294eb2cbd56","Type":"ContainerStarted","Data":"016a29a290a82ecec1b05b00a2a32ab36ea20b5a58e3ce1682be826de6643285"}
Jul 01 20:47:48.151317 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:48.151139    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de41a7ac-ca29-46f1-8145-f294eb2cbd56","Type":"ContainerStarted","Data":"e2be532576140f63d68169351a95c3a4f9169daf3592c96917c1bc0195de1ee9"}
Jul 01 20:47:48.151317 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:48.151179    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de41a7ac-ca29-46f1-8145-f294eb2cbd56","Type":"ContainerStarted","Data":"8d8d8cd6e79490d9b35a6c3c79fcf9a73dea18fd376d2a6e5ae0356aa8f93895"}
Jul 01 20:47:48.189491 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:48.189408    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 20:47:48.193958 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:47:48.193925    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ab9a24_1175_41f1_b736_93bc2ca01ae8.slice/crio-d40f605763df9fdb56cb2e09b274c5bb677747c7c6b892478a2076559b18147a WatchSource:0}: Error finding container d40f605763df9fdb56cb2e09b274c5bb677747c7c6b892478a2076559b18147a: Status 404 returned error can't find the container with id d40f605763df9fdb56cb2e09b274c5bb677747c7c6b892478a2076559b18147a
Jul 01 20:47:49.159294 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:49.159190    2564 generic.go:356] "Generic (PLEG): container finished" podID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerID="e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69" exitCode=0
Jul 01 20:47:49.159923 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:49.159733    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14ab9a24-1175-41f1-b736-93bc2ca01ae8","Type":"ContainerDied","Data":"e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69"}
Jul 01 20:47:49.159923 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:49.159823    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14ab9a24-1175-41f1-b736-93bc2ca01ae8","Type":"ContainerStarted","Data":"d40f605763df9fdb56cb2e09b274c5bb677747c7c6b892478a2076559b18147a"}
Jul 01 20:47:49.163448 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:49.163418    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9xhz4" event={"ID":"c606e570-2fd6-43cb-9a84-9cdeab442815","Type":"ContainerStarted","Data":"00e4b8bb575d482a209f59c40c62f22780c79fe0059bc78e96a83c8e88ed493e"}
Jul 01 20:47:49.163529 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:49.163472    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9xhz4" event={"ID":"c606e570-2fd6-43cb-9a84-9cdeab442815","Type":"ContainerStarted","Data":"26f5dda85a532a10c42fadb9ae7021d0690d793b8a5dd9c4cd44677d437a1f28"}
Jul 01 20:47:49.216972 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:49.216907    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9xhz4" podStartSLOduration=66.466572918 podStartE2EDuration="1m8.216882634s" totalImagesPullingTime="1.750309716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:49.214811384 +0000 UTC m=+68.890826962" watchObservedRunningTime="2026-07-01 20:47:49.216882634 +0000 UTC m=+68.892898198"
Jul 01 20:47:50.171084 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:50.171046    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de41a7ac-ca29-46f1-8145-f294eb2cbd56","Type":"ContainerStarted","Data":"0eaef45940645673d0769693ec56f9a276f9ff4d08b307549d6c0ae25df5ad62"}
Jul 01 20:47:52.098120 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:52.098089    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zt5qb"
Jul 01 20:47:52.117281 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:52.117216    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.303929282 podStartE2EDuration="10.117193147s" totalImagesPullingTime="4.794637223s" totalInitContainerRuntime="18.626642ms" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:50.208360877 +0000 UTC m=+69.884376444" watchObservedRunningTime="2026-07-01 20:47:52.117193147 +0000 UTC m=+71.793208719"
Jul 01 20:47:54.189406 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:54.189370    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx" event={"ID":"d584412e-45b7-45b6-8b32-158eb4a93739","Type":"ContainerStarted","Data":"386d299d778be39a165b7bc00b6d060766497a304efa242ad362f375174f7d55"}
Jul 01 20:47:54.189834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:54.189479    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx" event={"ID":"d584412e-45b7-45b6-8b32-158eb4a93739","Type":"ContainerStarted","Data":"4ef1260b7b5e4cae899e8b2e24e52bcd85edee59ffb965adcd9f0afc43ea7dae"}
Jul 01 20:47:54.189834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:54.189515    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx" event={"ID":"d584412e-45b7-45b6-8b32-158eb4a93739","Type":"ContainerStarted","Data":"89166995371beafc60b3df6d3674324f5798adffdc598b5dc455e83f89b66019"}
Jul 01 20:47:54.191598 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:54.191576    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14ab9a24-1175-41f1-b736-93bc2ca01ae8","Type":"ContainerStarted","Data":"ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6"}
Jul 01 20:47:54.191707 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:54.191635    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14ab9a24-1175-41f1-b736-93bc2ca01ae8","Type":"ContainerStarted","Data":"55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9"}
Jul 01 20:47:54.192885 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:54.192850    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9bl8d" event={"ID":"f4699b63-6ba0-4401-a43e-57c15fea25b3","Type":"ContainerStarted","Data":"2f57a505f147891242f51dfe69d6a8fac9e9e9a0d766198b5ffc1f939f8d6011"}
Jul 01 20:47:54.192994 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:54.192924    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:47:54.194155 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:54.194135    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-64pmt" event={"ID":"ed4e44b6-7fbe-4e18-974e-edbc64ce0398","Type":"ContainerStarted","Data":"7d1fb2b27442da73ac4268580f1a16c751c3a83194b00f9be54540f05b1b9cfc"}
Jul 01 20:47:54.223610 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:54.223557    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5fbb849ddd-7w6dx" podStartSLOduration=1.522407854 podStartE2EDuration="8.223538909s" totalImagesPullingTime="6.701131055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:54.222474786 +0000 UTC m=+73.898490355" watchObservedRunningTime="2026-07-01 20:47:54.223538909 +0000 UTC m=+73.899554477"
Jul 01 20:47:54.239229 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:54.239094    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-64pmt" podStartSLOduration=66.494557928 podStartE2EDuration="1m13.239076519s" totalImagesPullingTime="6.744518591s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:54.238914272 +0000 UTC m=+73.914929840" watchObservedRunningTime="2026-07-01 20:47:54.239076519 +0000 UTC m=+73.915092087"
Jul 01 20:47:54.260187 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:54.260149    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9bl8d" podStartSLOduration=66.723841565 podStartE2EDuration="1m13.260129093s" totalImagesPullingTime="6.536287528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:54.259403674 +0000 UTC m=+73.935419242" watchObservedRunningTime="2026-07-01 20:47:54.260129093 +0000 UTC m=+73.936144702"
Jul 01 20:47:56.207199 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:56.207162    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14ab9a24-1175-41f1-b736-93bc2ca01ae8","Type":"ContainerStarted","Data":"e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31"}
Jul 01 20:47:56.207568 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:56.207253    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14ab9a24-1175-41f1-b736-93bc2ca01ae8","Type":"ContainerStarted","Data":"a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b"}
Jul 01 20:47:56.207568 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:56.207285    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14ab9a24-1175-41f1-b736-93bc2ca01ae8","Type":"ContainerStarted","Data":"6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887"}
Jul 01 20:47:56.207568 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:56.207317    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14ab9a24-1175-41f1-b736-93bc2ca01ae8","Type":"ContainerStarted","Data":"59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7"}
Jul 01 20:47:56.248850 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:56.248796    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.141385511 podStartE2EDuration="9.248779249s" totalImagesPullingTime="6.075973071s" totalInitContainerRuntime="31.420667ms" isStatefulPod=false podCreationTimestamp="2026-07-01 20:47:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:47:56.24152187 +0000 UTC m=+75.917537438" watchObservedRunningTime="2026-07-01 20:47:56.248779249 +0000 UTC m=+75.924794815"
Jul 01 20:47:57.985805 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:47:57.985777    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:25.203612 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:48:25.203582    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9bl8d"
Jul 01 20:48:47.985805 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:48:47.985775    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:48.005334 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:48:48.005304    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:48:48.407919 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:48:48.407888    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:01.786273 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:01.786242    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 20:49:01.786770 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:01.786556    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerName="kube-rbac-proxy" containerID="cri-o://e9be90caef40461db05c2b9b966c06053e850ca373ebd1f5f8baca44f028913a" gracePeriod=120
Jul 01 20:49:01.786770 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:01.786587    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerName="config-reloader" containerID="cri-o://e2be532576140f63d68169351a95c3a4f9169daf3592c96917c1bc0195de1ee9" gracePeriod=120
Jul 01 20:49:01.786770 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:01.786565    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerName="kube-rbac-proxy-web" containerID="cri-o://016a29a290a82ecec1b05b00a2a32ab36ea20b5a58e3ce1682be826de6643285" gracePeriod=120
Jul 01 20:49:01.786770 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:01.786629    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerName="kube-rbac-proxy-metric" containerID="cri-o://46f18a4f724dea3b4e7acef72117cf0b9e01fc33d20c8c104203036e8eb1a7da" gracePeriod=120
Jul 01 20:49:01.786770 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:01.786565    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerName="alertmanager" containerID="cri-o://8d8d8cd6e79490d9b35a6c3c79fcf9a73dea18fd376d2a6e5ae0356aa8f93895" gracePeriod=120
Jul 01 20:49:01.786770 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:01.786620    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerName="prom-label-proxy" containerID="cri-o://0eaef45940645673d0769693ec56f9a276f9ff4d08b307549d6c0ae25df5ad62" gracePeriod=120
Jul 01 20:49:02.445600 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:02.445568    2564 generic.go:356] "Generic (PLEG): container finished" podID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerID="0eaef45940645673d0769693ec56f9a276f9ff4d08b307549d6c0ae25df5ad62" exitCode=0
Jul 01 20:49:02.445600 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:02.445595    2564 generic.go:356] "Generic (PLEG): container finished" podID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerID="46f18a4f724dea3b4e7acef72117cf0b9e01fc33d20c8c104203036e8eb1a7da" exitCode=0
Jul 01 20:49:02.445600 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:02.445602    2564 generic.go:356] "Generic (PLEG): container finished" podID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerID="e9be90caef40461db05c2b9b966c06053e850ca373ebd1f5f8baca44f028913a" exitCode=0
Jul 01 20:49:02.445600 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:02.445608    2564 generic.go:356] "Generic (PLEG): container finished" podID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerID="e2be532576140f63d68169351a95c3a4f9169daf3592c96917c1bc0195de1ee9" exitCode=0
Jul 01 20:49:02.445600 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:02.445613    2564 generic.go:356] "Generic (PLEG): container finished" podID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerID="8d8d8cd6e79490d9b35a6c3c79fcf9a73dea18fd376d2a6e5ae0356aa8f93895" exitCode=0
Jul 01 20:49:02.445915 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:02.445644    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de41a7ac-ca29-46f1-8145-f294eb2cbd56","Type":"ContainerDied","Data":"0eaef45940645673d0769693ec56f9a276f9ff4d08b307549d6c0ae25df5ad62"}
Jul 01 20:49:02.445915 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:02.445741    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de41a7ac-ca29-46f1-8145-f294eb2cbd56","Type":"ContainerDied","Data":"46f18a4f724dea3b4e7acef72117cf0b9e01fc33d20c8c104203036e8eb1a7da"}
Jul 01 20:49:02.445915 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:02.445768    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de41a7ac-ca29-46f1-8145-f294eb2cbd56","Type":"ContainerDied","Data":"e9be90caef40461db05c2b9b966c06053e850ca373ebd1f5f8baca44f028913a"}
Jul 01 20:49:02.445915 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:02.445792    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de41a7ac-ca29-46f1-8145-f294eb2cbd56","Type":"ContainerDied","Data":"e2be532576140f63d68169351a95c3a4f9169daf3592c96917c1bc0195de1ee9"}
Jul 01 20:49:02.445915 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:02.445816    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de41a7ac-ca29-46f1-8145-f294eb2cbd56","Type":"ContainerDied","Data":"8d8d8cd6e79490d9b35a6c3c79fcf9a73dea18fd376d2a6e5ae0356aa8f93895"}
Jul 01 20:49:03.024148 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.024123    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.139529 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.139500    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-secret-alertmanager-kube-rbac-proxy-metric\") pod \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") "
Jul 01 20:49:03.139702 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.139540    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de41a7ac-ca29-46f1-8145-f294eb2cbd56-config-out\") pod \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") "
Jul 01 20:49:03.139702 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.139560    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-cluster-tls-config\") pod \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") "
Jul 01 20:49:03.139702 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.139579    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de41a7ac-ca29-46f1-8145-f294eb2cbd56-alertmanager-trusted-ca-bundle\") pod \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") "
Jul 01 20:49:03.139702 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.139613    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-secret-alertmanager-kube-rbac-proxy\") pod \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") "
Jul 01 20:49:03.139702 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.139639    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-config-volume\") pod \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") "
Jul 01 20:49:03.139702 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.139679    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de41a7ac-ca29-46f1-8145-f294eb2cbd56-metrics-client-ca\") pod \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") "
Jul 01 20:49:03.140050 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.139723    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6r7dw\" (UniqueName: \"kubernetes.io/projected/de41a7ac-ca29-46f1-8145-f294eb2cbd56-kube-api-access-6r7dw\") pod \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") "
Jul 01 20:49:03.140108 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.140066    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de41a7ac-ca29-46f1-8145-f294eb2cbd56-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "de41a7ac-ca29-46f1-8145-f294eb2cbd56" (UID: "de41a7ac-ca29-46f1-8145-f294eb2cbd56"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:49:03.140108 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.140086    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de41a7ac-ca29-46f1-8145-f294eb2cbd56-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "de41a7ac-ca29-46f1-8145-f294eb2cbd56" (UID: "de41a7ac-ca29-46f1-8145-f294eb2cbd56"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:49:03.140210 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.139759    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-secret-alertmanager-kube-rbac-proxy-web\") pod \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") "
Jul 01 20:49:03.140210 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.140200    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-secret-alertmanager-main-tls\") pod \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") "
Jul 01 20:49:03.140317 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.140236    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-web-config\") pod \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") "
Jul 01 20:49:03.140317 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.140267    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de41a7ac-ca29-46f1-8145-f294eb2cbd56-tls-assets\") pod \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") "
Jul 01 20:49:03.140419 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.140317    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/de41a7ac-ca29-46f1-8145-f294eb2cbd56-alertmanager-main-db\") pod \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\" (UID: \"de41a7ac-ca29-46f1-8145-f294eb2cbd56\") "
Jul 01 20:49:03.140605 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.140585    2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de41a7ac-ca29-46f1-8145-f294eb2cbd56-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:03.140677 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.140610    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de41a7ac-ca29-46f1-8145-f294eb2cbd56-metrics-client-ca\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:03.141040 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.141013    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de41a7ac-ca29-46f1-8145-f294eb2cbd56-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "de41a7ac-ca29-46f1-8145-f294eb2cbd56" (UID: "de41a7ac-ca29-46f1-8145-f294eb2cbd56"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 20:49:03.142805 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.142756    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-config-volume" (OuterVolumeSpecName: "config-volume") pod "de41a7ac-ca29-46f1-8145-f294eb2cbd56" (UID: "de41a7ac-ca29-46f1-8145-f294eb2cbd56"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:49:03.142927 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.142819    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de41a7ac-ca29-46f1-8145-f294eb2cbd56-kube-api-access-6r7dw" (OuterVolumeSpecName: "kube-api-access-6r7dw") pod "de41a7ac-ca29-46f1-8145-f294eb2cbd56" (UID: "de41a7ac-ca29-46f1-8145-f294eb2cbd56"). InnerVolumeSpecName "kube-api-access-6r7dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:49:03.142997 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.142934    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de41a7ac-ca29-46f1-8145-f294eb2cbd56-config-out" (OuterVolumeSpecName: "config-out") pod "de41a7ac-ca29-46f1-8145-f294eb2cbd56" (UID: "de41a7ac-ca29-46f1-8145-f294eb2cbd56"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 20:49:03.143230 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.143208    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "de41a7ac-ca29-46f1-8145-f294eb2cbd56" (UID: "de41a7ac-ca29-46f1-8145-f294eb2cbd56"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:49:03.143410 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.143383    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "de41a7ac-ca29-46f1-8145-f294eb2cbd56" (UID: "de41a7ac-ca29-46f1-8145-f294eb2cbd56"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:49:03.143986 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.143953    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "de41a7ac-ca29-46f1-8145-f294eb2cbd56" (UID: "de41a7ac-ca29-46f1-8145-f294eb2cbd56"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:49:03.144340 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.144318    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de41a7ac-ca29-46f1-8145-f294eb2cbd56-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "de41a7ac-ca29-46f1-8145-f294eb2cbd56" (UID: "de41a7ac-ca29-46f1-8145-f294eb2cbd56"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:49:03.144466 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.144446    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "de41a7ac-ca29-46f1-8145-f294eb2cbd56" (UID: "de41a7ac-ca29-46f1-8145-f294eb2cbd56"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:49:03.146814 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.146791    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "de41a7ac-ca29-46f1-8145-f294eb2cbd56" (UID: "de41a7ac-ca29-46f1-8145-f294eb2cbd56"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:49:03.152446 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.152422    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-web-config" (OuterVolumeSpecName: "web-config") pod "de41a7ac-ca29-46f1-8145-f294eb2cbd56" (UID: "de41a7ac-ca29-46f1-8145-f294eb2cbd56"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:49:03.241599 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.241562    2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:03.241599 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.241592    2564 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-config-volume\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:03.241599 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.241605    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r7dw\" (UniqueName: \"kubernetes.io/projected/de41a7ac-ca29-46f1-8145-f294eb2cbd56-kube-api-access-6r7dw\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:03.241825 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.241618    2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:03.241825 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.241631    2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-secret-alertmanager-main-tls\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:03.241825 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.241642    2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-web-config\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:03.241825 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.241652    2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de41a7ac-ca29-46f1-8145-f294eb2cbd56-tls-assets\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:03.241825 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.241664    2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/de41a7ac-ca29-46f1-8145-f294eb2cbd56-alertmanager-main-db\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:03.241825 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.241675    2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:03.241825 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.241687    2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de41a7ac-ca29-46f1-8145-f294eb2cbd56-config-out\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:03.241825 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.241698    2564 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/de41a7ac-ca29-46f1-8145-f294eb2cbd56-cluster-tls-config\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:03.452392 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.452303    2564 generic.go:356] "Generic (PLEG): container finished" podID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerID="016a29a290a82ecec1b05b00a2a32ab36ea20b5a58e3ce1682be826de6643285" exitCode=0
Jul 01 20:49:03.452512 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.452400    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.452512 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.452401    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de41a7ac-ca29-46f1-8145-f294eb2cbd56","Type":"ContainerDied","Data":"016a29a290a82ecec1b05b00a2a32ab36ea20b5a58e3ce1682be826de6643285"}
Jul 01 20:49:03.452512 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.452480    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de41a7ac-ca29-46f1-8145-f294eb2cbd56","Type":"ContainerDied","Data":"3d1e1c8c35169ea426b285c4c6b0e34b57d277683061ee1173b3ae250bf89db0"}
Jul 01 20:49:03.452615 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.452515    2564 scope.go:117] "RemoveContainer" containerID="0eaef45940645673d0769693ec56f9a276f9ff4d08b307549d6c0ae25df5ad62"
Jul 01 20:49:03.460207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.460188    2564 scope.go:117] "RemoveContainer" containerID="46f18a4f724dea3b4e7acef72117cf0b9e01fc33d20c8c104203036e8eb1a7da"
Jul 01 20:49:03.466691 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.466675    2564 scope.go:117] "RemoveContainer" containerID="e9be90caef40461db05c2b9b966c06053e850ca373ebd1f5f8baca44f028913a"
Jul 01 20:49:03.472806 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.472790    2564 scope.go:117] "RemoveContainer" containerID="016a29a290a82ecec1b05b00a2a32ab36ea20b5a58e3ce1682be826de6643285"
Jul 01 20:49:03.477675 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.477650    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 20:49:03.479914 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.479898    2564 scope.go:117] "RemoveContainer" containerID="e2be532576140f63d68169351a95c3a4f9169daf3592c96917c1bc0195de1ee9"
Jul 01 20:49:03.485314 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.485293    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 20:49:03.487443 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.487426    2564 scope.go:117] "RemoveContainer" containerID="8d8d8cd6e79490d9b35a6c3c79fcf9a73dea18fd376d2a6e5ae0356aa8f93895"
Jul 01 20:49:03.493932 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.493918    2564 scope.go:117] "RemoveContainer" containerID="97091edf3d8b2392f5f65b0373fc5b06ba273abcda5ffd7428b045c2fbc5ef12"
Jul 01 20:49:03.500166 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.500148    2564 scope.go:117] "RemoveContainer" containerID="0eaef45940645673d0769693ec56f9a276f9ff4d08b307549d6c0ae25df5ad62"
Jul 01 20:49:03.500414 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:49:03.500396    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eaef45940645673d0769693ec56f9a276f9ff4d08b307549d6c0ae25df5ad62\": container with ID starting with 0eaef45940645673d0769693ec56f9a276f9ff4d08b307549d6c0ae25df5ad62 not found: ID does not exist" containerID="0eaef45940645673d0769693ec56f9a276f9ff4d08b307549d6c0ae25df5ad62"
Jul 01 20:49:03.500463 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.500421    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eaef45940645673d0769693ec56f9a276f9ff4d08b307549d6c0ae25df5ad62"} err="failed to get container status \"0eaef45940645673d0769693ec56f9a276f9ff4d08b307549d6c0ae25df5ad62\": rpc error: code = NotFound desc = could not find container \"0eaef45940645673d0769693ec56f9a276f9ff4d08b307549d6c0ae25df5ad62\": container with ID starting with 0eaef45940645673d0769693ec56f9a276f9ff4d08b307549d6c0ae25df5ad62 not found: ID does not exist"
Jul 01 20:49:03.500463 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.500450    2564 scope.go:117] "RemoveContainer" containerID="46f18a4f724dea3b4e7acef72117cf0b9e01fc33d20c8c104203036e8eb1a7da"
Jul 01 20:49:03.500663 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:49:03.500645    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46f18a4f724dea3b4e7acef72117cf0b9e01fc33d20c8c104203036e8eb1a7da\": container with ID starting with 46f18a4f724dea3b4e7acef72117cf0b9e01fc33d20c8c104203036e8eb1a7da not found: ID does not exist" containerID="46f18a4f724dea3b4e7acef72117cf0b9e01fc33d20c8c104203036e8eb1a7da"
Jul 01 20:49:03.500728 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.500673    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46f18a4f724dea3b4e7acef72117cf0b9e01fc33d20c8c104203036e8eb1a7da"} err="failed to get container status \"46f18a4f724dea3b4e7acef72117cf0b9e01fc33d20c8c104203036e8eb1a7da\": rpc error: code = NotFound desc = could not find container \"46f18a4f724dea3b4e7acef72117cf0b9e01fc33d20c8c104203036e8eb1a7da\": container with ID starting with 46f18a4f724dea3b4e7acef72117cf0b9e01fc33d20c8c104203036e8eb1a7da not found: ID does not exist"
Jul 01 20:49:03.500728 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.500696    2564 scope.go:117] "RemoveContainer" containerID="e9be90caef40461db05c2b9b966c06053e850ca373ebd1f5f8baca44f028913a"
Jul 01 20:49:03.500938 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:49:03.500922    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9be90caef40461db05c2b9b966c06053e850ca373ebd1f5f8baca44f028913a\": container with ID starting with e9be90caef40461db05c2b9b966c06053e850ca373ebd1f5f8baca44f028913a not found: ID does not exist" containerID="e9be90caef40461db05c2b9b966c06053e850ca373ebd1f5f8baca44f028913a"
Jul 01 20:49:03.500975 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.500943    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9be90caef40461db05c2b9b966c06053e850ca373ebd1f5f8baca44f028913a"} err="failed to get container status \"e9be90caef40461db05c2b9b966c06053e850ca373ebd1f5f8baca44f028913a\": rpc error: code = NotFound desc = could not find container \"e9be90caef40461db05c2b9b966c06053e850ca373ebd1f5f8baca44f028913a\": container with ID starting with e9be90caef40461db05c2b9b966c06053e850ca373ebd1f5f8baca44f028913a not found: ID does not exist"
Jul 01 20:49:03.500975 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.500955    2564 scope.go:117] "RemoveContainer" containerID="016a29a290a82ecec1b05b00a2a32ab36ea20b5a58e3ce1682be826de6643285"
Jul 01 20:49:03.501151 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:49:03.501137    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"016a29a290a82ecec1b05b00a2a32ab36ea20b5a58e3ce1682be826de6643285\": container with ID starting with 016a29a290a82ecec1b05b00a2a32ab36ea20b5a58e3ce1682be826de6643285 not found: ID does not exist" containerID="016a29a290a82ecec1b05b00a2a32ab36ea20b5a58e3ce1682be826de6643285"
Jul 01 20:49:03.501192 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.501154    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"016a29a290a82ecec1b05b00a2a32ab36ea20b5a58e3ce1682be826de6643285"} err="failed to get container status \"016a29a290a82ecec1b05b00a2a32ab36ea20b5a58e3ce1682be826de6643285\": rpc error: code = NotFound desc = could not find container \"016a29a290a82ecec1b05b00a2a32ab36ea20b5a58e3ce1682be826de6643285\": container with ID starting with 016a29a290a82ecec1b05b00a2a32ab36ea20b5a58e3ce1682be826de6643285 not found: ID does not exist"
Jul 01 20:49:03.501192 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.501166    2564 scope.go:117] "RemoveContainer" containerID="e2be532576140f63d68169351a95c3a4f9169daf3592c96917c1bc0195de1ee9"
Jul 01 20:49:03.501352 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:49:03.501338    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2be532576140f63d68169351a95c3a4f9169daf3592c96917c1bc0195de1ee9\": container with ID starting with e2be532576140f63d68169351a95c3a4f9169daf3592c96917c1bc0195de1ee9 not found: ID does not exist" containerID="e2be532576140f63d68169351a95c3a4f9169daf3592c96917c1bc0195de1ee9"
Jul 01 20:49:03.501391 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.501355    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2be532576140f63d68169351a95c3a4f9169daf3592c96917c1bc0195de1ee9"} err="failed to get container status \"e2be532576140f63d68169351a95c3a4f9169daf3592c96917c1bc0195de1ee9\": rpc error: code = NotFound desc = could not find container \"e2be532576140f63d68169351a95c3a4f9169daf3592c96917c1bc0195de1ee9\": container with ID starting with e2be532576140f63d68169351a95c3a4f9169daf3592c96917c1bc0195de1ee9 not found: ID does not exist"
Jul 01 20:49:03.501391 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.501369    2564 scope.go:117] "RemoveContainer" containerID="8d8d8cd6e79490d9b35a6c3c79fcf9a73dea18fd376d2a6e5ae0356aa8f93895"
Jul 01 20:49:03.501563 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:49:03.501548    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d8d8cd6e79490d9b35a6c3c79fcf9a73dea18fd376d2a6e5ae0356aa8f93895\": container with ID starting with 8d8d8cd6e79490d9b35a6c3c79fcf9a73dea18fd376d2a6e5ae0356aa8f93895 not found: ID does not exist" containerID="8d8d8cd6e79490d9b35a6c3c79fcf9a73dea18fd376d2a6e5ae0356aa8f93895"
Jul 01 20:49:03.501622 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.501571    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d8d8cd6e79490d9b35a6c3c79fcf9a73dea18fd376d2a6e5ae0356aa8f93895"} err="failed to get container status \"8d8d8cd6e79490d9b35a6c3c79fcf9a73dea18fd376d2a6e5ae0356aa8f93895\": rpc error: code = NotFound desc = could not find container \"8d8d8cd6e79490d9b35a6c3c79fcf9a73dea18fd376d2a6e5ae0356aa8f93895\": container with ID starting with 8d8d8cd6e79490d9b35a6c3c79fcf9a73dea18fd376d2a6e5ae0356aa8f93895 not found: ID does not exist"
Jul 01 20:49:03.501622 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.501590    2564 scope.go:117] "RemoveContainer" containerID="97091edf3d8b2392f5f65b0373fc5b06ba273abcda5ffd7428b045c2fbc5ef12"
Jul 01 20:49:03.501810 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:49:03.501791    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97091edf3d8b2392f5f65b0373fc5b06ba273abcda5ffd7428b045c2fbc5ef12\": container with ID starting with 97091edf3d8b2392f5f65b0373fc5b06ba273abcda5ffd7428b045c2fbc5ef12 not found: ID does not exist" containerID="97091edf3d8b2392f5f65b0373fc5b06ba273abcda5ffd7428b045c2fbc5ef12"
Jul 01 20:49:03.501850 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.501814    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97091edf3d8b2392f5f65b0373fc5b06ba273abcda5ffd7428b045c2fbc5ef12"} err="failed to get container status \"97091edf3d8b2392f5f65b0373fc5b06ba273abcda5ffd7428b045c2fbc5ef12\": rpc error: code = NotFound desc = could not find container \"97091edf3d8b2392f5f65b0373fc5b06ba273abcda5ffd7428b045c2fbc5ef12\": container with ID starting with 97091edf3d8b2392f5f65b0373fc5b06ba273abcda5ffd7428b045c2fbc5ef12 not found: ID does not exist"
Jul 01 20:49:03.513076 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.513054    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 20:49:03.513303 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.513291    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerName="alertmanager"
Jul 01 20:49:03.513349 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.513304    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerName="alertmanager"
Jul 01 20:49:03.513349 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.513313    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerName="prom-label-proxy"
Jul 01 20:49:03.513349 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.513318    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerName="prom-label-proxy"
Jul 01 20:49:03.513349 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.513329    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerName="init-config-reloader"
Jul 01 20:49:03.513349 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.513334    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerName="init-config-reloader"
Jul 01 20:49:03.513349 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.513340    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerName="config-reloader"
Jul 01 20:49:03.513349 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.513345    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerName="config-reloader"
Jul 01 20:49:03.513604 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.513353    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerName="kube-rbac-proxy"
Jul 01 20:49:03.513604 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.513357    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerName="kube-rbac-proxy"
Jul 01 20:49:03.513604 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.513363    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerName="kube-rbac-proxy-web"
Jul 01 20:49:03.513604 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.513369    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerName="kube-rbac-proxy-web"
Jul 01 20:49:03.513604 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.513381    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerName="kube-rbac-proxy-metric"
Jul 01 20:49:03.513604 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.513386    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerName="kube-rbac-proxy-metric"
Jul 01 20:49:03.513604 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.513433    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerName="config-reloader"
Jul 01 20:49:03.513604 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.513439    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerName="kube-rbac-proxy-web"
Jul 01 20:49:03.513604 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.513446    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerName="kube-rbac-proxy-metric"
Jul 01 20:49:03.513604 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.513452    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerName="alertmanager"
Jul 01 20:49:03.513604 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.513458    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerName="prom-label-proxy"
Jul 01 20:49:03.513604 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.513465    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" containerName="kube-rbac-proxy"
Jul 01 20:49:03.516949 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.516933    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.519959 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.519939    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 01 20:49:03.519959 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.519955    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 01 20:49:03.520087 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.519957    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 01 20:49:03.520087 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.519981    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 01 20:49:03.520243 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.520227    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 01 20:49:03.520340 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.520324    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 01 20:49:03.520416 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.520395    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 01 20:49:03.520539 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.520524    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 01 20:49:03.521539 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.521524    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ldxkw\""
Jul 01 20:49:03.525250 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.525234    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 01 20:49:03.534578 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.534561    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 20:49:03.644483 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.644452    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8fc5460b-1728-45b9-b9a4-f26031e5e37e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.644628 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.644493    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8fc5460b-1728-45b9-b9a4-f26031e5e37e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.644628 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.644536    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr2lz\" (UniqueName: \"kubernetes.io/projected/8fc5460b-1728-45b9-b9a4-f26031e5e37e-kube-api-access-vr2lz\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.644628 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.644579    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8fc5460b-1728-45b9-b9a4-f26031e5e37e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.644805 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.644641    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fc5460b-1728-45b9-b9a4-f26031e5e37e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.644805 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.644742    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8fc5460b-1728-45b9-b9a4-f26031e5e37e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.644805 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.644780    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8fc5460b-1728-45b9-b9a4-f26031e5e37e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.645038 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.644807    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8fc5460b-1728-45b9-b9a4-f26031e5e37e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.645038 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.644837    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8fc5460b-1728-45b9-b9a4-f26031e5e37e-config-volume\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.645038 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.644885    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fc5460b-1728-45b9-b9a4-f26031e5e37e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.645038 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.644911    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8fc5460b-1728-45b9-b9a4-f26031e5e37e-config-out\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.645038 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.644931    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8fc5460b-1728-45b9-b9a4-f26031e5e37e-web-config\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.645038 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.644954    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8fc5460b-1728-45b9-b9a4-f26031e5e37e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.745499 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.745434    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8fc5460b-1728-45b9-b9a4-f26031e5e37e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.745499 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.745464    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8fc5460b-1728-45b9-b9a4-f26031e5e37e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.745499 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.745481    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8fc5460b-1728-45b9-b9a4-f26031e5e37e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.745499 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.745499    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8fc5460b-1728-45b9-b9a4-f26031e5e37e-config-volume\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.745755 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.745701    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fc5460b-1728-45b9-b9a4-f26031e5e37e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.745755 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.745746    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8fc5460b-1728-45b9-b9a4-f26031e5e37e-config-out\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.745890 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.745774    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8fc5460b-1728-45b9-b9a4-f26031e5e37e-web-config\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.745890 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.745818    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8fc5460b-1728-45b9-b9a4-f26031e5e37e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.745890 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.745848    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8fc5460b-1728-45b9-b9a4-f26031e5e37e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.746240 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.746210    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8fc5460b-1728-45b9-b9a4-f26031e5e37e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.747563 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.746413    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fc5460b-1728-45b9-b9a4-f26031e5e37e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.747563 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.746478    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8fc5460b-1728-45b9-b9a4-f26031e5e37e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.747563 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.746519    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr2lz\" (UniqueName: \"kubernetes.io/projected/8fc5460b-1728-45b9-b9a4-f26031e5e37e-kube-api-access-vr2lz\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.747563 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.746553    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8fc5460b-1728-45b9-b9a4-f26031e5e37e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.747563 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.746584    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fc5460b-1728-45b9-b9a4-f26031e5e37e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.747563 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.747519    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fc5460b-1728-45b9-b9a4-f26031e5e37e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.748526 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.748503    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8fc5460b-1728-45b9-b9a4-f26031e5e37e-web-config\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.748662 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.748564    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8fc5460b-1728-45b9-b9a4-f26031e5e37e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.748733 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.748678    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8fc5460b-1728-45b9-b9a4-f26031e5e37e-config-out\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.748792 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.748775    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8fc5460b-1728-45b9-b9a4-f26031e5e37e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.749009 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.748978    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8fc5460b-1728-45b9-b9a4-f26031e5e37e-config-volume\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.749304 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.749285    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8fc5460b-1728-45b9-b9a4-f26031e5e37e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.749657 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.749636    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8fc5460b-1728-45b9-b9a4-f26031e5e37e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.749896 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.749878    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8fc5460b-1728-45b9-b9a4-f26031e5e37e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.750182 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.750168    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8fc5460b-1728-45b9-b9a4-f26031e5e37e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.755305 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.755290    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr2lz\" (UniqueName: \"kubernetes.io/projected/8fc5460b-1728-45b9-b9a4-f26031e5e37e-kube-api-access-vr2lz\") pod \"alertmanager-main-0\" (UID: \"8fc5460b-1728-45b9-b9a4-f26031e5e37e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.826348 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.826313    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 20:49:03.968744 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:03.968550    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 20:49:03.971008 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:49:03.970984    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc5460b_1728_45b9_b9a4_f26031e5e37e.slice/crio-9ddc716265002d37fa74d185f782222ee190ce4d3882d4c84a205b01bb3c18bd WatchSource:0}: Error finding container 9ddc716265002d37fa74d185f782222ee190ce4d3882d4c84a205b01bb3c18bd: Status 404 returned error can't find the container with id 9ddc716265002d37fa74d185f782222ee190ce4d3882d4c84a205b01bb3c18bd
Jul 01 20:49:04.462231 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:04.462194    2564 generic.go:356] "Generic (PLEG): container finished" podID="8fc5460b-1728-45b9-b9a4-f26031e5e37e" containerID="97bbe0dfade67e7da97c7dbab8cc0e0f42fb61a65676cc34ea3e7b99302641e7" exitCode=0
Jul 01 20:49:04.462595 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:04.462253    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fc5460b-1728-45b9-b9a4-f26031e5e37e","Type":"ContainerDied","Data":"97bbe0dfade67e7da97c7dbab8cc0e0f42fb61a65676cc34ea3e7b99302641e7"}
Jul 01 20:49:04.462595 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:04.462337    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fc5460b-1728-45b9-b9a4-f26031e5e37e","Type":"ContainerStarted","Data":"9ddc716265002d37fa74d185f782222ee190ce4d3882d4c84a205b01bb3c18bd"}
Jul 01 20:49:04.758195 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:04.758146    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de41a7ac-ca29-46f1-8145-f294eb2cbd56" path="/var/lib/kubelet/pods/de41a7ac-ca29-46f1-8145-f294eb2cbd56/volumes"
Jul 01 20:49:05.468816 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:05.468781    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fc5460b-1728-45b9-b9a4-f26031e5e37e","Type":"ContainerStarted","Data":"ed391496d4f32f74885cd62dc4258443dd8f2dfdb803589bd69708f59b0d44df"}
Jul 01 20:49:05.469257 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:05.468844    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fc5460b-1728-45b9-b9a4-f26031e5e37e","Type":"ContainerStarted","Data":"1666ae1d827b74b3cbb5b0b20d19410bb9572c65421b5a130b4b9de97e357cc8"}
Jul 01 20:49:05.469257 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:05.468891    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fc5460b-1728-45b9-b9a4-f26031e5e37e","Type":"ContainerStarted","Data":"b3c42a96c5f0ca3cede226501c65afebd175dadbd0feb78dd4843bbcbe723b36"}
Jul 01 20:49:05.469257 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:05.468919    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fc5460b-1728-45b9-b9a4-f26031e5e37e","Type":"ContainerStarted","Data":"dbba8d3bbe1bdb442a5aa5fbc667064ef4099155f42dc71861e550706c6959fe"}
Jul 01 20:49:05.469257 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:05.468943    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fc5460b-1728-45b9-b9a4-f26031e5e37e","Type":"ContainerStarted","Data":"fbf353a6dfc2e7cfafbf7bea5d3e36d9cb46a2187aa3fab434bb5f1ad28a9904"}
Jul 01 20:49:05.469257 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:05.468965    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fc5460b-1728-45b9-b9a4-f26031e5e37e","Type":"ContainerStarted","Data":"6a47390aa81b2290e082d9616f51de15e069155b22c47181d6b3572a403e83d5"}
Jul 01 20:49:05.524214 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:05.524165    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.506487747 podStartE2EDuration="2.524150263s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.662516ms" isStatefulPod=false podCreationTimestamp="2026-07-01 20:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:49:05.521679176 +0000 UTC m=+145.197694779" watchObservedRunningTime="2026-07-01 20:49:05.524150263 +0000 UTC m=+145.200165829"
Jul 01 20:49:06.004389 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.004358    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 20:49:06.004680 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.004630    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerName="prometheus" containerID="cri-o://55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9" gracePeriod=600
Jul 01 20:49:06.005040 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.004673    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerName="kube-rbac-proxy" containerID="cri-o://a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b" gracePeriod=600
Jul 01 20:49:06.005040 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.004691    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerName="kube-rbac-proxy-web" containerID="cri-o://6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887" gracePeriod=600
Jul 01 20:49:06.005040 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.004676    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerName="thanos-sidecar" containerID="cri-o://59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7" gracePeriod=600
Jul 01 20:49:06.005040 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.004685    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerName="config-reloader" containerID="cri-o://ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6" gracePeriod=600
Jul 01 20:49:06.005040 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.004760    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerName="kube-rbac-proxy-thanos" containerID="cri-o://e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31" gracePeriod=600
Jul 01 20:49:06.049820 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:49:06.049784    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ab9a24_1175_41f1_b736_93bc2ca01ae8.slice/crio-conmon-ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ab9a24_1175_41f1_b736_93bc2ca01ae8.slice/crio-conmon-59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ab9a24_1175_41f1_b736_93bc2ca01ae8.slice/crio-conmon-6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ab9a24_1175_41f1_b736_93bc2ca01ae8.slice/crio-conmon-55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ab9a24_1175_41f1_b736_93bc2ca01ae8.slice/crio-59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7.scope\": RecentStats: unable to find data in memory cache]"
Jul 01 20:49:06.267533 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.267510    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.368370 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.368342    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-configmap-metrics-client-ca\") pod \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") "
Jul 01 20:49:06.368544 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.368381    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-config\") pod \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") "
Jul 01 20:49:06.368544 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.368406    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") "
Jul 01 20:49:06.368544 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.368434    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-k8s-db\") pod \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") "
Jul 01 20:49:06.368544 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.368472    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-configmap-serving-certs-ca-bundle\") pod \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") "
Jul 01 20:49:06.368544 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.368497    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-887wf\" (UniqueName: \"kubernetes.io/projected/14ab9a24-1175-41f1-b736-93bc2ca01ae8-kube-api-access-887wf\") pod \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") "
Jul 01 20:49:06.368544 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.368518    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-k8s-rulefiles-1\") pod \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") "
Jul 01 20:49:06.368828 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.368550    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-prometheus-k8s-tls\") pod \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") "
Jul 01 20:49:06.368828 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.368574    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-metrics-client-certs\") pod \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") "
Jul 01 20:49:06.368828 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.368599    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-thanos-prometheus-http-client-file\") pod \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") "
Jul 01 20:49:06.368828 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.368630    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") "
Jul 01 20:49:06.368828 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.368656    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-grpc-tls\") pod \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") "
Jul 01 20:49:06.368828 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.368689    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-web-config\") pod \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") "
Jul 01 20:49:06.368828 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.368702    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "14ab9a24-1175-41f1-b736-93bc2ca01ae8" (UID: "14ab9a24-1175-41f1-b736-93bc2ca01ae8"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:49:06.368828 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.368715    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14ab9a24-1175-41f1-b736-93bc2ca01ae8-config-out\") pod \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") "
Jul 01 20:49:06.368828 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.368747    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-trusted-ca-bundle\") pod \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") "
Jul 01 20:49:06.369287 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.368933    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-configmap-kubelet-serving-ca-bundle\") pod \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") "
Jul 01 20:49:06.369287 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.369011    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14ab9a24-1175-41f1-b736-93bc2ca01ae8-tls-assets\") pod \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") "
Jul 01 20:49:06.369287 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.369203    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "14ab9a24-1175-41f1-b736-93bc2ca01ae8" (UID: "14ab9a24-1175-41f1-b736-93bc2ca01ae8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:49:06.369439 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.369368    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "14ab9a24-1175-41f1-b736-93bc2ca01ae8" (UID: "14ab9a24-1175-41f1-b736-93bc2ca01ae8"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:49:06.369529 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.369040    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-k8s-rulefiles-0\") pod \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") "
Jul 01 20:49:06.369636 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.369623    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-k8s-rulefiles-2\") pod \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") "
Jul 01 20:49:06.369754 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.369740    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-kube-rbac-proxy\") pod \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\" (UID: \"14ab9a24-1175-41f1-b736-93bc2ca01ae8\") "
Jul 01 20:49:06.370049 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.370019    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "14ab9a24-1175-41f1-b736-93bc2ca01ae8" (UID: "14ab9a24-1175-41f1-b736-93bc2ca01ae8"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:49:06.370315 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.370297    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-configmap-metrics-client-ca\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:06.370418 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.370405    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:06.370508 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.370495    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:06.370601 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.370589    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:06.371236 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.370420    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "14ab9a24-1175-41f1-b736-93bc2ca01ae8" (UID: "14ab9a24-1175-41f1-b736-93bc2ca01ae8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:49:06.371236 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.370563    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "14ab9a24-1175-41f1-b736-93bc2ca01ae8" (UID: "14ab9a24-1175-41f1-b736-93bc2ca01ae8"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:49:06.371394 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.371333    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "14ab9a24-1175-41f1-b736-93bc2ca01ae8" (UID: "14ab9a24-1175-41f1-b736-93bc2ca01ae8"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 20:49:06.371394 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.371340    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "14ab9a24-1175-41f1-b736-93bc2ca01ae8" (UID: "14ab9a24-1175-41f1-b736-93bc2ca01ae8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:49:06.373093 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.373048    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ab9a24-1175-41f1-b736-93bc2ca01ae8-kube-api-access-887wf" (OuterVolumeSpecName: "kube-api-access-887wf") pod "14ab9a24-1175-41f1-b736-93bc2ca01ae8" (UID: "14ab9a24-1175-41f1-b736-93bc2ca01ae8"). InnerVolumeSpecName "kube-api-access-887wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:49:06.373093 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.373053    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "14ab9a24-1175-41f1-b736-93bc2ca01ae8" (UID: "14ab9a24-1175-41f1-b736-93bc2ca01ae8"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:49:06.373245 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.373188    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "14ab9a24-1175-41f1-b736-93bc2ca01ae8" (UID: "14ab9a24-1175-41f1-b736-93bc2ca01ae8"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:49:06.373245 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.373231    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ab9a24-1175-41f1-b736-93bc2ca01ae8-config-out" (OuterVolumeSpecName: "config-out") pod "14ab9a24-1175-41f1-b736-93bc2ca01ae8" (UID: "14ab9a24-1175-41f1-b736-93bc2ca01ae8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 20:49:06.373408 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.373379    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "14ab9a24-1175-41f1-b736-93bc2ca01ae8" (UID: "14ab9a24-1175-41f1-b736-93bc2ca01ae8"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:49:06.373466 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.373419    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "14ab9a24-1175-41f1-b736-93bc2ca01ae8" (UID: "14ab9a24-1175-41f1-b736-93bc2ca01ae8"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:49:06.373536 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.373510    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "14ab9a24-1175-41f1-b736-93bc2ca01ae8" (UID: "14ab9a24-1175-41f1-b736-93bc2ca01ae8"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:49:06.373587 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.373571    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-config" (OuterVolumeSpecName: "config") pod "14ab9a24-1175-41f1-b736-93bc2ca01ae8" (UID: "14ab9a24-1175-41f1-b736-93bc2ca01ae8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:49:06.373653 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.373634    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "14ab9a24-1175-41f1-b736-93bc2ca01ae8" (UID: "14ab9a24-1175-41f1-b736-93bc2ca01ae8"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:49:06.373948 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.373930    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ab9a24-1175-41f1-b736-93bc2ca01ae8-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "14ab9a24-1175-41f1-b736-93bc2ca01ae8" (UID: "14ab9a24-1175-41f1-b736-93bc2ca01ae8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:49:06.374243 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.374224    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "14ab9a24-1175-41f1-b736-93bc2ca01ae8" (UID: "14ab9a24-1175-41f1-b736-93bc2ca01ae8"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:49:06.382427 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.382399    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-web-config" (OuterVolumeSpecName: "web-config") pod "14ab9a24-1175-41f1-b736-93bc2ca01ae8" (UID: "14ab9a24-1175-41f1-b736-93bc2ca01ae8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:49:06.471512 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.471484    2564 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-config\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:06.471512 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.471504    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:06.471512 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.471514    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-k8s-db\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:06.471995 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.471523    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-887wf\" (UniqueName: \"kubernetes.io/projected/14ab9a24-1175-41f1-b736-93bc2ca01ae8-kube-api-access-887wf\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:06.471995 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.471532    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:06.471995 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.471541    2564 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-metrics-client-certs\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:06.471995 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.471549    2564 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:06.471995 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.471558    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:06.471995 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.471566    2564 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-grpc-tls\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:06.471995 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.471594    2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-web-config\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:06.471995 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.471602    2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14ab9a24-1175-41f1-b736-93bc2ca01ae8-config-out\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:06.471995 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.471609    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:06.471995 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.471617    2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14ab9a24-1175-41f1-b736-93bc2ca01ae8-tls-assets\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:06.471995 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.471625    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:06.471995 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.471633    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14ab9a24-1175-41f1-b736-93bc2ca01ae8-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:06.471995 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.471641    2564 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14ab9a24-1175-41f1-b736-93bc2ca01ae8-secret-kube-rbac-proxy\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:49:06.478137 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.478113    2564 generic.go:356] "Generic (PLEG): container finished" podID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerID="e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31" exitCode=0
Jul 01 20:49:06.478137 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.478136    2564 generic.go:356] "Generic (PLEG): container finished" podID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerID="a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b" exitCode=0
Jul 01 20:49:06.478258 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.478145    2564 generic.go:356] "Generic (PLEG): container finished" podID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerID="6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887" exitCode=0
Jul 01 20:49:06.478258 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.478153    2564 generic.go:356] "Generic (PLEG): container finished" podID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerID="59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7" exitCode=0
Jul 01 20:49:06.478258 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.478160    2564 generic.go:356] "Generic (PLEG): container finished" podID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerID="ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6" exitCode=0
Jul 01 20:49:06.478258 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.478167    2564 generic.go:356] "Generic (PLEG): container finished" podID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerID="55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9" exitCode=0
Jul 01 20:49:06.478258 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.478187    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.478258 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.478194    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14ab9a24-1175-41f1-b736-93bc2ca01ae8","Type":"ContainerDied","Data":"e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31"}
Jul 01 20:49:06.478497 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.478289    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14ab9a24-1175-41f1-b736-93bc2ca01ae8","Type":"ContainerDied","Data":"a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b"}
Jul 01 20:49:06.478497 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.478319    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14ab9a24-1175-41f1-b736-93bc2ca01ae8","Type":"ContainerDied","Data":"6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887"}
Jul 01 20:49:06.478497 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.478346    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14ab9a24-1175-41f1-b736-93bc2ca01ae8","Type":"ContainerDied","Data":"59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7"}
Jul 01 20:49:06.478497 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.478361    2564 scope.go:117] "RemoveContainer" containerID="e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31"
Jul 01 20:49:06.478497 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.478370    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14ab9a24-1175-41f1-b736-93bc2ca01ae8","Type":"ContainerDied","Data":"ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6"}
Jul 01 20:49:06.478497 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.478394    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14ab9a24-1175-41f1-b736-93bc2ca01ae8","Type":"ContainerDied","Data":"55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9"}
Jul 01 20:49:06.478497 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.478419    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14ab9a24-1175-41f1-b736-93bc2ca01ae8","Type":"ContainerDied","Data":"d40f605763df9fdb56cb2e09b274c5bb677747c7c6b892478a2076559b18147a"}
Jul 01 20:49:06.486262 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.486240    2564 scope.go:117] "RemoveContainer" containerID="a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b"
Jul 01 20:49:06.493102 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.493087    2564 scope.go:117] "RemoveContainer" containerID="6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887"
Jul 01 20:49:06.499196 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.499182    2564 scope.go:117] "RemoveContainer" containerID="59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7"
Jul 01 20:49:06.503205 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.503185    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 20:49:06.506412 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.506343    2564 scope.go:117] "RemoveContainer" containerID="ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6"
Jul 01 20:49:06.508280 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.508240    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 20:49:06.512828 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.512813    2564 scope.go:117] "RemoveContainer" containerID="55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9"
Jul 01 20:49:06.519630 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.519612    2564 scope.go:117] "RemoveContainer" containerID="e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69"
Jul 01 20:49:06.525844 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.525830    2564 scope.go:117] "RemoveContainer" containerID="e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31"
Jul 01 20:49:06.526094 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:49:06.526076    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31\": container with ID starting with e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31 not found: ID does not exist" containerID="e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31"
Jul 01 20:49:06.526146 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.526101    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31"} err="failed to get container status \"e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31\": rpc error: code = NotFound desc = could not find container \"e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31\": container with ID starting with e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31 not found: ID does not exist"
Jul 01 20:49:06.526146 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.526118    2564 scope.go:117] "RemoveContainer" containerID="a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b"
Jul 01 20:49:06.526348 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:49:06.526332    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b\": container with ID starting with a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b not found: ID does not exist" containerID="a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b"
Jul 01 20:49:06.526387 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.526353    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b"} err="failed to get container status \"a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b\": rpc error: code = NotFound desc = could not find container \"a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b\": container with ID starting with a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b not found: ID does not exist"
Jul 01 20:49:06.526387 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.526367    2564 scope.go:117] "RemoveContainer" containerID="6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887"
Jul 01 20:49:06.526556 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:49:06.526542    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887\": container with ID starting with 6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887 not found: ID does not exist" containerID="6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887"
Jul 01 20:49:06.526596 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.526560    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887"} err="failed to get container status \"6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887\": rpc error: code = NotFound desc = could not find container \"6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887\": container with ID starting with 6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887 not found: ID does not exist"
Jul 01 20:49:06.526596 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.526571    2564 scope.go:117] "RemoveContainer" containerID="59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7"
Jul 01 20:49:06.526766 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:49:06.526750    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7\": container with ID starting with 59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7 not found: ID does not exist" containerID="59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7"
Jul 01 20:49:06.526803 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.526770    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7"} err="failed to get container status \"59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7\": rpc error: code = NotFound desc = could not find container \"59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7\": container with ID starting with 59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7 not found: ID does not exist"
Jul 01 20:49:06.526803 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.526781    2564 scope.go:117] "RemoveContainer" containerID="ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6"
Jul 01 20:49:06.527031 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:49:06.527013    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6\": container with ID starting with ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6 not found: ID does not exist" containerID="ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6"
Jul 01 20:49:06.527083 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.527038    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6"} err="failed to get container status \"ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6\": rpc error: code = NotFound desc = could not find container \"ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6\": container with ID starting with ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6 not found: ID does not exist"
Jul 01 20:49:06.527083 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.527053    2564 scope.go:117] "RemoveContainer" containerID="55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9"
Jul 01 20:49:06.527265 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:49:06.527251    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9\": container with ID starting with 55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9 not found: ID does not exist" containerID="55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9"
Jul 01 20:49:06.527307 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.527268    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9"} err="failed to get container status \"55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9\": rpc error: code = NotFound desc = could not find container \"55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9\": container with ID starting with 55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9 not found: ID does not exist"
Jul 01 20:49:06.527307 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.527279    2564 scope.go:117] "RemoveContainer" containerID="e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69"
Jul 01 20:49:06.527607 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:49:06.527587    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69\": container with ID starting with e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69 not found: ID does not exist" containerID="e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69"
Jul 01 20:49:06.527688 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.527613    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69"} err="failed to get container status \"e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69\": rpc error: code = NotFound desc = could not find container \"e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69\": container with ID starting with e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69 not found: ID does not exist"
Jul 01 20:49:06.527688 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.527631    2564 scope.go:117] "RemoveContainer" containerID="e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31"
Jul 01 20:49:06.527913 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.527893    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31"} err="failed to get container status \"e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31\": rpc error: code = NotFound desc = could not find container \"e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31\": container with ID starting with e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31 not found: ID does not exist"
Jul 01 20:49:06.527913 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.527913    2564 scope.go:117] "RemoveContainer" containerID="a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b"
Jul 01 20:49:06.528171 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.528151    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b"} err="failed to get container status \"a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b\": rpc error: code = NotFound desc = could not find container \"a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b\": container with ID starting with a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b not found: ID does not exist"
Jul 01 20:49:06.528171 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.528171    2564 scope.go:117] "RemoveContainer" containerID="6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887"
Jul 01 20:49:06.528482 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.528420    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887"} err="failed to get container status \"6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887\": rpc error: code = NotFound desc = could not find container \"6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887\": container with ID starting with 6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887 not found: ID does not exist"
Jul 01 20:49:06.528482 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.528445    2564 scope.go:117] "RemoveContainer" containerID="59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7"
Jul 01 20:49:06.528688 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.528672    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7"} err="failed to get container status \"59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7\": rpc error: code = NotFound desc = could not find container \"59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7\": container with ID starting with 59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7 not found: ID does not exist"
Jul 01 20:49:06.528738 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.528689    2564 scope.go:117] "RemoveContainer" containerID="ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6"
Jul 01 20:49:06.528914 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.528897    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6"} err="failed to get container status \"ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6\": rpc error: code = NotFound desc = could not find container \"ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6\": container with ID starting with ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6 not found: ID does not exist"
Jul 01 20:49:06.528959 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.528915    2564 scope.go:117] "RemoveContainer" containerID="55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9"
Jul 01 20:49:06.529127 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.529109    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9"} err="failed to get container status \"55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9\": rpc error: code = NotFound desc = could not find container \"55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9\": container with ID starting with 55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9 not found: ID does not exist"
Jul 01 20:49:06.529167 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.529128    2564 scope.go:117] "RemoveContainer" containerID="e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69"
Jul 01 20:49:06.529331 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.529313    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69"} err="failed to get container status \"e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69\": rpc error: code = NotFound desc = could not find container \"e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69\": container with ID starting with e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69 not found: ID does not exist"
Jul 01 20:49:06.529405 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.529333    2564 scope.go:117] "RemoveContainer" containerID="e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31"
Jul 01 20:49:06.529550 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.529535    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31"} err="failed to get container status \"e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31\": rpc error: code = NotFound desc = could not find container \"e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31\": container with ID starting with e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31 not found: ID does not exist"
Jul 01 20:49:06.529594 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.529550    2564 scope.go:117] "RemoveContainer" containerID="a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b"
Jul 01 20:49:06.529727 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.529707    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b"} err="failed to get container status \"a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b\": rpc error: code = NotFound desc = could not find container \"a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b\": container with ID starting with a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b not found: ID does not exist"
Jul 01 20:49:06.529768 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.529727    2564 scope.go:117] "RemoveContainer" containerID="6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887"
Jul 01 20:49:06.529907 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.529886    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887"} err="failed to get container status \"6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887\": rpc error: code = NotFound desc = could not find container \"6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887\": container with ID starting with 6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887 not found: ID does not exist"
Jul 01 20:49:06.529959 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.529906    2564 scope.go:117] "RemoveContainer" containerID="59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7"
Jul 01 20:49:06.530083 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.530067    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7"} err="failed to get container status \"59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7\": rpc error: code = NotFound desc = could not find container \"59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7\": container with ID starting with 59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7 not found: ID does not exist"
Jul 01 20:49:06.530121 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.530083    2564 scope.go:117] "RemoveContainer" containerID="ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6"
Jul 01 20:49:06.530248 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.530234    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6"} err="failed to get container status \"ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6\": rpc error: code = NotFound desc = could not find container \"ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6\": container with ID starting with ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6 not found: ID does not exist"
Jul 01 20:49:06.530290 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.530247    2564 scope.go:117] "RemoveContainer" containerID="55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9"
Jul 01 20:49:06.530472 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.530453    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9"} err="failed to get container status \"55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9\": rpc error: code = NotFound desc = could not find container \"55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9\": container with ID starting with 55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9 not found: ID does not exist"
Jul 01 20:49:06.530516 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.530474    2564 scope.go:117] "RemoveContainer" containerID="e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69"
Jul 01 20:49:06.530687 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.530671    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69"} err="failed to get container status \"e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69\": rpc error: code = NotFound desc = could not find container \"e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69\": container with ID starting with e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69 not found: ID does not exist"
Jul 01 20:49:06.530725 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.530688    2564 scope.go:117] "RemoveContainer" containerID="e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31"
Jul 01 20:49:06.530931 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.530912    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31"} err="failed to get container status \"e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31\": rpc error: code = NotFound desc = could not find container \"e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31\": container with ID starting with e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31 not found: ID does not exist"
Jul 01 20:49:06.530981 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.530932    2564 scope.go:117] "RemoveContainer" containerID="a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b"
Jul 01 20:49:06.531138 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.531121    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b"} err="failed to get container status \"a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b\": rpc error: code = NotFound desc = could not find container \"a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b\": container with ID starting with a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b not found: ID does not exist"
Jul 01 20:49:06.531178 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.531139    2564 scope.go:117] "RemoveContainer" containerID="6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887"
Jul 01 20:49:06.531327 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.531311    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887"} err="failed to get container status \"6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887\": rpc error: code = NotFound desc = could not find container \"6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887\": container with ID starting with 6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887 not found: ID does not exist"
Jul 01 20:49:06.531365 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.531328    2564 scope.go:117] "RemoveContainer" containerID="59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7"
Jul 01 20:49:06.531518 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.531501    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7"} err="failed to get container status \"59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7\": rpc error: code = NotFound desc = could not find container \"59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7\": container with ID starting with 59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7 not found: ID does not exist"
Jul 01 20:49:06.531561 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.531519    2564 scope.go:117] "RemoveContainer" containerID="ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6"
Jul 01 20:49:06.531713 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.531696    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6"} err="failed to get container status \"ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6\": rpc error: code = NotFound desc = could not find container \"ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6\": container with ID starting with ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6 not found: ID does not exist"
Jul 01 20:49:06.531751 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.531713    2564 scope.go:117] "RemoveContainer" containerID="55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9"
Jul 01 20:49:06.531941 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.531922    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9"} err="failed to get container status \"55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9\": rpc error: code = NotFound desc = could not find container \"55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9\": container with ID starting with 55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9 not found: ID does not exist"
Jul 01 20:49:06.531993 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.531943    2564 scope.go:117] "RemoveContainer" containerID="e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69"
Jul 01 20:49:06.532127 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.532109    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69"} err="failed to get container status \"e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69\": rpc error: code = NotFound desc = could not find container \"e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69\": container with ID starting with e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69 not found: ID does not exist"
Jul 01 20:49:06.532169 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.532127    2564 scope.go:117] "RemoveContainer" containerID="e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31"
Jul 01 20:49:06.532333 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.532304    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31"} err="failed to get container status \"e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31\": rpc error: code = NotFound desc = could not find container \"e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31\": container with ID starting with e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31 not found: ID does not exist"
Jul 01 20:49:06.532376 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.532333    2564 scope.go:117] "RemoveContainer" containerID="a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b"
Jul 01 20:49:06.532547 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.532525    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b"} err="failed to get container status \"a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b\": rpc error: code = NotFound desc = could not find container \"a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b\": container with ID starting with a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b not found: ID does not exist"
Jul 01 20:49:06.532547 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.532546    2564 scope.go:117] "RemoveContainer" containerID="6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887"
Jul 01 20:49:06.532755 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.532737    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887"} err="failed to get container status \"6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887\": rpc error: code = NotFound desc = could not find container \"6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887\": container with ID starting with 6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887 not found: ID does not exist"
Jul 01 20:49:06.532807 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.532756    2564 scope.go:117] "RemoveContainer" containerID="59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7"
Jul 01 20:49:06.532961 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.532945    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7"} err="failed to get container status \"59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7\": rpc error: code = NotFound desc = could not find container \"59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7\": container with ID starting with 59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7 not found: ID does not exist"
Jul 01 20:49:06.533021 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.532962    2564 scope.go:117] "RemoveContainer" containerID="ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6"
Jul 01 20:49:06.533167 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.533152    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6"} err="failed to get container status \"ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6\": rpc error: code = NotFound desc = could not find container \"ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6\": container with ID starting with ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6 not found: ID does not exist"
Jul 01 20:49:06.533217 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.533168    2564 scope.go:117] "RemoveContainer" containerID="55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9"
Jul 01 20:49:06.533356 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.533337    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9"} err="failed to get container status \"55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9\": rpc error: code = NotFound desc = could not find container \"55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9\": container with ID starting with 55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9 not found: ID does not exist"
Jul 01 20:49:06.533409 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.533356    2564 scope.go:117] "RemoveContainer" containerID="e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69"
Jul 01 20:49:06.533523 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.533508    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69"} err="failed to get container status \"e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69\": rpc error: code = NotFound desc = could not find container \"e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69\": container with ID starting with e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69 not found: ID does not exist"
Jul 01 20:49:06.533523 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.533522    2564 scope.go:117] "RemoveContainer" containerID="e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31"
Jul 01 20:49:06.533746 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.533727    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31"} err="failed to get container status \"e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31\": rpc error: code = NotFound desc = could not find container \"e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31\": container with ID starting with e12d3f999520e304c2910b818d5a3c8b381c908d36a7f2ad009a181aa96eee31 not found: ID does not exist"
Jul 01 20:49:06.533813 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.533751    2564 scope.go:117] "RemoveContainer" containerID="a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b"
Jul 01 20:49:06.533986 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.533972    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b"} err="failed to get container status \"a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b\": rpc error: code = NotFound desc = could not find container \"a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b\": container with ID starting with a71cb94b2cd832121a49d284434c89827ead7902adb162a9f43865352b42780b not found: ID does not exist"
Jul 01 20:49:06.533986 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.533986    2564 scope.go:117] "RemoveContainer" containerID="6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887"
Jul 01 20:49:06.534201 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.534179    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887"} err="failed to get container status \"6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887\": rpc error: code = NotFound desc = could not find container \"6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887\": container with ID starting with 6429ed89a79eed1bccb9a3a23fd0b7f72babe65624959b29f02d1b3c43998887 not found: ID does not exist"
Jul 01 20:49:06.534201 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.534201    2564 scope.go:117] "RemoveContainer" containerID="59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7"
Jul 01 20:49:06.534375 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.534360    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7"} err="failed to get container status \"59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7\": rpc error: code = NotFound desc = could not find container \"59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7\": container with ID starting with 59398aaf39ab270c8fcffe682926008ee6ec32c5d32589bb994a4edf8f4e2fe7 not found: ID does not exist"
Jul 01 20:49:06.534375 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.534374    2564 scope.go:117] "RemoveContainer" containerID="ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6"
Jul 01 20:49:06.534567 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.534546    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6"} err="failed to get container status \"ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6\": rpc error: code = NotFound desc = could not find container \"ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6\": container with ID starting with ae8cd341a0081f9f72e7633dcba8a54f829bff123808c37c1f9409f91d3bd7f6 not found: ID does not exist"
Jul 01 20:49:06.534567 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.534565    2564 scope.go:117] "RemoveContainer" containerID="55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9"
Jul 01 20:49:06.534777 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.534760    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9"} err="failed to get container status \"55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9\": rpc error: code = NotFound desc = could not find container \"55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9\": container with ID starting with 55a6ede6f482be88a14f2e3bae92d0f0b195c090f563b3a6081de1396f2d84b9 not found: ID does not exist"
Jul 01 20:49:06.534817 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.534779    2564 scope.go:117] "RemoveContainer" containerID="e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69"
Jul 01 20:49:06.535008 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.534992    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69"} err="failed to get container status \"e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69\": rpc error: code = NotFound desc = could not find container \"e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69\": container with ID starting with e7f07c13575c75e5c2e10c3e9fcdf0b890b2bea445ad1accecbd9ac73b7d0c69 not found: ID does not exist"
Jul 01 20:49:06.538700 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.538676    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 20:49:06.538959 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.538945    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerName="thanos-sidecar"
Jul 01 20:49:06.538998 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.538961    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerName="thanos-sidecar"
Jul 01 20:49:06.538998 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.538976    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerName="kube-rbac-proxy-web"
Jul 01 20:49:06.538998 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.538981    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerName="kube-rbac-proxy-web"
Jul 01 20:49:06.538998 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.538987    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerName="config-reloader"
Jul 01 20:49:06.538998 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.538992    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerName="config-reloader"
Jul 01 20:49:06.539145 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.539005    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerName="init-config-reloader"
Jul 01 20:49:06.539145 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.539010    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerName="init-config-reloader"
Jul 01 20:49:06.539145 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.539016    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerName="kube-rbac-proxy"
Jul 01 20:49:06.539145 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.539024    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerName="kube-rbac-proxy"
Jul 01 20:49:06.539145 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.539031    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerName="kube-rbac-proxy-thanos"
Jul 01 20:49:06.539145 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.539036    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerName="kube-rbac-proxy-thanos"
Jul 01 20:49:06.539145 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.539046    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerName="prometheus"
Jul 01 20:49:06.539145 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.539051    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerName="prometheus"
Jul 01 20:49:06.539145 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.539094    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerName="kube-rbac-proxy-web"
Jul 01 20:49:06.539145 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.539105    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerName="kube-rbac-proxy"
Jul 01 20:49:06.539145 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.539113    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerName="kube-rbac-proxy-thanos"
Jul 01 20:49:06.539145 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.539119    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerName="prometheus"
Jul 01 20:49:06.539145 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.539125    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerName="thanos-sidecar"
Jul 01 20:49:06.539145 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.539130    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" containerName="config-reloader"
Jul 01 20:49:06.542082 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.542067    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.544746 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.544730    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 01 20:49:06.545090 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.545058    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 01 20:49:06.545214 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.545107    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 01 20:49:06.545214 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.545191    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 01 20:49:06.545214 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.545197    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 01 20:49:06.545214 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.545202    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dfsh0inbenim0\""
Jul 01 20:49:06.545214 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.545211    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 01 20:49:06.545583 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.545250    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 01 20:49:06.545583 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.545313    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-t2kph\""
Jul 01 20:49:06.545583 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.545504    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 01 20:49:06.545583 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.545512    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 01 20:49:06.545583 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.545532    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 01 20:49:06.545583 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.545544    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 01 20:49:06.545971 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.545786    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 01 20:49:06.545971 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.545817    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 01 20:49:06.549049 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.549021    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 01 20:49:06.551654 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.551638    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 01 20:49:06.561752 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.561734    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 20:49:06.673796 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.673758    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.673796 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.673801    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.674009 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.673851    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.674009 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.673909    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.674009 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.673944    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.674009 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.673968    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.674009 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.673991    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjlxs\" (UniqueName: \"kubernetes.io/projected/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-kube-api-access-tjlxs\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.674178 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.674015    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.674178 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.674039    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.674178 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.674086    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.674178 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.674106    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.674178 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.674125    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-web-config\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.674347 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.674189    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-config\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.674347 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.674217    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.674347 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.674241    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.674347 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.674267    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-config-out\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.674347 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.674291    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.674347 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.674314    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.674347 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.674340    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.674549 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.674359    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.757457 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.757431    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14ab9a24-1175-41f1-b736-93bc2ca01ae8" path="/var/lib/kubelet/pods/14ab9a24-1175-41f1-b736-93bc2ca01ae8/volumes"
Jul 01 20:49:06.774897 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.774838    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.774897 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.774881    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.775031 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.774902    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-config-out\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.775031 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.774918    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.775031 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.774937    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.775031 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.774957    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.775031 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.774983    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.775388 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.775360    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.775513 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.775496    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.775707 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.775688    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.775831 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.775805    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.775831 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.775716    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.775831 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.775546    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.775831 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.775879    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.775831 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.775844    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.775831 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.775908    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.775831 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.775934    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjlxs\" (UniqueName: \"kubernetes.io/projected/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-kube-api-access-tjlxs\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.775831 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.775989    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.775831 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.776020    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.775831 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.776054    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.775831 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.776080    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.775831 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.776102    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-web-config\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.775831 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.776148    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-config\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.775831 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.776420    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.775831 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.776488    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.775831 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.776528    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.778760 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.777968    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.778760 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.778279    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.778760 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.778324    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.778760 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.778446    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.778760 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.778666    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.778760 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.778672    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-config-out\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.779091 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.778875    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-config\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.779244 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.779221    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.780452 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.780417    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.780528 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.780480    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-web-config\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.780528 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.780505    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.780663 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.780648    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.780718 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.780702    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.784656 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.784639    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjlxs\" (UniqueName: \"kubernetes.io/projected/84d38e1e-02b7-4104-8f60-ca9b7fa89cfd-kube-api-access-tjlxs\") pod \"prometheus-k8s-0\" (UID: \"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.851083 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.851048    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:49:06.986242 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:06.986213    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 20:49:06.993458 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:49:06.993414    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d38e1e_02b7_4104_8f60_ca9b7fa89cfd.slice/crio-d85b704b4b6892cde84f9b983f9257fb7d0a728185f687bd6480aab968794721 WatchSource:0}: Error finding container d85b704b4b6892cde84f9b983f9257fb7d0a728185f687bd6480aab968794721: Status 404 returned error can't find the container with id d85b704b4b6892cde84f9b983f9257fb7d0a728185f687bd6480aab968794721
Jul 01 20:49:07.484175 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:07.484138    2564 generic.go:356] "Generic (PLEG): container finished" podID="84d38e1e-02b7-4104-8f60-ca9b7fa89cfd" containerID="a9503c06be437339f0995ee43da56ea326a4110eccd39fbc5be8a363c14154ae" exitCode=0
Jul 01 20:49:07.484617 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:07.484234    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd","Type":"ContainerDied","Data":"a9503c06be437339f0995ee43da56ea326a4110eccd39fbc5be8a363c14154ae"}
Jul 01 20:49:07.484617 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:07.484373    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd","Type":"ContainerStarted","Data":"d85b704b4b6892cde84f9b983f9257fb7d0a728185f687bd6480aab968794721"}
Jul 01 20:49:08.495512 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:08.495474    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd","Type":"ContainerStarted","Data":"b6e85b2f61277895a80259273cc3d1b4976a44b9b079383d503b3e9c96532bf9"}
Jul 01 20:49:08.495991 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:08.495552    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd","Type":"ContainerStarted","Data":"885c0dabf7d19eb3bb044aec139d620a8bcbfbcb5252561bae9f30da4e5633f9"}
Jul 01 20:49:08.495991 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:08.495598    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd","Type":"ContainerStarted","Data":"488acb8b948c9ec226d8bb9910c23f2c79c79048c79948b69eb3ae20b64cd346"}
Jul 01 20:49:08.495991 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:08.495639    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd","Type":"ContainerStarted","Data":"21be8db408dd8e145f37556b7a8bae56dd054caf01471fae823cdfeae4410b51"}
Jul 01 20:49:08.495991 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:08.495687    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd","Type":"ContainerStarted","Data":"d5b30c7cdd91776f9c94258a19d195a47d3e99e12f7800b82dd8f3ce64272e0c"}
Jul 01 20:49:08.495991 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:08.495722    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"84d38e1e-02b7-4104-8f60-ca9b7fa89cfd","Type":"ContainerStarted","Data":"9489a5288e9c331ec8108bb0352936e3fc8806910ac6f3539cb0766de8951e08"}
Jul 01 20:49:08.531065 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:08.528024    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.51169141 podStartE2EDuration="2.528001855s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.310445ms" isStatefulPod=false podCreationTimestamp="2026-07-01 20:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:49:08.525555149 +0000 UTC m=+148.201570750" watchObservedRunningTime="2026-07-01 20:49:08.528001855 +0000 UTC m=+148.204017425"
Jul 01 20:49:11.851958 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:49:11.851915    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:50:06.851804 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:50:06.851768    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:50:06.866698 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:50:06.866675    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:50:07.722122 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:50:07.722096    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 20:51:33.688387 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:51:33.688347    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-qrxw8"]
Jul 01 20:51:33.691449 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:51:33.691428    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-qrxw8"
Jul 01 20:51:33.694031 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:51:33.694012    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-d49r8\""
Jul 01 20:51:33.694326 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:51:33.694307    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 01 20:51:33.695376 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:51:33.695357    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 01 20:51:33.695442 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:51:33.695385    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 01 20:51:33.699335 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:51:33.699310    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-qrxw8"]
Jul 01 20:51:33.708285 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:51:33.708256    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxtl7\" (UniqueName: \"kubernetes.io/projected/5e570536-4933-4cf0-bbbb-9783c637914b-kube-api-access-fxtl7\") pod \"seaweedfs-64568bcd49-qrxw8\" (UID: \"5e570536-4933-4cf0-bbbb-9783c637914b\") " pod="kserve/seaweedfs-64568bcd49-qrxw8"
Jul 01 20:51:33.708402 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:51:33.708306    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5e570536-4933-4cf0-bbbb-9783c637914b-data\") pod \"seaweedfs-64568bcd49-qrxw8\" (UID: \"5e570536-4933-4cf0-bbbb-9783c637914b\") " pod="kserve/seaweedfs-64568bcd49-qrxw8"
Jul 01 20:51:33.808670 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:51:33.808631    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxtl7\" (UniqueName: \"kubernetes.io/projected/5e570536-4933-4cf0-bbbb-9783c637914b-kube-api-access-fxtl7\") pod \"seaweedfs-64568bcd49-qrxw8\" (UID: \"5e570536-4933-4cf0-bbbb-9783c637914b\") " pod="kserve/seaweedfs-64568bcd49-qrxw8"
Jul 01 20:51:33.808896 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:51:33.808691    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5e570536-4933-4cf0-bbbb-9783c637914b-data\") pod \"seaweedfs-64568bcd49-qrxw8\" (UID: \"5e570536-4933-4cf0-bbbb-9783c637914b\") " pod="kserve/seaweedfs-64568bcd49-qrxw8"
Jul 01 20:51:33.809036 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:51:33.809019    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5e570536-4933-4cf0-bbbb-9783c637914b-data\") pod \"seaweedfs-64568bcd49-qrxw8\" (UID: \"5e570536-4933-4cf0-bbbb-9783c637914b\") " pod="kserve/seaweedfs-64568bcd49-qrxw8"
Jul 01 20:51:33.817492 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:51:33.817472    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxtl7\" (UniqueName: \"kubernetes.io/projected/5e570536-4933-4cf0-bbbb-9783c637914b-kube-api-access-fxtl7\") pod \"seaweedfs-64568bcd49-qrxw8\" (UID: \"5e570536-4933-4cf0-bbbb-9783c637914b\") " pod="kserve/seaweedfs-64568bcd49-qrxw8"
Jul 01 20:51:34.001077 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:51:34.000982    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-qrxw8"
Jul 01 20:51:34.127633 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:51:34.127527    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-qrxw8"]
Jul 01 20:51:34.130336 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:51:34.130305    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e570536_4933_4cf0_bbbb_9783c637914b.slice/crio-8c21176f14a65341c2a20dfd53fefc32dc63056480ff5dae114c49035d3d04f3 WatchSource:0}: Error finding container 8c21176f14a65341c2a20dfd53fefc32dc63056480ff5dae114c49035d3d04f3: Status 404 returned error can't find the container with id 8c21176f14a65341c2a20dfd53fefc32dc63056480ff5dae114c49035d3d04f3
Jul 01 20:51:35.028089 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:51:35.028045    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-qrxw8" event={"ID":"5e570536-4933-4cf0-bbbb-9783c637914b","Type":"ContainerStarted","Data":"8c21176f14a65341c2a20dfd53fefc32dc63056480ff5dae114c49035d3d04f3"}
Jul 01 20:51:38.041713 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:51:38.041619    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-qrxw8" event={"ID":"5e570536-4933-4cf0-bbbb-9783c637914b","Type":"ContainerStarted","Data":"6a59a8ba7eb58fc01665b3e4b91847115731715d737e7a176a7692c046414949"}
Jul 01 20:51:38.042222 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:51:38.041744    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-qrxw8"
Jul 01 20:51:38.065930 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:51:38.065882    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-qrxw8" podStartSLOduration=1.559330467 podStartE2EDuration="5.065848644s" totalImagesPullingTime="3.506518177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:51:38.064994061 +0000 UTC m=+297.741009632" watchObservedRunningTime="2026-07-01 20:51:38.065848644 +0000 UTC m=+297.741864211"
Jul 01 20:51:40.701644 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:51:40.701615    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/ovn-acl-logging/0.log"
Jul 01 20:51:40.704761 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:51:40.704742    2564 kubelet.go:1648] "Image garbage collection succeeded"
Jul 01 20:51:40.707682 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:51:40.707663    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/ovn-acl-logging/0.log"
Jul 01 20:51:44.046942 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:51:44.046913    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-qrxw8"
Jul 01 20:53:01.131427 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:53:01.131391    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-xssr5"]
Jul 01 20:53:01.134778 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:53:01.134763    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-xssr5"
Jul 01 20:53:01.142381 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:53:01.142358    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-xssr5"]
Jul 01 20:53:01.197053 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:53:01.197025    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg5dk\" (UniqueName: \"kubernetes.io/projected/6a9b8c43-1627-4b2c-a4df-aab673fe4577-kube-api-access-sg5dk\") pod \"s3-init-xssr5\" (UID: \"6a9b8c43-1627-4b2c-a4df-aab673fe4577\") " pod="kserve/s3-init-xssr5"
Jul 01 20:53:01.197173 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:53:01.197067    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a9b8c43-1627-4b2c-a4df-aab673fe4577-model-cache\") pod \"s3-init-xssr5\" (UID: \"6a9b8c43-1627-4b2c-a4df-aab673fe4577\") " pod="kserve/s3-init-xssr5"
Jul 01 20:53:01.298301 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:53:01.298266    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg5dk\" (UniqueName: \"kubernetes.io/projected/6a9b8c43-1627-4b2c-a4df-aab673fe4577-kube-api-access-sg5dk\") pod \"s3-init-xssr5\" (UID: \"6a9b8c43-1627-4b2c-a4df-aab673fe4577\") " pod="kserve/s3-init-xssr5"
Jul 01 20:53:01.298474 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:53:01.298315    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a9b8c43-1627-4b2c-a4df-aab673fe4577-model-cache\") pod \"s3-init-xssr5\" (UID: \"6a9b8c43-1627-4b2c-a4df-aab673fe4577\") " pod="kserve/s3-init-xssr5"
Jul 01 20:53:01.298676 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:53:01.298658    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a9b8c43-1627-4b2c-a4df-aab673fe4577-model-cache\") pod \"s3-init-xssr5\" (UID: \"6a9b8c43-1627-4b2c-a4df-aab673fe4577\") " pod="kserve/s3-init-xssr5"
Jul 01 20:53:01.308062 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:53:01.308039    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg5dk\" (UniqueName: \"kubernetes.io/projected/6a9b8c43-1627-4b2c-a4df-aab673fe4577-kube-api-access-sg5dk\") pod \"s3-init-xssr5\" (UID: \"6a9b8c43-1627-4b2c-a4df-aab673fe4577\") " pod="kserve/s3-init-xssr5"
Jul 01 20:53:01.455923 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:53:01.455895    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-xssr5"
Jul 01 20:53:01.584146 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:53:01.584110    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-xssr5"]
Jul 01 20:53:01.587461 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:53:01.587433    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a9b8c43_1627_4b2c_a4df_aab673fe4577.slice/crio-58711bb87ad0eecba824aaeb57875e1d99e80a12fc57ff488731389cc88b9aaa WatchSource:0}: Error finding container 58711bb87ad0eecba824aaeb57875e1d99e80a12fc57ff488731389cc88b9aaa: Status 404 returned error can't find the container with id 58711bb87ad0eecba824aaeb57875e1d99e80a12fc57ff488731389cc88b9aaa
Jul 01 20:53:01.589217 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:53:01.589202    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 20:53:02.354990 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:53:02.354955    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-xssr5" event={"ID":"6a9b8c43-1627-4b2c-a4df-aab673fe4577","Type":"ContainerStarted","Data":"58711bb87ad0eecba824aaeb57875e1d99e80a12fc57ff488731389cc88b9aaa"}
Jul 01 20:53:06.374252 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:53:06.374220    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-xssr5" event={"ID":"6a9b8c43-1627-4b2c-a4df-aab673fe4577","Type":"ContainerStarted","Data":"3479023e9bd5ff0721c17e7f661197725bea9ce6e650c4d0f8342b7a5d63dd70"}
Jul 01 20:53:08.383548 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:53:08.383516    2564 generic.go:356] "Generic (PLEG): container finished" podID="6a9b8c43-1627-4b2c-a4df-aab673fe4577" containerID="3479023e9bd5ff0721c17e7f661197725bea9ce6e650c4d0f8342b7a5d63dd70" exitCode=0
Jul 01 20:53:08.384006 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:53:08.383553    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-xssr5" event={"ID":"6a9b8c43-1627-4b2c-a4df-aab673fe4577","Type":"ContainerDied","Data":"3479023e9bd5ff0721c17e7f661197725bea9ce6e650c4d0f8342b7a5d63dd70"}
Jul 01 20:53:13.409359 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:53:13.409324    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-xssr5" event={"ID":"6a9b8c43-1627-4b2c-a4df-aab673fe4577","Type":"ContainerStarted","Data":"8840ab998ec090d93c58218afa63cd316a4f2aff75ec9dd366371bfe47fcf1d9"}
Jul 01 20:53:13.428181 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:53:13.428135    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-xssr5" podStartSLOduration=2.076184551 podStartE2EDuration="12.428120678s" totalImagesPullingTime="8.654846853s" totalInitContainerRuntime="1.697089274s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:53:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:53:13.427520328 +0000 UTC m=+393.103535897" watchObservedRunningTime="2026-07-01 20:53:13.428120678 +0000 UTC m=+393.104136222"
Jul 01 20:54:18.650330 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:18.650297    2564 generic.go:356] "Generic (PLEG): container finished" podID="6a9b8c43-1627-4b2c-a4df-aab673fe4577" containerID="8840ab998ec090d93c58218afa63cd316a4f2aff75ec9dd366371bfe47fcf1d9" exitCode=0
Jul 01 20:54:18.650730 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:18.650365    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-xssr5" event={"ID":"6a9b8c43-1627-4b2c-a4df-aab673fe4577","Type":"ContainerDied","Data":"8840ab998ec090d93c58218afa63cd316a4f2aff75ec9dd366371bfe47fcf1d9"}
Jul 01 20:54:19.773772 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:19.773747    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-xssr5"
Jul 01 20:54:19.930143 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:19.930052    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sg5dk\" (UniqueName: \"kubernetes.io/projected/6a9b8c43-1627-4b2c-a4df-aab673fe4577-kube-api-access-sg5dk\") pod \"6a9b8c43-1627-4b2c-a4df-aab673fe4577\" (UID: \"6a9b8c43-1627-4b2c-a4df-aab673fe4577\") "
Jul 01 20:54:19.930143 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:19.930112    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a9b8c43-1627-4b2c-a4df-aab673fe4577-model-cache\") pod \"6a9b8c43-1627-4b2c-a4df-aab673fe4577\" (UID: \"6a9b8c43-1627-4b2c-a4df-aab673fe4577\") "
Jul 01 20:54:19.932218 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:19.932192    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a9b8c43-1627-4b2c-a4df-aab673fe4577-kube-api-access-sg5dk" (OuterVolumeSpecName: "kube-api-access-sg5dk") pod "6a9b8c43-1627-4b2c-a4df-aab673fe4577" (UID: "6a9b8c43-1627-4b2c-a4df-aab673fe4577"). InnerVolumeSpecName "kube-api-access-sg5dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:54:19.951841 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:19.951814    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a9b8c43-1627-4b2c-a4df-aab673fe4577-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a9b8c43-1627-4b2c-a4df-aab673fe4577" (UID: "6a9b8c43-1627-4b2c-a4df-aab673fe4577"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 20:54:20.031232 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:20.031193    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg5dk\" (UniqueName: \"kubernetes.io/projected/6a9b8c43-1627-4b2c-a4df-aab673fe4577-kube-api-access-sg5dk\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:54:20.031232 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:20.031224    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a9b8c43-1627-4b2c-a4df-aab673fe4577-model-cache\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:54:20.658697 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:20.658664    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-xssr5" event={"ID":"6a9b8c43-1627-4b2c-a4df-aab673fe4577","Type":"ContainerDied","Data":"58711bb87ad0eecba824aaeb57875e1d99e80a12fc57ff488731389cc88b9aaa"}
Jul 01 20:54:20.658697 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:20.658701    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58711bb87ad0eecba824aaeb57875e1d99e80a12fc57ff488731389cc88b9aaa"
Jul 01 20:54:20.658927 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:20.658699    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-xssr5"
Jul 01 20:54:32.140424 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.140392    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"]
Jul 01 20:54:32.140881 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.140695    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a9b8c43-1627-4b2c-a4df-aab673fe4577" containerName="download-hf-model"
Jul 01 20:54:32.140881 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.140705    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a9b8c43-1627-4b2c-a4df-aab673fe4577" containerName="download-hf-model"
Jul 01 20:54:32.140881 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.140714    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a9b8c43-1627-4b2c-a4df-aab673fe4577" containerName="s3-init"
Jul 01 20:54:32.140881 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.140719    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a9b8c43-1627-4b2c-a4df-aab673fe4577" containerName="s3-init"
Jul 01 20:54:32.140881 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.140777    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a9b8c43-1627-4b2c-a4df-aab673fe4577" containerName="s3-init"
Jul 01 20:54:32.142764 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.142750    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"
Jul 01 20:54:32.145680 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.145656    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
Jul 01 20:54:32.145973 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.145945    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
Jul 01 20:54:32.146083 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.145979    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 20:54:32.146083 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.146010    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vwmkn\""
Jul 01 20:54:32.146083 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.146034    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 20:54:32.146083 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.146077    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 20:54:32.147162 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.147146    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 20:54:32.152048 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.152026    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"]
Jul 01 20:54:32.317779 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.317740    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx"]
Jul 01 20:54:32.320189 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.320170    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx"
Jul 01 20:54:32.323008 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.322985    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-25c51-kube-rbac-proxy-sar-config\""
Jul 01 20:54:32.323008 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.323002    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-25c51-predictor-serving-cert\""
Jul 01 20:54:32.327493 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.327473    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2\" (UID: \"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"
Jul 01 20:54:32.327608 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.327499    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2\" (UID: \"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"
Jul 01 20:54:32.327608 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.327519    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdb9h\" (UniqueName: \"kubernetes.io/projected/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-kube-api-access-wdb9h\") pod \"isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2\" (UID: \"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"
Jul 01 20:54:32.327719 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.327654    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2\" (UID: \"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"
Jul 01 20:54:32.327719 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.327704    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2\" (UID: \"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"
Jul 01 20:54:32.328698 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.328680    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx"]
Jul 01 20:54:32.421470 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.421386    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc"]
Jul 01 20:54:32.423753 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.423731    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc"
Jul 01 20:54:32.426630 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.426611    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-25c51-kube-rbac-proxy-sar-config\""
Jul 01 20:54:32.426720 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.426637    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-25c51-predictor-serving-cert\""
Jul 01 20:54:32.429022 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.428999    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2\" (UID: \"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"
Jul 01 20:54:32.429181 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.429040    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db188407-051f-40e3-b2f0-8623d40d0429-proxy-tls\") pod \"success-200-isvc-25c51-predictor-54665ff85-mxfgx\" (UID: \"db188407-051f-40e3-b2f0-8623d40d0429\") " pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx"
Jul 01 20:54:32.429181 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.429143    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db188407-051f-40e3-b2f0-8623d40d0429-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-25c51-predictor-54665ff85-mxfgx\" (UID: \"db188407-051f-40e3-b2f0-8623d40d0429\") " pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx"
Jul 01 20:54:32.429303 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.429189    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2\" (UID: \"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"
Jul 01 20:54:32.429303 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.429258    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2\" (UID: \"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"
Jul 01 20:54:32.429303 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.429280    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2\" (UID: \"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"
Jul 01 20:54:32.429303 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.429299    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdb9h\" (UniqueName: \"kubernetes.io/projected/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-kube-api-access-wdb9h\") pod \"isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2\" (UID: \"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"
Jul 01 20:54:32.429511 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:54:32.429353    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-serving-cert: secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Jul 01 20:54:32.429511 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:54:32.429418    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-proxy-tls podName:603c3269-3a3d-4ff6-b52a-41bb9af0d5f4 nodeName:}" failed. No retries permitted until 2026-07-01 20:54:32.929399146 +0000 UTC m=+472.605414709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-proxy-tls") pod "isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2" (UID: "603c3269-3a3d-4ff6-b52a-41bb9af0d5f4") : secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Jul 01 20:54:32.429511 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.429440    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6tdg\" (UniqueName: \"kubernetes.io/projected/db188407-051f-40e3-b2f0-8623d40d0429-kube-api-access-v6tdg\") pod \"success-200-isvc-25c51-predictor-54665ff85-mxfgx\" (UID: \"db188407-051f-40e3-b2f0-8623d40d0429\") " pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx"
Jul 01 20:54:32.429685 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.429618    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2\" (UID: \"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"
Jul 01 20:54:32.429757 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.429736    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2\" (UID: \"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"
Jul 01 20:54:32.430225 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.430205    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2\" (UID: \"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"
Jul 01 20:54:32.434488 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.434469    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc"]
Jul 01 20:54:32.444645 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.444622    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdb9h\" (UniqueName: \"kubernetes.io/projected/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-kube-api-access-wdb9h\") pod \"isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2\" (UID: \"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"
Jul 01 20:54:32.525943 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.525918    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"]
Jul 01 20:54:32.528624 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.528607    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"
Jul 01 20:54:32.529834 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.529813    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db188407-051f-40e3-b2f0-8623d40d0429-proxy-tls\") pod \"success-200-isvc-25c51-predictor-54665ff85-mxfgx\" (UID: \"db188407-051f-40e3-b2f0-8623d40d0429\") " pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx"
Jul 01 20:54:32.529998 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.529842    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db188407-051f-40e3-b2f0-8623d40d0429-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-25c51-predictor-54665ff85-mxfgx\" (UID: \"db188407-051f-40e3-b2f0-8623d40d0429\") " pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx"
Jul 01 20:54:32.529998 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.529904    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxv9j\" (UniqueName: \"kubernetes.io/projected/1a485aa7-57db-42b5-98c1-9f180e964632-kube-api-access-fxv9j\") pod \"error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc\" (UID: \"1a485aa7-57db-42b5-98c1-9f180e964632\") " pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc"
Jul 01 20:54:32.529998 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.529926    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a485aa7-57db-42b5-98c1-9f180e964632-proxy-tls\") pod \"error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc\" (UID: \"1a485aa7-57db-42b5-98c1-9f180e964632\") " pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc"
Jul 01 20:54:32.529998 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.529952    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a485aa7-57db-42b5-98c1-9f180e964632-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc\" (UID: \"1a485aa7-57db-42b5-98c1-9f180e964632\") " pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc"
Jul 01 20:54:32.529998 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.529977    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6tdg\" (UniqueName: \"kubernetes.io/projected/db188407-051f-40e3-b2f0-8623d40d0429-kube-api-access-v6tdg\") pod \"success-200-isvc-25c51-predictor-54665ff85-mxfgx\" (UID: \"db188407-051f-40e3-b2f0-8623d40d0429\") " pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx"
Jul 01 20:54:32.530532 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.530514    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db188407-051f-40e3-b2f0-8623d40d0429-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-25c51-predictor-54665ff85-mxfgx\" (UID: \"db188407-051f-40e3-b2f0-8623d40d0429\") " pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx"
Jul 01 20:54:32.531564 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.531545    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
Jul 01 20:54:32.531663 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.531605    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
Jul 01 20:54:32.532249 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.532234    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db188407-051f-40e3-b2f0-8623d40d0429-proxy-tls\") pod \"success-200-isvc-25c51-predictor-54665ff85-mxfgx\" (UID: \"db188407-051f-40e3-b2f0-8623d40d0429\") " pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx"
Jul 01 20:54:32.538440 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.538415    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6tdg\" (UniqueName: \"kubernetes.io/projected/db188407-051f-40e3-b2f0-8623d40d0429-kube-api-access-v6tdg\") pod \"success-200-isvc-25c51-predictor-54665ff85-mxfgx\" (UID: \"db188407-051f-40e3-b2f0-8623d40d0429\") " pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx"
Jul 01 20:54:32.540497 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.540469    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"]
Jul 01 20:54:32.627667 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.627631    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"]
Jul 01 20:54:32.630142 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.630123    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"
Jul 01 20:54:32.630271 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.630145    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx"
Jul 01 20:54:32.631068 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.631047    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-9b7pg\" (UID: \"140bfc0f-a451-4a3b-9acc-1fdcde1702e5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"
Jul 01 20:54:32.631189 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.631094    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e28fcffa-cde8-44ce-9015-a1ad49645e7a-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm\" (UID: \"e28fcffa-cde8-44ce-9015-a1ad49645e7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"
Jul 01 20:54:32.631189 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.631117    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e28fcffa-cde8-44ce-9015-a1ad49645e7a-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm\" (UID: \"e28fcffa-cde8-44ce-9015-a1ad49645e7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"
Jul 01 20:54:32.631308 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.631186    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxv9j\" (UniqueName: \"kubernetes.io/projected/1a485aa7-57db-42b5-98c1-9f180e964632-kube-api-access-fxv9j\") pod \"error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc\" (UID: \"1a485aa7-57db-42b5-98c1-9f180e964632\") " pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc"
Jul 01 20:54:32.631308 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.631223    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e28fcffa-cde8-44ce-9015-a1ad49645e7a-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm\" (UID: \"e28fcffa-cde8-44ce-9015-a1ad49645e7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"
Jul 01 20:54:32.631308 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.631260    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-9b7pg\" (UID: \"140bfc0f-a451-4a3b-9acc-1fdcde1702e5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"
Jul 01 20:54:32.631308 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.631290    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a485aa7-57db-42b5-98c1-9f180e964632-proxy-tls\") pod \"error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc\" (UID: \"1a485aa7-57db-42b5-98c1-9f180e964632\") " pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc"
Jul 01 20:54:32.631507 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.631318    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-9b7pg\" (UID: \"140bfc0f-a451-4a3b-9acc-1fdcde1702e5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"
Jul 01 20:54:32.631507 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.631360    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a485aa7-57db-42b5-98c1-9f180e964632-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc\" (UID: \"1a485aa7-57db-42b5-98c1-9f180e964632\") " pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc"
Jul 01 20:54:32.631507 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.631387    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-9b7pg\" (UID: \"140bfc0f-a451-4a3b-9acc-1fdcde1702e5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"
Jul 01 20:54:32.631507 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.631411    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mfc2\" (UniqueName: \"kubernetes.io/projected/e28fcffa-cde8-44ce-9015-a1ad49645e7a-kube-api-access-8mfc2\") pod \"isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm\" (UID: \"e28fcffa-cde8-44ce-9015-a1ad49645e7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"
Jul 01 20:54:32.631507 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.631450    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h74tf\" (UniqueName: \"kubernetes.io/projected/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-kube-api-access-h74tf\") pod \"isvc-xgboost-graph-predictor-555fc69d56-9b7pg\" (UID: \"140bfc0f-a451-4a3b-9acc-1fdcde1702e5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"
Jul 01 20:54:32.631752 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.631530    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e28fcffa-cde8-44ce-9015-a1ad49645e7a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm\" (UID: \"e28fcffa-cde8-44ce-9015-a1ad49645e7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"
Jul 01 20:54:32.632235 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.632047    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a485aa7-57db-42b5-98c1-9f180e964632-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc\" (UID: \"1a485aa7-57db-42b5-98c1-9f180e964632\") " pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc"
Jul 01 20:54:32.632973 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.632954    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Jul 01 20:54:32.633109 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.633024    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Jul 01 20:54:32.634064 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.634040    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a485aa7-57db-42b5-98c1-9f180e964632-proxy-tls\") pod \"error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc\" (UID: \"1a485aa7-57db-42b5-98c1-9f180e964632\") " pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc"
Jul 01 20:54:32.640095 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.639682    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxv9j\" (UniqueName: \"kubernetes.io/projected/1a485aa7-57db-42b5-98c1-9f180e964632-kube-api-access-fxv9j\") pod \"error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc\" (UID: \"1a485aa7-57db-42b5-98c1-9f180e964632\") " pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc"
Jul 01 20:54:32.641236 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.641214    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"]
Jul 01 20:54:32.732669 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.732634    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e28fcffa-cde8-44ce-9015-a1ad49645e7a-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm\" (UID: \"e28fcffa-cde8-44ce-9015-a1ad49645e7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"
Jul 01 20:54:32.732823 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.732688    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-9b7pg\" (UID: \"140bfc0f-a451-4a3b-9acc-1fdcde1702e5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"
Jul 01 20:54:32.732823 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.732719    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-9b7pg\" (UID: \"140bfc0f-a451-4a3b-9acc-1fdcde1702e5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"
Jul 01 20:54:32.732823 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.732766    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-9b7pg\" (UID: \"140bfc0f-a451-4a3b-9acc-1fdcde1702e5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"
Jul 01 20:54:32.732823 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.732810    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mfc2\" (UniqueName: \"kubernetes.io/projected/e28fcffa-cde8-44ce-9015-a1ad49645e7a-kube-api-access-8mfc2\") pod \"isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm\" (UID: \"e28fcffa-cde8-44ce-9015-a1ad49645e7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"
Jul 01 20:54:32.733107 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.732854    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h74tf\" (UniqueName: \"kubernetes.io/projected/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-kube-api-access-h74tf\") pod \"isvc-xgboost-graph-predictor-555fc69d56-9b7pg\" (UID: \"140bfc0f-a451-4a3b-9acc-1fdcde1702e5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"
Jul 01 20:54:32.733107 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.732930    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e28fcffa-cde8-44ce-9015-a1ad49645e7a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm\" (UID: \"e28fcffa-cde8-44ce-9015-a1ad49645e7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"
Jul 01 20:54:32.733107 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.732968    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-9b7pg\" (UID: \"140bfc0f-a451-4a3b-9acc-1fdcde1702e5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"
Jul 01 20:54:32.733107 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.733019    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e28fcffa-cde8-44ce-9015-a1ad49645e7a-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm\" (UID: \"e28fcffa-cde8-44ce-9015-a1ad49645e7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"
Jul 01 20:54:32.733107 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.733054    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e28fcffa-cde8-44ce-9015-a1ad49645e7a-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm\" (UID: \"e28fcffa-cde8-44ce-9015-a1ad49645e7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"
Jul 01 20:54:32.733368 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.733196    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e28fcffa-cde8-44ce-9015-a1ad49645e7a-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm\" (UID: \"e28fcffa-cde8-44ce-9015-a1ad49645e7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"
Jul 01 20:54:32.733638 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:54:32.733499    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-predictor-serving-cert: secret "isvc-xgboost-graph-predictor-serving-cert" not found
Jul 01 20:54:32.733638 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.733514    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-9b7pg\" (UID: \"140bfc0f-a451-4a3b-9acc-1fdcde1702e5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"
Jul 01 20:54:32.733638 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.733553    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-9b7pg\" (UID: \"140bfc0f-a451-4a3b-9acc-1fdcde1702e5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"
Jul 01 20:54:32.733638 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:54:32.733578    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-proxy-tls podName:140bfc0f-a451-4a3b-9acc-1fdcde1702e5 nodeName:}" failed. No retries permitted until 2026-07-01 20:54:33.233549976 +0000 UTC m=+472.909565536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-proxy-tls") pod "isvc-xgboost-graph-predictor-555fc69d56-9b7pg" (UID: "140bfc0f-a451-4a3b-9acc-1fdcde1702e5") : secret "isvc-xgboost-graph-predictor-serving-cert" not found
Jul 01 20:54:32.734130 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.733587    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc"
Jul 01 20:54:32.734240 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.734183    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-9b7pg\" (UID: \"140bfc0f-a451-4a3b-9acc-1fdcde1702e5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"
Jul 01 20:54:32.734240 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.734010    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e28fcffa-cde8-44ce-9015-a1ad49645e7a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm\" (UID: \"e28fcffa-cde8-44ce-9015-a1ad49645e7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"
Jul 01 20:54:32.734240 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.733852    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e28fcffa-cde8-44ce-9015-a1ad49645e7a-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm\" (UID: \"e28fcffa-cde8-44ce-9015-a1ad49645e7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"
Jul 01 20:54:32.736059 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.736039    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e28fcffa-cde8-44ce-9015-a1ad49645e7a-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm\" (UID: \"e28fcffa-cde8-44ce-9015-a1ad49645e7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"
Jul 01 20:54:32.740732 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.740713    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mfc2\" (UniqueName: \"kubernetes.io/projected/e28fcffa-cde8-44ce-9015-a1ad49645e7a-kube-api-access-8mfc2\") pod \"isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm\" (UID: \"e28fcffa-cde8-44ce-9015-a1ad49645e7a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"
Jul 01 20:54:32.744252 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.744233    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h74tf\" (UniqueName: \"kubernetes.io/projected/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-kube-api-access-h74tf\") pod \"isvc-xgboost-graph-predictor-555fc69d56-9b7pg\" (UID: \"140bfc0f-a451-4a3b-9acc-1fdcde1702e5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"
Jul 01 20:54:32.766488 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.766458    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx"]
Jul 01 20:54:32.769322 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:54:32.769296    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb188407_051f_40e3_b2f0_8623d40d0429.slice/crio-bb277e90424d0dd913d2d7e876abe5ee8a2d05d8d0f046f35ba1822396219108 WatchSource:0}: Error finding container bb277e90424d0dd913d2d7e876abe5ee8a2d05d8d0f046f35ba1822396219108: Status 404 returned error can't find the container with id bb277e90424d0dd913d2d7e876abe5ee8a2d05d8d0f046f35ba1822396219108
Jul 01 20:54:32.859695 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.859673    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc"]
Jul 01 20:54:32.861990 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:54:32.861971    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a485aa7_57db_42b5_98c1_9f180e964632.slice/crio-7ff5914f9e75355158c1fb49a29d7468ed900edbee4014ba6b0ee87b44015e9e WatchSource:0}: Error finding container 7ff5914f9e75355158c1fb49a29d7468ed900edbee4014ba6b0ee87b44015e9e: Status 404 returned error can't find the container with id 7ff5914f9e75355158c1fb49a29d7468ed900edbee4014ba6b0ee87b44015e9e
Jul 01 20:54:32.935247 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.935180    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2\" (UID: \"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"
Jul 01 20:54:32.937469 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.937443    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2\" (UID: \"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"
Jul 01 20:54:32.957338 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:32.957315    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"
Jul 01 20:54:33.053305 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:33.053278    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"
Jul 01 20:54:33.091375 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:33.087595    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"]
Jul 01 20:54:33.192631 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:33.192553    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"]
Jul 01 20:54:33.196246 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:54:33.196216    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod603c3269_3a3d_4ff6_b52a_41bb9af0d5f4.slice/crio-6c34df51820264e97895613603e004d48607a9f28a523b48c174544e79a83a5e WatchSource:0}: Error finding container 6c34df51820264e97895613603e004d48607a9f28a523b48c174544e79a83a5e: Status 404 returned error can't find the container with id 6c34df51820264e97895613603e004d48607a9f28a523b48c174544e79a83a5e
Jul 01 20:54:33.237193 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:33.237169    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-9b7pg\" (UID: \"140bfc0f-a451-4a3b-9acc-1fdcde1702e5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"
Jul 01 20:54:33.239342 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:33.239323    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-9b7pg\" (UID: \"140bfc0f-a451-4a3b-9acc-1fdcde1702e5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"
Jul 01 20:54:33.451656 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:33.451572    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"
Jul 01 20:54:33.684129 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:33.684026    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"]
Jul 01 20:54:33.708704 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:54:33.708612    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod140bfc0f_a451_4a3b_9acc_1fdcde1702e5.slice/crio-ce572fac15a38951dbadd65ea7366387a70290c305f384165f46bd2fae8b9b95 WatchSource:0}: Error finding container ce572fac15a38951dbadd65ea7366387a70290c305f384165f46bd2fae8b9b95: Status 404 returned error can't find the container with id ce572fac15a38951dbadd65ea7366387a70290c305f384165f46bd2fae8b9b95
Jul 01 20:54:33.733303 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:33.732460    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm" event={"ID":"e28fcffa-cde8-44ce-9015-a1ad49645e7a","Type":"ContainerStarted","Data":"b190adb7ee23a3a02162490d6c1c7661f30eac8e67423cd3005b7734a35efbc4"}
Jul 01 20:54:33.733303 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:33.732536    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm" event={"ID":"e28fcffa-cde8-44ce-9015-a1ad49645e7a","Type":"ContainerStarted","Data":"2cf51427f9dd4e968caacec129e92cce60d7c896487aa3cf5250f116306174ff"}
Jul 01 20:54:33.763825 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:33.763790    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc" event={"ID":"1a485aa7-57db-42b5-98c1-9f180e964632","Type":"ContainerStarted","Data":"7ff5914f9e75355158c1fb49a29d7468ed900edbee4014ba6b0ee87b44015e9e"}
Jul 01 20:54:33.784640 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:33.783160    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2" event={"ID":"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4","Type":"ContainerStarted","Data":"9fde6b9870a9cef77f324602ad181211a171b23de25631cb9bbcc90b9ed43883"}
Jul 01 20:54:33.784640 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:33.783236    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2" event={"ID":"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4","Type":"ContainerStarted","Data":"6c34df51820264e97895613603e004d48607a9f28a523b48c174544e79a83a5e"}
Jul 01 20:54:33.792603 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:33.792485    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx" event={"ID":"db188407-051f-40e3-b2f0-8623d40d0429","Type":"ContainerStarted","Data":"bb277e90424d0dd913d2d7e876abe5ee8a2d05d8d0f046f35ba1822396219108"}
Jul 01 20:54:34.809815 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:34.809664    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg" event={"ID":"140bfc0f-a451-4a3b-9acc-1fdcde1702e5","Type":"ContainerStarted","Data":"f8c03d52d5370081448c1533839f24b1bea589a1f199d69e2030f85d5af3e0eb"}
Jul 01 20:54:34.809815 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:34.809745    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg" event={"ID":"140bfc0f-a451-4a3b-9acc-1fdcde1702e5","Type":"ContainerStarted","Data":"ce572fac15a38951dbadd65ea7366387a70290c305f384165f46bd2fae8b9b95"}
Jul 01 20:54:37.834935 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:37.834895    2564 generic.go:356] "Generic (PLEG): container finished" podID="603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" containerID="9fde6b9870a9cef77f324602ad181211a171b23de25631cb9bbcc90b9ed43883" exitCode=0
Jul 01 20:54:37.835446 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:37.834994    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2" event={"ID":"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4","Type":"ContainerDied","Data":"9fde6b9870a9cef77f324602ad181211a171b23de25631cb9bbcc90b9ed43883"}
Jul 01 20:54:37.839387 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:37.839314    2564 generic.go:356] "Generic (PLEG): container finished" podID="e28fcffa-cde8-44ce-9015-a1ad49645e7a" containerID="b190adb7ee23a3a02162490d6c1c7661f30eac8e67423cd3005b7734a35efbc4" exitCode=0
Jul 01 20:54:37.839497 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:37.839380    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm" event={"ID":"e28fcffa-cde8-44ce-9015-a1ad49645e7a","Type":"ContainerDied","Data":"b190adb7ee23a3a02162490d6c1c7661f30eac8e67423cd3005b7734a35efbc4"}
Jul 01 20:54:39.857841 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:39.857724    2564 generic.go:356] "Generic (PLEG): container finished" podID="140bfc0f-a451-4a3b-9acc-1fdcde1702e5" containerID="f8c03d52d5370081448c1533839f24b1bea589a1f199d69e2030f85d5af3e0eb" exitCode=0
Jul 01 20:54:39.857841 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:54:39.857797    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg" event={"ID":"140bfc0f-a451-4a3b-9acc-1fdcde1702e5","Type":"ContainerDied","Data":"f8c03d52d5370081448c1533839f24b1bea589a1f199d69e2030f85d5af3e0eb"}
Jul 01 20:55:10.044588 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:10.044547    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2" event={"ID":"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4","Type":"ContainerStarted","Data":"3d401e3428a1ce8e8356bed74b80db9385676c7c9aacb672aa31984e20cdecda"}
Jul 01 20:55:10.046540 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:10.046508    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx" event={"ID":"db188407-051f-40e3-b2f0-8623d40d0429","Type":"ContainerStarted","Data":"027f1a3ef0398ae7cbe0bd82122497fc2f80d55bfe8efe3bb6093e1debc772a0"}
Jul 01 20:55:10.048786 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:10.048716    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm" event={"ID":"e28fcffa-cde8-44ce-9015-a1ad49645e7a","Type":"ContainerStarted","Data":"59a868890366f9236fbef38b9be005b46dac9a8616a5633b647c66b6a86a31b2"}
Jul 01 20:55:10.051613 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:10.051542    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc" event={"ID":"1a485aa7-57db-42b5-98c1-9f180e964632","Type":"ContainerStarted","Data":"d3804b51c282f68b1878bcb247fada75dd25c6997668a6c7418b40a1ed278bae"}
Jul 01 20:55:10.055011 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:10.054950    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg" event={"ID":"140bfc0f-a451-4a3b-9acc-1fdcde1702e5","Type":"ContainerStarted","Data":"cde4850a0cd164f76179a39ed26154436a08968c098eecc6f63e57ee6103b4d2"}
Jul 01 20:55:14.080465 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:14.080358    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg" event={"ID":"140bfc0f-a451-4a3b-9acc-1fdcde1702e5","Type":"ContainerStarted","Data":"ee1d416942708d927bebf43875c2df20cb412c5d4418da32d8170e60c2e73187"}
Jul 01 20:55:14.080980 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:14.080484    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"
Jul 01 20:55:14.082457 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:14.082435    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2" event={"ID":"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4","Type":"ContainerStarted","Data":"2a2725f6ed0e1d310b56c79cee0dade4d1ea369da43b45f80addda04ea41b2a5"}
Jul 01 20:55:14.082585 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:14.082565    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"
Jul 01 20:55:14.084118 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:14.084085    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx" event={"ID":"db188407-051f-40e3-b2f0-8623d40d0429","Type":"ContainerStarted","Data":"798e83bcf15b2b1b7ce0909655d7352f11a15464cd0c84de2d06188f29e6ec3b"}
Jul 01 20:55:14.084238 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:14.084219    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx"
Jul 01 20:55:14.086051 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:14.086027    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm" event={"ID":"e28fcffa-cde8-44ce-9015-a1ad49645e7a","Type":"ContainerStarted","Data":"c87f86307ab796673411960278297376de8ceebc5ec5e55043e4d231bf36ae2d"}
Jul 01 20:55:14.086229 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:14.086211    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"
Jul 01 20:55:14.086291 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:14.086263    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"
Jul 01 20:55:14.087758 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:14.087715    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm" podUID="e28fcffa-cde8-44ce-9015-a1ad49645e7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 01 20:55:14.087758 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:14.087754    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc" event={"ID":"1a485aa7-57db-42b5-98c1-9f180e964632","Type":"ContainerStarted","Data":"1385805bf23c8cbc06ea4843190e3280f53d3ef14ede7cd81bde8f4199bb4e0f"}
Jul 01 20:55:14.087923 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:14.087894    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc"
Jul 01 20:55:14.103660 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:14.103589    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg" podStartSLOduration=4.593561857 podStartE2EDuration="42.103569111s" totalImagesPullingTime="33.673042623s" totalInitContainerRuntime="3.836964631s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:54:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:55:14.099460944 +0000 UTC m=+513.775476524" watchObservedRunningTime="2026-07-01 20:55:14.103569111 +0000 UTC m=+513.779584676"
Jul 01 20:55:14.119961 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:14.119915    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx" podStartSLOduration=1.187234318 podStartE2EDuration="42.119899609s" totalImagesPullingTime="40.932665291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:54:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:55:14.119145598 +0000 UTC m=+513.795161177" watchObservedRunningTime="2026-07-01 20:55:14.119899609 +0000 UTC m=+513.795915154"
Jul 01 20:55:14.137998 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:14.137944    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc" podStartSLOduration=1.481591672 podStartE2EDuration="42.137927554s" totalImagesPullingTime="40.656335882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:54:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:55:14.136448344 +0000 UTC m=+513.812463913" watchObservedRunningTime="2026-07-01 20:55:14.137927554 +0000 UTC m=+513.813943121"
Jul 01 20:55:14.165158 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:14.165108    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2" podStartSLOduration=3.049210518 podStartE2EDuration="42.16509075s" totalImagesPullingTime="35.293029285s" totalInitContainerRuntime="3.822850947s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:54:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:55:14.163541007 +0000 UTC m=+513.839556601" watchObservedRunningTime="2026-07-01 20:55:14.16509075 +0000 UTC m=+513.841106318"
Jul 01 20:55:14.203787 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:14.203726    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm" podStartSLOduration=2.4416169180000002 podStartE2EDuration="42.203703918s" totalImagesPullingTime="35.760455855s" totalInitContainerRuntime="4.001631145s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:54:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:55:14.201631069 +0000 UTC m=+513.877646636" watchObservedRunningTime="2026-07-01 20:55:14.203703918 +0000 UTC m=+513.879719486"
Jul 01 20:55:15.092813 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:15.092779    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx"
Jul 01 20:55:15.093286 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:15.093149    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"
Jul 01 20:55:15.093286 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:15.093197    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"
Jul 01 20:55:15.093661 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:15.093628    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm" podUID="e28fcffa-cde8-44ce-9015-a1ad49645e7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 01 20:55:15.093874 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:15.093840    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc"
Jul 01 20:55:15.094367 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:15.094338    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx" podUID="db188407-051f-40e3-b2f0-8623d40d0429" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 01 20:55:15.094438 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:15.094349    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg" podUID="140bfc0f-a451-4a3b-9acc-1fdcde1702e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 01 20:55:15.094515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:15.094497    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2" podUID="603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 01 20:55:15.094755 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:15.094740    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc" podUID="1a485aa7-57db-42b5-98c1-9f180e964632" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 01 20:55:16.098203 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:16.098165    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg" podUID="140bfc0f-a451-4a3b-9acc-1fdcde1702e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 01 20:55:16.098829 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:16.098165    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2" podUID="603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 01 20:55:16.098949 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:16.098213    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx" podUID="db188407-051f-40e3-b2f0-8623d40d0429" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 01 20:55:16.099334 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:16.099307    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc" podUID="1a485aa7-57db-42b5-98c1-9f180e964632" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 01 20:55:20.098644 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:20.098611    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"
Jul 01 20:55:20.099569 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:20.099533    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm" podUID="e28fcffa-cde8-44ce-9015-a1ad49645e7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 01 20:55:21.105622 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:21.105458    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx"
Jul 01 20:55:21.106241 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:21.106220    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc"
Jul 01 20:55:21.106388 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:21.106354    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx" podUID="db188407-051f-40e3-b2f0-8623d40d0429" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 01 20:55:21.106972 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:21.106929    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc" podUID="1a485aa7-57db-42b5-98c1-9f180e964632" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 01 20:55:21.107873 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:21.107837    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"
Jul 01 20:55:21.108428 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:21.108383    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2" podUID="603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 01 20:55:21.108888 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:21.108872    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"
Jul 01 20:55:21.109296 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:21.109270    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg" podUID="140bfc0f-a451-4a3b-9acc-1fdcde1702e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 01 20:55:30.100246 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:30.100201    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm" podUID="e28fcffa-cde8-44ce-9015-a1ad49645e7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 01 20:55:31.106845 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:31.106783    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx" podUID="db188407-051f-40e3-b2f0-8623d40d0429" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 01 20:55:31.107315 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:31.106879    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc" podUID="1a485aa7-57db-42b5-98c1-9f180e964632" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 01 20:55:31.108346 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:31.108323    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2" podUID="603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 01 20:55:31.109564 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:31.109546    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg" podUID="140bfc0f-a451-4a3b-9acc-1fdcde1702e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 01 20:55:40.100122 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:40.100012    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm" podUID="e28fcffa-cde8-44ce-9015-a1ad49645e7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 01 20:55:41.106140 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:41.106098    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx" podUID="db188407-051f-40e3-b2f0-8623d40d0429" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 01 20:55:41.106899 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:41.106837    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc" podUID="1a485aa7-57db-42b5-98c1-9f180e964632" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 01 20:55:41.109119 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:41.109096    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2" podUID="603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 01 20:55:41.109275 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:41.109255    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg" podUID="140bfc0f-a451-4a3b-9acc-1fdcde1702e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 01 20:55:50.099968 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:50.099924    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm" podUID="e28fcffa-cde8-44ce-9015-a1ad49645e7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 01 20:55:51.106562 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:51.106516    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx" podUID="db188407-051f-40e3-b2f0-8623d40d0429" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 01 20:55:51.107001 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:51.106898    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc" podUID="1a485aa7-57db-42b5-98c1-9f180e964632" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 01 20:55:51.109118 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:51.109098    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2" podUID="603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 01 20:55:51.109235 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:55:51.109216    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg" podUID="140bfc0f-a451-4a3b-9acc-1fdcde1702e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 01 20:56:00.099739 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:00.099693    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm" podUID="e28fcffa-cde8-44ce-9015-a1ad49645e7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 01 20:56:01.106713 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:01.106654    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx" podUID="db188407-051f-40e3-b2f0-8623d40d0429" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 01 20:56:01.107479 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:01.107459    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc"
Jul 01 20:56:01.108282 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:01.108260    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2" podUID="603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 01 20:56:01.109478 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:01.109449    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg" podUID="140bfc0f-a451-4a3b-9acc-1fdcde1702e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 01 20:56:10.100081 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:10.100040    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm" podUID="e28fcffa-cde8-44ce-9015-a1ad49645e7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 01 20:56:11.107660 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:11.107634    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx"
Jul 01 20:56:11.108368 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:11.108346    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2" podUID="603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 01 20:56:11.109559 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:11.109537    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg" podUID="140bfc0f-a451-4a3b-9acc-1fdcde1702e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 01 20:56:20.100171 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:20.100129    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm" podUID="e28fcffa-cde8-44ce-9015-a1ad49645e7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 01 20:56:21.109258 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:21.109220    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2" podUID="603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 01 20:56:21.109793 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:21.109770    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"
Jul 01 20:56:30.100083 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:30.100043    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm" podUID="e28fcffa-cde8-44ce-9015-a1ad49645e7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 01 20:56:31.108971 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:31.108940    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"
Jul 01 20:56:32.369061 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.369026    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4"]
Jul 01 20:56:32.372207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.372184    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4"
Jul 01 20:56:32.376036 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.376007    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f347e-kube-rbac-proxy-sar-config\""
Jul 01 20:56:32.376164 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.376007    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f347e-predictor-serving-cert\""
Jul 01 20:56:32.385313 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.385289    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4"]
Jul 01 20:56:32.393050 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.393014    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx"]
Jul 01 20:56:32.393508 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.393270    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx" podUID="db188407-051f-40e3-b2f0-8623d40d0429" containerName="kserve-container" containerID="cri-o://027f1a3ef0398ae7cbe0bd82122497fc2f80d55bfe8efe3bb6093e1debc772a0" gracePeriod=30
Jul 01 20:56:32.393508 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.393454    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx" podUID="db188407-051f-40e3-b2f0-8623d40d0429" containerName="kube-rbac-proxy" containerID="cri-o://798e83bcf15b2b1b7ce0909655d7352f11a15464cd0c84de2d06188f29e6ec3b" gracePeriod=30
Jul 01 20:56:32.439926 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.439891    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7"]
Jul 01 20:56:32.442575 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.442558    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7"
Jul 01 20:56:32.445288 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.445273    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f347e-kube-rbac-proxy-sar-config\""
Jul 01 20:56:32.445494 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.445481    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f347e-predictor-serving-cert\""
Jul 01 20:56:32.452781 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.452761    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7"]
Jul 01 20:56:32.489850 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.489822    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc"]
Jul 01 20:56:32.490115 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.490086    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc" podUID="1a485aa7-57db-42b5-98c1-9f180e964632" containerName="kserve-container" containerID="cri-o://d3804b51c282f68b1878bcb247fada75dd25c6997668a6c7418b40a1ed278bae" gracePeriod=30
Jul 01 20:56:32.490240 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.490133    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc" podUID="1a485aa7-57db-42b5-98c1-9f180e964632" containerName="kube-rbac-proxy" containerID="cri-o://1385805bf23c8cbc06ea4843190e3280f53d3ef14ede7cd81bde8f4199bb4e0f" gracePeriod=30
Jul 01 20:56:32.537067 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.537035    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72b059bc-a318-419f-b477-f0d1b982dcac-proxy-tls\") pod \"error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7\" (UID: \"72b059bc-a318-419f-b477-f0d1b982dcac\") " pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7"
Jul 01 20:56:32.537181 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.537071    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/447b6d3c-1be1-4053-91db-e8f11d74445f-proxy-tls\") pod \"success-200-isvc-f347e-predictor-9c7897d59-775b4\" (UID: \"447b6d3c-1be1-4053-91db-e8f11d74445f\") " pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4"
Jul 01 20:56:32.537181 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.537092    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6b95\" (UniqueName: \"kubernetes.io/projected/447b6d3c-1be1-4053-91db-e8f11d74445f-kube-api-access-h6b95\") pod \"success-200-isvc-f347e-predictor-9c7897d59-775b4\" (UID: \"447b6d3c-1be1-4053-91db-e8f11d74445f\") " pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4"
Jul 01 20:56:32.537277 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.537199    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8c7v\" (UniqueName: \"kubernetes.io/projected/72b059bc-a318-419f-b477-f0d1b982dcac-kube-api-access-j8c7v\") pod \"error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7\" (UID: \"72b059bc-a318-419f-b477-f0d1b982dcac\") " pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7"
Jul 01 20:56:32.537277 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.537250    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72b059bc-a318-419f-b477-f0d1b982dcac-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7\" (UID: \"72b059bc-a318-419f-b477-f0d1b982dcac\") " pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7"
Jul 01 20:56:32.537373 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.537337    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/447b6d3c-1be1-4053-91db-e8f11d74445f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f347e-predictor-9c7897d59-775b4\" (UID: \"447b6d3c-1be1-4053-91db-e8f11d74445f\") " pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4"
Jul 01 20:56:32.638312 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.638291    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8c7v\" (UniqueName: \"kubernetes.io/projected/72b059bc-a318-419f-b477-f0d1b982dcac-kube-api-access-j8c7v\") pod \"error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7\" (UID: \"72b059bc-a318-419f-b477-f0d1b982dcac\") " pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7"
Jul 01 20:56:32.638438 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.638332    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72b059bc-a318-419f-b477-f0d1b982dcac-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7\" (UID: \"72b059bc-a318-419f-b477-f0d1b982dcac\") " pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7"
Jul 01 20:56:32.638438 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.638360    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/447b6d3c-1be1-4053-91db-e8f11d74445f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f347e-predictor-9c7897d59-775b4\" (UID: \"447b6d3c-1be1-4053-91db-e8f11d74445f\") " pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4"
Jul 01 20:56:32.638438 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.638409    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72b059bc-a318-419f-b477-f0d1b982dcac-proxy-tls\") pod \"error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7\" (UID: \"72b059bc-a318-419f-b477-f0d1b982dcac\") " pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7"
Jul 01 20:56:32.638438 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.638427    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/447b6d3c-1be1-4053-91db-e8f11d74445f-proxy-tls\") pod \"success-200-isvc-f347e-predictor-9c7897d59-775b4\" (UID: \"447b6d3c-1be1-4053-91db-e8f11d74445f\") " pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4"
Jul 01 20:56:32.638745 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.638446    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6b95\" (UniqueName: \"kubernetes.io/projected/447b6d3c-1be1-4053-91db-e8f11d74445f-kube-api-access-h6b95\") pod \"success-200-isvc-f347e-predictor-9c7897d59-775b4\" (UID: \"447b6d3c-1be1-4053-91db-e8f11d74445f\") " pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4"
Jul 01 20:56:32.639078 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.639044    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72b059bc-a318-419f-b477-f0d1b982dcac-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7\" (UID: \"72b059bc-a318-419f-b477-f0d1b982dcac\") " pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7"
Jul 01 20:56:32.639360 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.639343    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/447b6d3c-1be1-4053-91db-e8f11d74445f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f347e-predictor-9c7897d59-775b4\" (UID: \"447b6d3c-1be1-4053-91db-e8f11d74445f\") " pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4"
Jul 01 20:56:32.641278 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.641259    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72b059bc-a318-419f-b477-f0d1b982dcac-proxy-tls\") pod \"error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7\" (UID: \"72b059bc-a318-419f-b477-f0d1b982dcac\") " pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7"
Jul 01 20:56:32.641406 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.641385    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/447b6d3c-1be1-4053-91db-e8f11d74445f-proxy-tls\") pod \"success-200-isvc-f347e-predictor-9c7897d59-775b4\" (UID: \"447b6d3c-1be1-4053-91db-e8f11d74445f\") " pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4"
Jul 01 20:56:32.646527 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.646499    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6b95\" (UniqueName: \"kubernetes.io/projected/447b6d3c-1be1-4053-91db-e8f11d74445f-kube-api-access-h6b95\") pod \"success-200-isvc-f347e-predictor-9c7897d59-775b4\" (UID: \"447b6d3c-1be1-4053-91db-e8f11d74445f\") " pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4"
Jul 01 20:56:32.646599 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.646524    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8c7v\" (UniqueName: \"kubernetes.io/projected/72b059bc-a318-419f-b477-f0d1b982dcac-kube-api-access-j8c7v\") pod \"error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7\" (UID: \"72b059bc-a318-419f-b477-f0d1b982dcac\") " pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7"
Jul 01 20:56:32.685391 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.685371    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4"
Jul 01 20:56:32.753422 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.753388    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7"
Jul 01 20:56:32.833623 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.833555    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4"]
Jul 01 20:56:32.836049 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:56:32.836025    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod447b6d3c_1be1_4053_91db_e8f11d74445f.slice/crio-409ceb70ccd5b5e5e37b8000fa09cfe6a89dd963a4c437edb179f376af93cfbe WatchSource:0}: Error finding container 409ceb70ccd5b5e5e37b8000fa09cfe6a89dd963a4c437edb179f376af93cfbe: Status 404 returned error can't find the container with id 409ceb70ccd5b5e5e37b8000fa09cfe6a89dd963a4c437edb179f376af93cfbe
Jul 01 20:56:32.909317 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:32.909289    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7"]
Jul 01 20:56:32.911631 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:56:32.911604    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b059bc_a318_419f_b477_f0d1b982dcac.slice/crio-c474f79df433a7ec83f362d011931eb51f4d0cf562c19be2564687d65bc6d66c WatchSource:0}: Error finding container c474f79df433a7ec83f362d011931eb51f4d0cf562c19be2564687d65bc6d66c: Status 404 returned error can't find the container with id c474f79df433a7ec83f362d011931eb51f4d0cf562c19be2564687d65bc6d66c
Jul 01 20:56:33.424529 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:33.424498    2564 generic.go:356] "Generic (PLEG): container finished" podID="db188407-051f-40e3-b2f0-8623d40d0429" containerID="798e83bcf15b2b1b7ce0909655d7352f11a15464cd0c84de2d06188f29e6ec3b" exitCode=2
Jul 01 20:56:33.424930 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:33.424564    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx" event={"ID":"db188407-051f-40e3-b2f0-8623d40d0429","Type":"ContainerDied","Data":"798e83bcf15b2b1b7ce0909655d7352f11a15464cd0c84de2d06188f29e6ec3b"}
Jul 01 20:56:33.426362 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:33.426339    2564 generic.go:356] "Generic (PLEG): container finished" podID="1a485aa7-57db-42b5-98c1-9f180e964632" containerID="1385805bf23c8cbc06ea4843190e3280f53d3ef14ede7cd81bde8f4199bb4e0f" exitCode=2
Jul 01 20:56:33.426475 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:33.426416    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc" event={"ID":"1a485aa7-57db-42b5-98c1-9f180e964632","Type":"ContainerDied","Data":"1385805bf23c8cbc06ea4843190e3280f53d3ef14ede7cd81bde8f4199bb4e0f"}
Jul 01 20:56:33.428072 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:33.428050    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4" event={"ID":"447b6d3c-1be1-4053-91db-e8f11d74445f","Type":"ContainerStarted","Data":"ee7eafdb90be56234d212e6fc44ec204d330761cf3e25401782dfdbf0e823e7b"}
Jul 01 20:56:33.428169 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:33.428117    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4"
Jul 01 20:56:33.428169 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:33.428151    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4" event={"ID":"447b6d3c-1be1-4053-91db-e8f11d74445f","Type":"ContainerStarted","Data":"1d9d5f1b4f5f325efe6edffe1f018414d9996261a0fc4b4045b7d1f8d3c2f9ee"}
Jul 01 20:56:33.428286 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:33.428182    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4" event={"ID":"447b6d3c-1be1-4053-91db-e8f11d74445f","Type":"ContainerStarted","Data":"409ceb70ccd5b5e5e37b8000fa09cfe6a89dd963a4c437edb179f376af93cfbe"}
Jul 01 20:56:33.428286 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:33.428212    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4"
Jul 01 20:56:33.429675 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:33.429652    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4" podUID="447b6d3c-1be1-4053-91db-e8f11d74445f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 01 20:56:33.430099 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:33.430080    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7" event={"ID":"72b059bc-a318-419f-b477-f0d1b982dcac","Type":"ContainerStarted","Data":"9179fa5f317a0543c2e556631dde1a8859dae7d5bcbcfa9aaed87ea5eec6619d"}
Jul 01 20:56:33.430207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:33.430128    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7" event={"ID":"72b059bc-a318-419f-b477-f0d1b982dcac","Type":"ContainerStarted","Data":"249100f43230213f40fe82c0cf9c7ab485cdb5cf2a5458bc55575de3ab178f46"}
Jul 01 20:56:33.430207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:33.430162    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7" event={"ID":"72b059bc-a318-419f-b477-f0d1b982dcac","Type":"ContainerStarted","Data":"c474f79df433a7ec83f362d011931eb51f4d0cf562c19be2564687d65bc6d66c"}
Jul 01 20:56:33.430207 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:33.430195    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7"
Jul 01 20:56:33.430367 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:33.430226    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7"
Jul 01 20:56:33.431167 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:33.431146    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7" podUID="72b059bc-a318-419f-b477-f0d1b982dcac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 01 20:56:33.447219 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:33.447180    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4" podStartSLOduration=1.447167431 podStartE2EDuration="1.447167431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:56:33.446092809 +0000 UTC m=+593.122108378" watchObservedRunningTime="2026-07-01 20:56:33.447167431 +0000 UTC m=+593.123182989"
Jul 01 20:56:33.463209 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:33.463168    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7" podStartSLOduration=1.463154039 podStartE2EDuration="1.463154039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:56:33.462324239 +0000 UTC m=+593.138339805" watchObservedRunningTime="2026-07-01 20:56:33.463154039 +0000 UTC m=+593.139169607"
Jul 01 20:56:34.434228 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:34.434193    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7" podUID="72b059bc-a318-419f-b477-f0d1b982dcac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 01 20:56:34.434591 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:34.434277    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4" podUID="447b6d3c-1be1-4053-91db-e8f11d74445f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 01 20:56:35.635495 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:35.635472    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx"
Jul 01 20:56:35.763938 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:35.763848    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db188407-051f-40e3-b2f0-8623d40d0429-kube-rbac-proxy-sar-config\") pod \"db188407-051f-40e3-b2f0-8623d40d0429\" (UID: \"db188407-051f-40e3-b2f0-8623d40d0429\") "
Jul 01 20:56:35.764072 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:35.763953    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db188407-051f-40e3-b2f0-8623d40d0429-proxy-tls\") pod \"db188407-051f-40e3-b2f0-8623d40d0429\" (UID: \"db188407-051f-40e3-b2f0-8623d40d0429\") "
Jul 01 20:56:35.764072 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:35.763983    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6tdg\" (UniqueName: \"kubernetes.io/projected/db188407-051f-40e3-b2f0-8623d40d0429-kube-api-access-v6tdg\") pod \"db188407-051f-40e3-b2f0-8623d40d0429\" (UID: \"db188407-051f-40e3-b2f0-8623d40d0429\") "
Jul 01 20:56:35.764225 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:35.764201    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db188407-051f-40e3-b2f0-8623d40d0429-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "db188407-051f-40e3-b2f0-8623d40d0429" (UID: "db188407-051f-40e3-b2f0-8623d40d0429"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:56:35.766104 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:35.766081    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db188407-051f-40e3-b2f0-8623d40d0429-kube-api-access-v6tdg" (OuterVolumeSpecName: "kube-api-access-v6tdg") pod "db188407-051f-40e3-b2f0-8623d40d0429" (UID: "db188407-051f-40e3-b2f0-8623d40d0429"). InnerVolumeSpecName "kube-api-access-v6tdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:56:35.766245 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:35.766225    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db188407-051f-40e3-b2f0-8623d40d0429-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "db188407-051f-40e3-b2f0-8623d40d0429" (UID: "db188407-051f-40e3-b2f0-8623d40d0429"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:56:35.865359 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:35.865327    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db188407-051f-40e3-b2f0-8623d40d0429-proxy-tls\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:56:35.865359 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:35.865354    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6tdg\" (UniqueName: \"kubernetes.io/projected/db188407-051f-40e3-b2f0-8623d40d0429-kube-api-access-v6tdg\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:56:35.865538 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:35.865370    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db188407-051f-40e3-b2f0-8623d40d0429-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:56:36.099660 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.099613    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc" podUID="1a485aa7-57db-42b5-98c1-9f180e964632" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Jul 01 20:56:36.327905 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.327837    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc"
Jul 01 20:56:36.444022 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.443994    2564 generic.go:356] "Generic (PLEG): container finished" podID="db188407-051f-40e3-b2f0-8623d40d0429" containerID="027f1a3ef0398ae7cbe0bd82122497fc2f80d55bfe8efe3bb6093e1debc772a0" exitCode=0
Jul 01 20:56:36.444158 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.444061    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx"
Jul 01 20:56:36.444158 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.444073    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx" event={"ID":"db188407-051f-40e3-b2f0-8623d40d0429","Type":"ContainerDied","Data":"027f1a3ef0398ae7cbe0bd82122497fc2f80d55bfe8efe3bb6093e1debc772a0"}
Jul 01 20:56:36.444260 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.444176    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx" event={"ID":"db188407-051f-40e3-b2f0-8623d40d0429","Type":"ContainerDied","Data":"bb277e90424d0dd913d2d7e876abe5ee8a2d05d8d0f046f35ba1822396219108"}
Jul 01 20:56:36.444260 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.444214    2564 scope.go:117] "RemoveContainer" containerID="798e83bcf15b2b1b7ce0909655d7352f11a15464cd0c84de2d06188f29e6ec3b"
Jul 01 20:56:36.445709 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.445684    2564 generic.go:356] "Generic (PLEG): container finished" podID="1a485aa7-57db-42b5-98c1-9f180e964632" containerID="d3804b51c282f68b1878bcb247fada75dd25c6997668a6c7418b40a1ed278bae" exitCode=0
Jul 01 20:56:36.445833 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.445748    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc"
Jul 01 20:56:36.445833 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.445762    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc" event={"ID":"1a485aa7-57db-42b5-98c1-9f180e964632","Type":"ContainerDied","Data":"d3804b51c282f68b1878bcb247fada75dd25c6997668a6c7418b40a1ed278bae"}
Jul 01 20:56:36.445833 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.445818    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc" event={"ID":"1a485aa7-57db-42b5-98c1-9f180e964632","Type":"ContainerDied","Data":"7ff5914f9e75355158c1fb49a29d7468ed900edbee4014ba6b0ee87b44015e9e"}
Jul 01 20:56:36.453220 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.453206    2564 scope.go:117] "RemoveContainer" containerID="027f1a3ef0398ae7cbe0bd82122497fc2f80d55bfe8efe3bb6093e1debc772a0"
Jul 01 20:56:36.462070 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.462056    2564 scope.go:117] "RemoveContainer" containerID="798e83bcf15b2b1b7ce0909655d7352f11a15464cd0c84de2d06188f29e6ec3b"
Jul 01 20:56:36.462308 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:56:36.462290    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"798e83bcf15b2b1b7ce0909655d7352f11a15464cd0c84de2d06188f29e6ec3b\": container with ID starting with 798e83bcf15b2b1b7ce0909655d7352f11a15464cd0c84de2d06188f29e6ec3b not found: ID does not exist" containerID="798e83bcf15b2b1b7ce0909655d7352f11a15464cd0c84de2d06188f29e6ec3b"
Jul 01 20:56:36.462356 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.462316    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"798e83bcf15b2b1b7ce0909655d7352f11a15464cd0c84de2d06188f29e6ec3b"} err="failed to get container status \"798e83bcf15b2b1b7ce0909655d7352f11a15464cd0c84de2d06188f29e6ec3b\": rpc error: code = NotFound desc = could not find container \"798e83bcf15b2b1b7ce0909655d7352f11a15464cd0c84de2d06188f29e6ec3b\": container with ID starting with 798e83bcf15b2b1b7ce0909655d7352f11a15464cd0c84de2d06188f29e6ec3b not found: ID does not exist"
Jul 01 20:56:36.462356 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.462331    2564 scope.go:117] "RemoveContainer" containerID="027f1a3ef0398ae7cbe0bd82122497fc2f80d55bfe8efe3bb6093e1debc772a0"
Jul 01 20:56:36.462526 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:56:36.462511    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"027f1a3ef0398ae7cbe0bd82122497fc2f80d55bfe8efe3bb6093e1debc772a0\": container with ID starting with 027f1a3ef0398ae7cbe0bd82122497fc2f80d55bfe8efe3bb6093e1debc772a0 not found: ID does not exist" containerID="027f1a3ef0398ae7cbe0bd82122497fc2f80d55bfe8efe3bb6093e1debc772a0"
Jul 01 20:56:36.462562 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.462530    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"027f1a3ef0398ae7cbe0bd82122497fc2f80d55bfe8efe3bb6093e1debc772a0"} err="failed to get container status \"027f1a3ef0398ae7cbe0bd82122497fc2f80d55bfe8efe3bb6093e1debc772a0\": rpc error: code = NotFound desc = could not find container \"027f1a3ef0398ae7cbe0bd82122497fc2f80d55bfe8efe3bb6093e1debc772a0\": container with ID starting with 027f1a3ef0398ae7cbe0bd82122497fc2f80d55bfe8efe3bb6093e1debc772a0 not found: ID does not exist"
Jul 01 20:56:36.462562 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.462543    2564 scope.go:117] "RemoveContainer" containerID="1385805bf23c8cbc06ea4843190e3280f53d3ef14ede7cd81bde8f4199bb4e0f"
Jul 01 20:56:36.467311 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.467256    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx"]
Jul 01 20:56:36.469515 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.469493    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-25c51-predictor-54665ff85-mxfgx"]
Jul 01 20:56:36.469669 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.469653    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a485aa7-57db-42b5-98c1-9f180e964632-kube-rbac-proxy-sar-config\") pod \"1a485aa7-57db-42b5-98c1-9f180e964632\" (UID: \"1a485aa7-57db-42b5-98c1-9f180e964632\") "
Jul 01 20:56:36.469745 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.469733    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxv9j\" (UniqueName: \"kubernetes.io/projected/1a485aa7-57db-42b5-98c1-9f180e964632-kube-api-access-fxv9j\") pod \"1a485aa7-57db-42b5-98c1-9f180e964632\" (UID: \"1a485aa7-57db-42b5-98c1-9f180e964632\") "
Jul 01 20:56:36.469816 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.469800    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a485aa7-57db-42b5-98c1-9f180e964632-proxy-tls\") pod \"1a485aa7-57db-42b5-98c1-9f180e964632\" (UID: \"1a485aa7-57db-42b5-98c1-9f180e964632\") "
Jul 01 20:56:36.470064 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.470043    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a485aa7-57db-42b5-98c1-9f180e964632-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1a485aa7-57db-42b5-98c1-9f180e964632" (UID: "1a485aa7-57db-42b5-98c1-9f180e964632"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:56:36.471440 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.471410    2564 scope.go:117] "RemoveContainer" containerID="d3804b51c282f68b1878bcb247fada75dd25c6997668a6c7418b40a1ed278bae"
Jul 01 20:56:36.471705 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.471686    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a485aa7-57db-42b5-98c1-9f180e964632-kube-api-access-fxv9j" (OuterVolumeSpecName: "kube-api-access-fxv9j") pod "1a485aa7-57db-42b5-98c1-9f180e964632" (UID: "1a485aa7-57db-42b5-98c1-9f180e964632"). InnerVolumeSpecName "kube-api-access-fxv9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:56:36.471921 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.471899    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a485aa7-57db-42b5-98c1-9f180e964632-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1a485aa7-57db-42b5-98c1-9f180e964632" (UID: "1a485aa7-57db-42b5-98c1-9f180e964632"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:56:36.485537 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.485521    2564 scope.go:117] "RemoveContainer" containerID="1385805bf23c8cbc06ea4843190e3280f53d3ef14ede7cd81bde8f4199bb4e0f"
Jul 01 20:56:36.485778 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:56:36.485762    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1385805bf23c8cbc06ea4843190e3280f53d3ef14ede7cd81bde8f4199bb4e0f\": container with ID starting with 1385805bf23c8cbc06ea4843190e3280f53d3ef14ede7cd81bde8f4199bb4e0f not found: ID does not exist" containerID="1385805bf23c8cbc06ea4843190e3280f53d3ef14ede7cd81bde8f4199bb4e0f"
Jul 01 20:56:36.485837 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.485784    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1385805bf23c8cbc06ea4843190e3280f53d3ef14ede7cd81bde8f4199bb4e0f"} err="failed to get container status \"1385805bf23c8cbc06ea4843190e3280f53d3ef14ede7cd81bde8f4199bb4e0f\": rpc error: code = NotFound desc = could not find container \"1385805bf23c8cbc06ea4843190e3280f53d3ef14ede7cd81bde8f4199bb4e0f\": container with ID starting with 1385805bf23c8cbc06ea4843190e3280f53d3ef14ede7cd81bde8f4199bb4e0f not found: ID does not exist"
Jul 01 20:56:36.485837 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.485800    2564 scope.go:117] "RemoveContainer" containerID="d3804b51c282f68b1878bcb247fada75dd25c6997668a6c7418b40a1ed278bae"
Jul 01 20:56:36.486046 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:56:36.486030    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3804b51c282f68b1878bcb247fada75dd25c6997668a6c7418b40a1ed278bae\": container with ID starting with d3804b51c282f68b1878bcb247fada75dd25c6997668a6c7418b40a1ed278bae not found: ID does not exist" containerID="d3804b51c282f68b1878bcb247fada75dd25c6997668a6c7418b40a1ed278bae"
Jul 01 20:56:36.486089 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.486052    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3804b51c282f68b1878bcb247fada75dd25c6997668a6c7418b40a1ed278bae"} err="failed to get container status \"d3804b51c282f68b1878bcb247fada75dd25c6997668a6c7418b40a1ed278bae\": rpc error: code = NotFound desc = could not find container \"d3804b51c282f68b1878bcb247fada75dd25c6997668a6c7418b40a1ed278bae\": container with ID starting with d3804b51c282f68b1878bcb247fada75dd25c6997668a6c7418b40a1ed278bae not found: ID does not exist"
Jul 01 20:56:36.570545 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.570515    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxv9j\" (UniqueName: \"kubernetes.io/projected/1a485aa7-57db-42b5-98c1-9f180e964632-kube-api-access-fxv9j\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:56:36.570545 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.570539    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a485aa7-57db-42b5-98c1-9f180e964632-proxy-tls\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:56:36.570545 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.570551    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a485aa7-57db-42b5-98c1-9f180e964632-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:56:36.758171 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.758144    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db188407-051f-40e3-b2f0-8623d40d0429" path="/var/lib/kubelet/pods/db188407-051f-40e3-b2f0-8623d40d0429/volumes"
Jul 01 20:56:36.768211 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.768186    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc"]
Jul 01 20:56:36.772106 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:36.772078    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-25c51-predictor-64f4bf5df6-t9jxc"]
Jul 01 20:56:38.757909 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:38.757879    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a485aa7-57db-42b5-98c1-9f180e964632" path="/var/lib/kubelet/pods/1a485aa7-57db-42b5-98c1-9f180e964632/volumes"
Jul 01 20:56:39.438206 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:39.438179    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4"
Jul 01 20:56:39.438613 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:39.438592    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7"
Jul 01 20:56:39.438699 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:39.438658    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4" podUID="447b6d3c-1be1-4053-91db-e8f11d74445f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 01 20:56:39.438992 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:39.438972    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7" podUID="72b059bc-a318-419f-b477-f0d1b982dcac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 01 20:56:40.100964 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:40.100937    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"
Jul 01 20:56:40.732737 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:40.732706    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/ovn-acl-logging/0.log"
Jul 01 20:56:40.735360 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:40.735339    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/ovn-acl-logging/0.log"
Jul 01 20:56:49.439451 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:49.439420    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4" podUID="447b6d3c-1be1-4053-91db-e8f11d74445f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 01 20:56:49.441737 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:49.439415    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7" podUID="72b059bc-a318-419f-b477-f0d1b982dcac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 01 20:56:59.439295 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:59.439260    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4" podUID="447b6d3c-1be1-4053-91db-e8f11d74445f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 01 20:56:59.439660 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:56:59.439261    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7" podUID="72b059bc-a318-419f-b477-f0d1b982dcac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 01 20:57:02.297289 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.297192    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"]
Jul 01 20:57:02.297757 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.297495    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm" podUID="e28fcffa-cde8-44ce-9015-a1ad49645e7a" containerName="kserve-container" containerID="cri-o://59a868890366f9236fbef38b9be005b46dac9a8616a5633b647c66b6a86a31b2" gracePeriod=30
Jul 01 20:57:02.297757 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.297519    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm" podUID="e28fcffa-cde8-44ce-9015-a1ad49645e7a" containerName="kube-rbac-proxy" containerID="cri-o://c87f86307ab796673411960278297376de8ceebc5ec5e55043e4d231bf36ae2d" gracePeriod=30
Jul 01 20:57:02.369718 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.369686    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"]
Jul 01 20:57:02.369992 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.369962    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2" podUID="603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" containerName="kserve-container" containerID="cri-o://3d401e3428a1ce8e8356bed74b80db9385676c7c9aacb672aa31984e20cdecda" gracePeriod=30
Jul 01 20:57:02.370191 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.370016    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2" podUID="603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" containerName="kube-rbac-proxy" containerID="cri-o://2a2725f6ed0e1d310b56c79cee0dade4d1ea369da43b45f80addda04ea41b2a5" gracePeriod=30
Jul 01 20:57:02.371346 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.371326    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2"]
Jul 01 20:57:02.371711 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.371696    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a485aa7-57db-42b5-98c1-9f180e964632" containerName="kserve-container"
Jul 01 20:57:02.371789 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.371714    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a485aa7-57db-42b5-98c1-9f180e964632" containerName="kserve-container"
Jul 01 20:57:02.371789 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.371730    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a485aa7-57db-42b5-98c1-9f180e964632" containerName="kube-rbac-proxy"
Jul 01 20:57:02.371789 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.371738    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a485aa7-57db-42b5-98c1-9f180e964632" containerName="kube-rbac-proxy"
Jul 01 20:57:02.371789 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.371761    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db188407-051f-40e3-b2f0-8623d40d0429" containerName="kserve-container"
Jul 01 20:57:02.371789 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.371770    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="db188407-051f-40e3-b2f0-8623d40d0429" containerName="kserve-container"
Jul 01 20:57:02.372054 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.371796    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db188407-051f-40e3-b2f0-8623d40d0429" containerName="kube-rbac-proxy"
Jul 01 20:57:02.372054 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.371805    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="db188407-051f-40e3-b2f0-8623d40d0429" containerName="kube-rbac-proxy"
Jul 01 20:57:02.372054 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.371902    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a485aa7-57db-42b5-98c1-9f180e964632" containerName="kserve-container"
Jul 01 20:57:02.372054 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.371916    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="db188407-051f-40e3-b2f0-8623d40d0429" containerName="kube-rbac-proxy"
Jul 01 20:57:02.372054 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.371925    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a485aa7-57db-42b5-98c1-9f180e964632" containerName="kube-rbac-proxy"
Jul 01 20:57:02.372054 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.371937    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="db188407-051f-40e3-b2f0-8623d40d0429" containerName="kserve-container"
Jul 01 20:57:02.381766 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.381743    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2"
Jul 01 20:57:02.384754 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.384710    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a93e1-predictor-serving-cert\""
Jul 01 20:57:02.385060 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.385029    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a93e1-kube-rbac-proxy-sar-config\""
Jul 01 20:57:02.385421 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.385404    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2"]
Jul 01 20:57:02.475199 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.475158    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7"]
Jul 01 20:57:02.478665 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.478645    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7"
Jul 01 20:57:02.481849 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.481832    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a93e1-predictor-serving-cert\""
Jul 01 20:57:02.481949 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.481890    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a93e1-kube-rbac-proxy-sar-config\""
Jul 01 20:57:02.489068 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.489049    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7"]
Jul 01 20:57:02.504845 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.504811    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1392569-add7-437e-b0d7-ef1ad3b24949-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a93e1-predictor-657768547-67mn2\" (UID: \"e1392569-add7-437e-b0d7-ef1ad3b24949\") " pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2"
Jul 01 20:57:02.504988 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.504909    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-869xr\" (UniqueName: \"kubernetes.io/projected/e1392569-add7-437e-b0d7-ef1ad3b24949-kube-api-access-869xr\") pod \"success-200-isvc-a93e1-predictor-657768547-67mn2\" (UID: \"e1392569-add7-437e-b0d7-ef1ad3b24949\") " pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2"
Jul 01 20:57:02.505048 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.505028    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1392569-add7-437e-b0d7-ef1ad3b24949-proxy-tls\") pod \"success-200-isvc-a93e1-predictor-657768547-67mn2\" (UID: \"e1392569-add7-437e-b0d7-ef1ad3b24949\") " pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2"
Jul 01 20:57:02.541674 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.541647    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"]
Jul 01 20:57:02.541926 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.541890    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg" podUID="140bfc0f-a451-4a3b-9acc-1fdcde1702e5" containerName="kserve-container" containerID="cri-o://cde4850a0cd164f76179a39ed26154436a08968c098eecc6f63e57ee6103b4d2" gracePeriod=30
Jul 01 20:57:02.541926 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.541912    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg" podUID="140bfc0f-a451-4a3b-9acc-1fdcde1702e5" containerName="kube-rbac-proxy" containerID="cri-o://ee1d416942708d927bebf43875c2df20cb412c5d4418da32d8170e60c2e73187" gracePeriod=30
Jul 01 20:57:02.562854 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.562829    2564 generic.go:356] "Generic (PLEG): container finished" podID="603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" containerID="2a2725f6ed0e1d310b56c79cee0dade4d1ea369da43b45f80addda04ea41b2a5" exitCode=2
Jul 01 20:57:02.562953 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.562894    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2" event={"ID":"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4","Type":"ContainerDied","Data":"2a2725f6ed0e1d310b56c79cee0dade4d1ea369da43b45f80addda04ea41b2a5"}
Jul 01 20:57:02.564791 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.564769    2564 generic.go:356] "Generic (PLEG): container finished" podID="e28fcffa-cde8-44ce-9015-a1ad49645e7a" containerID="c87f86307ab796673411960278297376de8ceebc5ec5e55043e4d231bf36ae2d" exitCode=2
Jul 01 20:57:02.564892 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.564800    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm" event={"ID":"e28fcffa-cde8-44ce-9015-a1ad49645e7a","Type":"ContainerDied","Data":"c87f86307ab796673411960278297376de8ceebc5ec5e55043e4d231bf36ae2d"}
Jul 01 20:57:02.606119 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.606091    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-869xr\" (UniqueName: \"kubernetes.io/projected/e1392569-add7-437e-b0d7-ef1ad3b24949-kube-api-access-869xr\") pod \"success-200-isvc-a93e1-predictor-657768547-67mn2\" (UID: \"e1392569-add7-437e-b0d7-ef1ad3b24949\") " pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2"
Jul 01 20:57:02.606243 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.606144    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/361e65bc-14bb-4657-a4ea-85dedccfc845-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7\" (UID: \"361e65bc-14bb-4657-a4ea-85dedccfc845\") " pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7"
Jul 01 20:57:02.606243 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.606173    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd7xj\" (UniqueName: \"kubernetes.io/projected/361e65bc-14bb-4657-a4ea-85dedccfc845-kube-api-access-xd7xj\") pod \"error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7\" (UID: \"361e65bc-14bb-4657-a4ea-85dedccfc845\") " pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7"
Jul 01 20:57:02.606312 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.606276    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1392569-add7-437e-b0d7-ef1ad3b24949-proxy-tls\") pod \"success-200-isvc-a93e1-predictor-657768547-67mn2\" (UID: \"e1392569-add7-437e-b0d7-ef1ad3b24949\") " pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2"
Jul 01 20:57:02.606346 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.606309    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1392569-add7-437e-b0d7-ef1ad3b24949-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a93e1-predictor-657768547-67mn2\" (UID: \"e1392569-add7-437e-b0d7-ef1ad3b24949\") " pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2"
Jul 01 20:57:02.606346 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.606329    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/361e65bc-14bb-4657-a4ea-85dedccfc845-proxy-tls\") pod \"error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7\" (UID: \"361e65bc-14bb-4657-a4ea-85dedccfc845\") " pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7"
Jul 01 20:57:02.607097 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.607079    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1392569-add7-437e-b0d7-ef1ad3b24949-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a93e1-predictor-657768547-67mn2\" (UID: \"e1392569-add7-437e-b0d7-ef1ad3b24949\") " pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2"
Jul 01 20:57:02.608631 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.608613    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1392569-add7-437e-b0d7-ef1ad3b24949-proxy-tls\") pod \"success-200-isvc-a93e1-predictor-657768547-67mn2\" (UID: \"e1392569-add7-437e-b0d7-ef1ad3b24949\") " pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2"
Jul 01 20:57:02.621502 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.621474    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-869xr\" (UniqueName: \"kubernetes.io/projected/e1392569-add7-437e-b0d7-ef1ad3b24949-kube-api-access-869xr\") pod \"success-200-isvc-a93e1-predictor-657768547-67mn2\" (UID: \"e1392569-add7-437e-b0d7-ef1ad3b24949\") " pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2"
Jul 01 20:57:02.693319 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.693299    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2"
Jul 01 20:57:02.707441 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.707414    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/361e65bc-14bb-4657-a4ea-85dedccfc845-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7\" (UID: \"361e65bc-14bb-4657-a4ea-85dedccfc845\") " pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7"
Jul 01 20:57:02.707562 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.707475    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd7xj\" (UniqueName: \"kubernetes.io/projected/361e65bc-14bb-4657-a4ea-85dedccfc845-kube-api-access-xd7xj\") pod \"error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7\" (UID: \"361e65bc-14bb-4657-a4ea-85dedccfc845\") " pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7"
Jul 01 20:57:02.707562 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.707535    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/361e65bc-14bb-4657-a4ea-85dedccfc845-proxy-tls\") pod \"error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7\" (UID: \"361e65bc-14bb-4657-a4ea-85dedccfc845\") " pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7"
Jul 01 20:57:02.708073 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.708051    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/361e65bc-14bb-4657-a4ea-85dedccfc845-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7\" (UID: \"361e65bc-14bb-4657-a4ea-85dedccfc845\") " pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7"
Jul 01 20:57:02.710235 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.710216    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/361e65bc-14bb-4657-a4ea-85dedccfc845-proxy-tls\") pod \"error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7\" (UID: \"361e65bc-14bb-4657-a4ea-85dedccfc845\") " pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7"
Jul 01 20:57:02.718056 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.718035    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd7xj\" (UniqueName: \"kubernetes.io/projected/361e65bc-14bb-4657-a4ea-85dedccfc845-kube-api-access-xd7xj\") pod \"error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7\" (UID: \"361e65bc-14bb-4657-a4ea-85dedccfc845\") " pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7"
Jul 01 20:57:02.789215 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.789190    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7"
Jul 01 20:57:02.832306 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.832268    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2"]
Jul 01 20:57:02.834891 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:57:02.834845    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1392569_add7_437e_b0d7_ef1ad3b24949.slice/crio-6c273678bfbfa630cf21d58f8dac03629b32d1dff08097333559627adb8c7fcb WatchSource:0}: Error finding container 6c273678bfbfa630cf21d58f8dac03629b32d1dff08097333559627adb8c7fcb: Status 404 returned error can't find the container with id 6c273678bfbfa630cf21d58f8dac03629b32d1dff08097333559627adb8c7fcb
Jul 01 20:57:02.937067 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:02.937042    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7"]
Jul 01 20:57:02.939776 ip-10-0-131-198 kubenswrapper[2564]: W0701 20:57:02.939752    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod361e65bc_14bb_4657_a4ea_85dedccfc845.slice/crio-69d351f9c86bf0426e37ed0b8fb6972de046e9a63a5456497a47c0e4acf4139c WatchSource:0}: Error finding container 69d351f9c86bf0426e37ed0b8fb6972de046e9a63a5456497a47c0e4acf4139c: Status 404 returned error can't find the container with id 69d351f9c86bf0426e37ed0b8fb6972de046e9a63a5456497a47c0e4acf4139c
Jul 01 20:57:03.570952 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:03.570918    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7" event={"ID":"361e65bc-14bb-4657-a4ea-85dedccfc845","Type":"ContainerStarted","Data":"8a6f12f9049e30159e73bc76a5824603664ee49e1285d60663536deac642e2ac"}
Jul 01 20:57:03.571393 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:03.570975    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7" event={"ID":"361e65bc-14bb-4657-a4ea-85dedccfc845","Type":"ContainerStarted","Data":"5514eb0a6028a7606f6e8cfd04622e4f07f10184a788a98a48833e850bc269ae"}
Jul 01 20:57:03.571393 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:03.571002    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7" event={"ID":"361e65bc-14bb-4657-a4ea-85dedccfc845","Type":"ContainerStarted","Data":"69d351f9c86bf0426e37ed0b8fb6972de046e9a63a5456497a47c0e4acf4139c"}
Jul 01 20:57:03.571393 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:03.571146    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7"
Jul 01 20:57:03.571393 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:03.571188    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7"
Jul 01 20:57:03.572670 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:03.572639    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7" podUID="361e65bc-14bb-4657-a4ea-85dedccfc845" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 01 20:57:03.573311 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:03.573290    2564 generic.go:356] "Generic (PLEG): container finished" podID="140bfc0f-a451-4a3b-9acc-1fdcde1702e5" containerID="ee1d416942708d927bebf43875c2df20cb412c5d4418da32d8170e60c2e73187" exitCode=2
Jul 01 20:57:03.573396 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:03.573367    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg" event={"ID":"140bfc0f-a451-4a3b-9acc-1fdcde1702e5","Type":"ContainerDied","Data":"ee1d416942708d927bebf43875c2df20cb412c5d4418da32d8170e60c2e73187"}
Jul 01 20:57:03.574913 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:03.574885    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2" event={"ID":"e1392569-add7-437e-b0d7-ef1ad3b24949","Type":"ContainerStarted","Data":"153c044199852dc9923bfa5e7ab0a8558aa0fdc217ea0878083245cd67bb43cd"}
Jul 01 20:57:03.575023 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:03.574935    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2" event={"ID":"e1392569-add7-437e-b0d7-ef1ad3b24949","Type":"ContainerStarted","Data":"c76f237315f1588e82542f92a0a401e0c9fc646abbca85be4bdb5020f1b6a95b"}
Jul 01 20:57:03.575023 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:03.574961    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2" event={"ID":"e1392569-add7-437e-b0d7-ef1ad3b24949","Type":"ContainerStarted","Data":"6c273678bfbfa630cf21d58f8dac03629b32d1dff08097333559627adb8c7fcb"}
Jul 01 20:57:03.575023 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:03.574985    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2"
Jul 01 20:57:03.590681 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:03.590645    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7" podStartSLOduration=1.590634152 podStartE2EDuration="1.590634152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:57:03.588301801 +0000 UTC m=+623.264317390" watchObservedRunningTime="2026-07-01 20:57:03.590634152 +0000 UTC m=+623.266649719"
Jul 01 20:57:04.580320 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:04.580281    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2"
Jul 01 20:57:04.580320 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:04.580295    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7" podUID="361e65bc-14bb-4657-a4ea-85dedccfc845" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 01 20:57:04.581313 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:04.581288    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2" podUID="e1392569-add7-437e-b0d7-ef1ad3b24949" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 01 20:57:05.093868 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:05.093826    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm" podUID="e28fcffa-cde8-44ce-9015-a1ad49645e7a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Jul 01 20:57:05.584613 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:05.584576    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2" podUID="e1392569-add7-437e-b0d7-ef1ad3b24949" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 01 20:57:06.099509 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:06.099465    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2" podUID="603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.18:8643/healthz\": dial tcp 10.133.0.18:8643: connect: connection refused"
Jul 01 20:57:06.100074 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:06.100042    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg" podUID="140bfc0f-a451-4a3b-9acc-1fdcde1702e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Jul 01 20:57:07.291272 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.291251    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"
Jul 01 20:57:07.312450 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.312407    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2" podStartSLOduration=5.312391073 podStartE2EDuration="5.312391073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:57:03.606631069 +0000 UTC m=+623.282646638" watchObservedRunningTime="2026-07-01 20:57:07.312391073 +0000 UTC m=+626.988406640"
Jul 01 20:57:07.457124 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.457091    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-kserve-provision-location\") pod \"140bfc0f-a451-4a3b-9acc-1fdcde1702e5\" (UID: \"140bfc0f-a451-4a3b-9acc-1fdcde1702e5\") "
Jul 01 20:57:07.457330 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.457136    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-proxy-tls\") pod \"140bfc0f-a451-4a3b-9acc-1fdcde1702e5\" (UID: \"140bfc0f-a451-4a3b-9acc-1fdcde1702e5\") "
Jul 01 20:57:07.457330 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.457180    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h74tf\" (UniqueName: \"kubernetes.io/projected/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-kube-api-access-h74tf\") pod \"140bfc0f-a451-4a3b-9acc-1fdcde1702e5\" (UID: \"140bfc0f-a451-4a3b-9acc-1fdcde1702e5\") "
Jul 01 20:57:07.457330 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.457248    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-cabundle-cert\") pod \"140bfc0f-a451-4a3b-9acc-1fdcde1702e5\" (UID: \"140bfc0f-a451-4a3b-9acc-1fdcde1702e5\") "
Jul 01 20:57:07.457330 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.457302    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-kube-rbac-proxy-sar-config\") pod \"140bfc0f-a451-4a3b-9acc-1fdcde1702e5\" (UID: \"140bfc0f-a451-4a3b-9acc-1fdcde1702e5\") "
Jul 01 20:57:07.457604 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.457430    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "140bfc0f-a451-4a3b-9acc-1fdcde1702e5" (UID: "140bfc0f-a451-4a3b-9acc-1fdcde1702e5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 20:57:07.457664 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.457618    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "140bfc0f-a451-4a3b-9acc-1fdcde1702e5" (UID: "140bfc0f-a451-4a3b-9acc-1fdcde1702e5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:57:07.457723 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.457715    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-kserve-provision-location\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:57:07.457775 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.457732    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-cabundle-cert\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:57:07.457775 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.457727    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "140bfc0f-a451-4a3b-9acc-1fdcde1702e5" (UID: "140bfc0f-a451-4a3b-9acc-1fdcde1702e5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:57:07.459274 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.459252    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-kube-api-access-h74tf" (OuterVolumeSpecName: "kube-api-access-h74tf") pod "140bfc0f-a451-4a3b-9acc-1fdcde1702e5" (UID: "140bfc0f-a451-4a3b-9acc-1fdcde1702e5"). InnerVolumeSpecName "kube-api-access-h74tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:57:07.459325 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.459314    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "140bfc0f-a451-4a3b-9acc-1fdcde1702e5" (UID: "140bfc0f-a451-4a3b-9acc-1fdcde1702e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:57:07.559053 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.559021    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-proxy-tls\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:57:07.559053 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.559047    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h74tf\" (UniqueName: \"kubernetes.io/projected/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-kube-api-access-h74tf\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:57:07.559053 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.559059    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/140bfc0f-a451-4a3b-9acc-1fdcde1702e5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:57:07.596720 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.596680    2564 generic.go:356] "Generic (PLEG): container finished" podID="140bfc0f-a451-4a3b-9acc-1fdcde1702e5" containerID="cde4850a0cd164f76179a39ed26154436a08968c098eecc6f63e57ee6103b4d2" exitCode=0
Jul 01 20:57:07.596890 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.596758    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg" event={"ID":"140bfc0f-a451-4a3b-9acc-1fdcde1702e5","Type":"ContainerDied","Data":"cde4850a0cd164f76179a39ed26154436a08968c098eecc6f63e57ee6103b4d2"}
Jul 01 20:57:07.596890 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.596817    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg" event={"ID":"140bfc0f-a451-4a3b-9acc-1fdcde1702e5","Type":"ContainerDied","Data":"ce572fac15a38951dbadd65ea7366387a70290c305f384165f46bd2fae8b9b95"}
Jul 01 20:57:07.596890 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.596886    2564 scope.go:117] "RemoveContainer" containerID="ee1d416942708d927bebf43875c2df20cb412c5d4418da32d8170e60c2e73187"
Jul 01 20:57:07.597102 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.597084    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"
Jul 01 20:57:07.614042 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.614024    2564 scope.go:117] "RemoveContainer" containerID="cde4850a0cd164f76179a39ed26154436a08968c098eecc6f63e57ee6103b4d2"
Jul 01 20:57:07.621099 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.621074    2564 scope.go:117] "RemoveContainer" containerID="f8c03d52d5370081448c1533839f24b1bea589a1f199d69e2030f85d5af3e0eb"
Jul 01 20:57:07.623537 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.623516    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"]
Jul 01 20:57:07.627597 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.627573    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-9b7pg"]
Jul 01 20:57:07.629498 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.629464    2564 scope.go:117] "RemoveContainer" containerID="ee1d416942708d927bebf43875c2df20cb412c5d4418da32d8170e60c2e73187"
Jul 01 20:57:07.629737 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:57:07.629720    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee1d416942708d927bebf43875c2df20cb412c5d4418da32d8170e60c2e73187\": container with ID starting with ee1d416942708d927bebf43875c2df20cb412c5d4418da32d8170e60c2e73187 not found: ID does not exist" containerID="ee1d416942708d927bebf43875c2df20cb412c5d4418da32d8170e60c2e73187"
Jul 01 20:57:07.629784 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.629751    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee1d416942708d927bebf43875c2df20cb412c5d4418da32d8170e60c2e73187"} err="failed to get container status \"ee1d416942708d927bebf43875c2df20cb412c5d4418da32d8170e60c2e73187\": rpc error: code = NotFound desc = could not find container \"ee1d416942708d927bebf43875c2df20cb412c5d4418da32d8170e60c2e73187\": container with ID starting with ee1d416942708d927bebf43875c2df20cb412c5d4418da32d8170e60c2e73187 not found: ID does not exist"
Jul 01 20:57:07.629784 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.629768    2564 scope.go:117] "RemoveContainer" containerID="cde4850a0cd164f76179a39ed26154436a08968c098eecc6f63e57ee6103b4d2"
Jul 01 20:57:07.630059 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:57:07.630037    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cde4850a0cd164f76179a39ed26154436a08968c098eecc6f63e57ee6103b4d2\": container with ID starting with cde4850a0cd164f76179a39ed26154436a08968c098eecc6f63e57ee6103b4d2 not found: ID does not exist" containerID="cde4850a0cd164f76179a39ed26154436a08968c098eecc6f63e57ee6103b4d2"
Jul 01 20:57:07.630099 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.630066    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cde4850a0cd164f76179a39ed26154436a08968c098eecc6f63e57ee6103b4d2"} err="failed to get container status \"cde4850a0cd164f76179a39ed26154436a08968c098eecc6f63e57ee6103b4d2\": rpc error: code = NotFound desc = could not find container \"cde4850a0cd164f76179a39ed26154436a08968c098eecc6f63e57ee6103b4d2\": container with ID starting with cde4850a0cd164f76179a39ed26154436a08968c098eecc6f63e57ee6103b4d2 not found: ID does not exist"
Jul 01 20:57:07.630099 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.630088    2564 scope.go:117] "RemoveContainer" containerID="f8c03d52d5370081448c1533839f24b1bea589a1f199d69e2030f85d5af3e0eb"
Jul 01 20:57:07.630278 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:57:07.630261    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8c03d52d5370081448c1533839f24b1bea589a1f199d69e2030f85d5af3e0eb\": container with ID starting with f8c03d52d5370081448c1533839f24b1bea589a1f199d69e2030f85d5af3e0eb not found: ID does not exist" containerID="f8c03d52d5370081448c1533839f24b1bea589a1f199d69e2030f85d5af3e0eb"
Jul 01 20:57:07.630313 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:07.630283    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c03d52d5370081448c1533839f24b1bea589a1f199d69e2030f85d5af3e0eb"} err="failed to get container status \"f8c03d52d5370081448c1533839f24b1bea589a1f199d69e2030f85d5af3e0eb\": rpc error: code = NotFound desc = could not find container \"f8c03d52d5370081448c1533839f24b1bea589a1f199d69e2030f85d5af3e0eb\": container with ID starting with f8c03d52d5370081448c1533839f24b1bea589a1f199d69e2030f85d5af3e0eb not found: ID does not exist"
Jul 01 20:57:08.422616 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.422595    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"
Jul 01 20:57:08.449578 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.449558    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"
Jul 01 20:57:08.568558 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.568529    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-kserve-provision-location\") pod \"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4\" (UID: \"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4\") "
Jul 01 20:57:08.568558 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.568563    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e28fcffa-cde8-44ce-9015-a1ad49645e7a-kserve-provision-location\") pod \"e28fcffa-cde8-44ce-9015-a1ad49645e7a\" (UID: \"e28fcffa-cde8-44ce-9015-a1ad49645e7a\") "
Jul 01 20:57:08.568798 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.568601    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-proxy-tls\") pod \"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4\" (UID: \"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4\") "
Jul 01 20:57:08.568798 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.568638    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-cabundle-cert\") pod \"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4\" (UID: \"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4\") "
Jul 01 20:57:08.568798 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.568724    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mfc2\" (UniqueName: \"kubernetes.io/projected/e28fcffa-cde8-44ce-9015-a1ad49645e7a-kube-api-access-8mfc2\") pod \"e28fcffa-cde8-44ce-9015-a1ad49645e7a\" (UID: \"e28fcffa-cde8-44ce-9015-a1ad49645e7a\") "
Jul 01 20:57:08.568798 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.568775    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdb9h\" (UniqueName: \"kubernetes.io/projected/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-kube-api-access-wdb9h\") pod \"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4\" (UID: \"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4\") "
Jul 01 20:57:08.569035 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.568838    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e28fcffa-cde8-44ce-9015-a1ad49645e7a-kube-rbac-proxy-sar-config\") pod \"e28fcffa-cde8-44ce-9015-a1ad49645e7a\" (UID: \"e28fcffa-cde8-44ce-9015-a1ad49645e7a\") "
Jul 01 20:57:08.569035 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.568934    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-kube-rbac-proxy-sar-config\") pod \"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4\" (UID: \"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4\") "
Jul 01 20:57:08.569035 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.568944    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" (UID: "603c3269-3a3d-4ff6-b52a-41bb9af0d5f4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 20:57:08.569035 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.568947    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28fcffa-cde8-44ce-9015-a1ad49645e7a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e28fcffa-cde8-44ce-9015-a1ad49645e7a" (UID: "e28fcffa-cde8-44ce-9015-a1ad49645e7a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 20:57:08.569035 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.568960    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e28fcffa-cde8-44ce-9015-a1ad49645e7a-cabundle-cert\") pod \"e28fcffa-cde8-44ce-9015-a1ad49645e7a\" (UID: \"e28fcffa-cde8-44ce-9015-a1ad49645e7a\") "
Jul 01 20:57:08.569035 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.568995    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e28fcffa-cde8-44ce-9015-a1ad49645e7a-proxy-tls\") pod \"e28fcffa-cde8-44ce-9015-a1ad49645e7a\" (UID: \"e28fcffa-cde8-44ce-9015-a1ad49645e7a\") "
Jul 01 20:57:08.569035 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.569028    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" (UID: "603c3269-3a3d-4ff6-b52a-41bb9af0d5f4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:57:08.569381 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.569277    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-cabundle-cert\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:57:08.569381 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.569295    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-kserve-provision-location\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:57:08.569381 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.569309    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e28fcffa-cde8-44ce-9015-a1ad49645e7a-kserve-provision-location\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:57:08.569381 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.569334    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e28fcffa-cde8-44ce-9015-a1ad49645e7a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e28fcffa-cde8-44ce-9015-a1ad49645e7a" (UID: "e28fcffa-cde8-44ce-9015-a1ad49645e7a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:57:08.569381 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.569369    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" (UID: "603c3269-3a3d-4ff6-b52a-41bb9af0d5f4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:57:08.569618 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.569421    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e28fcffa-cde8-44ce-9015-a1ad49645e7a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e28fcffa-cde8-44ce-9015-a1ad49645e7a" (UID: "e28fcffa-cde8-44ce-9015-a1ad49645e7a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:57:08.570817 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.570790    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e28fcffa-cde8-44ce-9015-a1ad49645e7a-kube-api-access-8mfc2" (OuterVolumeSpecName: "kube-api-access-8mfc2") pod "e28fcffa-cde8-44ce-9015-a1ad49645e7a" (UID: "e28fcffa-cde8-44ce-9015-a1ad49645e7a"). InnerVolumeSpecName "kube-api-access-8mfc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:57:08.570919 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.570814    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" (UID: "603c3269-3a3d-4ff6-b52a-41bb9af0d5f4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:57:08.570956 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.570935    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-kube-api-access-wdb9h" (OuterVolumeSpecName: "kube-api-access-wdb9h") pod "603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" (UID: "603c3269-3a3d-4ff6-b52a-41bb9af0d5f4"). InnerVolumeSpecName "kube-api-access-wdb9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:57:08.571084 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.571067    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e28fcffa-cde8-44ce-9015-a1ad49645e7a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e28fcffa-cde8-44ce-9015-a1ad49645e7a" (UID: "e28fcffa-cde8-44ce-9015-a1ad49645e7a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:57:08.606015 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.605985    2564 generic.go:356] "Generic (PLEG): container finished" podID="e28fcffa-cde8-44ce-9015-a1ad49645e7a" containerID="59a868890366f9236fbef38b9be005b46dac9a8616a5633b647c66b6a86a31b2" exitCode=0
Jul 01 20:57:08.606168 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.606019    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm" event={"ID":"e28fcffa-cde8-44ce-9015-a1ad49645e7a","Type":"ContainerDied","Data":"59a868890366f9236fbef38b9be005b46dac9a8616a5633b647c66b6a86a31b2"}
Jul 01 20:57:08.606168 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.606069    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"
Jul 01 20:57:08.606168 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.606090    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm" event={"ID":"e28fcffa-cde8-44ce-9015-a1ad49645e7a","Type":"ContainerDied","Data":"2cf51427f9dd4e968caacec129e92cce60d7c896487aa3cf5250f116306174ff"}
Jul 01 20:57:08.606168 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.606129    2564 scope.go:117] "RemoveContainer" containerID="c87f86307ab796673411960278297376de8ceebc5ec5e55043e4d231bf36ae2d"
Jul 01 20:57:08.609163 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.609137    2564 generic.go:356] "Generic (PLEG): container finished" podID="603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" containerID="3d401e3428a1ce8e8356bed74b80db9385676c7c9aacb672aa31984e20cdecda" exitCode=0
Jul 01 20:57:08.609276 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.609184    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2" event={"ID":"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4","Type":"ContainerDied","Data":"3d401e3428a1ce8e8356bed74b80db9385676c7c9aacb672aa31984e20cdecda"}
Jul 01 20:57:08.609276 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.609199    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"
Jul 01 20:57:08.609276 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.609234    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2" event={"ID":"603c3269-3a3d-4ff6-b52a-41bb9af0d5f4","Type":"ContainerDied","Data":"6c34df51820264e97895613603e004d48607a9f28a523b48c174544e79a83a5e"}
Jul 01 20:57:08.617390 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.616959    2564 scope.go:117] "RemoveContainer" containerID="59a868890366f9236fbef38b9be005b46dac9a8616a5633b647c66b6a86a31b2"
Jul 01 20:57:08.625784 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.625769    2564 scope.go:117] "RemoveContainer" containerID="b190adb7ee23a3a02162490d6c1c7661f30eac8e67423cd3005b7734a35efbc4"
Jul 01 20:57:08.633152 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.633137    2564 scope.go:117] "RemoveContainer" containerID="c87f86307ab796673411960278297376de8ceebc5ec5e55043e4d231bf36ae2d"
Jul 01 20:57:08.633321 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.633288    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"]
Jul 01 20:57:08.633418 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:57:08.633390    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c87f86307ab796673411960278297376de8ceebc5ec5e55043e4d231bf36ae2d\": container with ID starting with c87f86307ab796673411960278297376de8ceebc5ec5e55043e4d231bf36ae2d not found: ID does not exist" containerID="c87f86307ab796673411960278297376de8ceebc5ec5e55043e4d231bf36ae2d"
Jul 01 20:57:08.633456 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.633413    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c87f86307ab796673411960278297376de8ceebc5ec5e55043e4d231bf36ae2d"} err="failed to get container status \"c87f86307ab796673411960278297376de8ceebc5ec5e55043e4d231bf36ae2d\": rpc error: code = NotFound desc = could not find container \"c87f86307ab796673411960278297376de8ceebc5ec5e55043e4d231bf36ae2d\": container with ID starting with c87f86307ab796673411960278297376de8ceebc5ec5e55043e4d231bf36ae2d not found: ID does not exist"
Jul 01 20:57:08.633456 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.633429    2564 scope.go:117] "RemoveContainer" containerID="59a868890366f9236fbef38b9be005b46dac9a8616a5633b647c66b6a86a31b2"
Jul 01 20:57:08.633666 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:57:08.633652    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59a868890366f9236fbef38b9be005b46dac9a8616a5633b647c66b6a86a31b2\": container with ID starting with 59a868890366f9236fbef38b9be005b46dac9a8616a5633b647c66b6a86a31b2 not found: ID does not exist" containerID="59a868890366f9236fbef38b9be005b46dac9a8616a5633b647c66b6a86a31b2"
Jul 01 20:57:08.633708 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.633672    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59a868890366f9236fbef38b9be005b46dac9a8616a5633b647c66b6a86a31b2"} err="failed to get container status \"59a868890366f9236fbef38b9be005b46dac9a8616a5633b647c66b6a86a31b2\": rpc error: code = NotFound desc = could not find container \"59a868890366f9236fbef38b9be005b46dac9a8616a5633b647c66b6a86a31b2\": container with ID starting with 59a868890366f9236fbef38b9be005b46dac9a8616a5633b647c66b6a86a31b2 not found: ID does not exist"
Jul 01 20:57:08.633708 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.633691    2564 scope.go:117] "RemoveContainer" containerID="b190adb7ee23a3a02162490d6c1c7661f30eac8e67423cd3005b7734a35efbc4"
Jul 01 20:57:08.633933 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:57:08.633917    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b190adb7ee23a3a02162490d6c1c7661f30eac8e67423cd3005b7734a35efbc4\": container with ID starting with b190adb7ee23a3a02162490d6c1c7661f30eac8e67423cd3005b7734a35efbc4 not found: ID does not exist" containerID="b190adb7ee23a3a02162490d6c1c7661f30eac8e67423cd3005b7734a35efbc4"
Jul 01 20:57:08.633977 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.633937    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b190adb7ee23a3a02162490d6c1c7661f30eac8e67423cd3005b7734a35efbc4"} err="failed to get container status \"b190adb7ee23a3a02162490d6c1c7661f30eac8e67423cd3005b7734a35efbc4\": rpc error: code = NotFound desc = could not find container \"b190adb7ee23a3a02162490d6c1c7661f30eac8e67423cd3005b7734a35efbc4\": container with ID starting with b190adb7ee23a3a02162490d6c1c7661f30eac8e67423cd3005b7734a35efbc4 not found: ID does not exist"
Jul 01 20:57:08.633977 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.633955    2564 scope.go:117] "RemoveContainer" containerID="2a2725f6ed0e1d310b56c79cee0dade4d1ea369da43b45f80addda04ea41b2a5"
Jul 01 20:57:08.636811 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.636792    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6f5c857d79-5p5g2"]
Jul 01 20:57:08.640949 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.640928    2564 scope.go:117] "RemoveContainer" containerID="3d401e3428a1ce8e8356bed74b80db9385676c7c9aacb672aa31984e20cdecda"
Jul 01 20:57:08.648269 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.648250    2564 scope.go:117] "RemoveContainer" containerID="9fde6b9870a9cef77f324602ad181211a171b23de25631cb9bbcc90b9ed43883"
Jul 01 20:57:08.648843 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.648823    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"]
Jul 01 20:57:08.651356 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.651338    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-6fb9776d89-npwbm"]
Jul 01 20:57:08.655180 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.655164    2564 scope.go:117] "RemoveContainer" containerID="2a2725f6ed0e1d310b56c79cee0dade4d1ea369da43b45f80addda04ea41b2a5"
Jul 01 20:57:08.655402 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:57:08.655386    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a2725f6ed0e1d310b56c79cee0dade4d1ea369da43b45f80addda04ea41b2a5\": container with ID starting with 2a2725f6ed0e1d310b56c79cee0dade4d1ea369da43b45f80addda04ea41b2a5 not found: ID does not exist" containerID="2a2725f6ed0e1d310b56c79cee0dade4d1ea369da43b45f80addda04ea41b2a5"
Jul 01 20:57:08.655471 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.655410    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a2725f6ed0e1d310b56c79cee0dade4d1ea369da43b45f80addda04ea41b2a5"} err="failed to get container status \"2a2725f6ed0e1d310b56c79cee0dade4d1ea369da43b45f80addda04ea41b2a5\": rpc error: code = NotFound desc = could not find container \"2a2725f6ed0e1d310b56c79cee0dade4d1ea369da43b45f80addda04ea41b2a5\": container with ID starting with 2a2725f6ed0e1d310b56c79cee0dade4d1ea369da43b45f80addda04ea41b2a5 not found: ID does not exist"
Jul 01 20:57:08.655471 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.655431    2564 scope.go:117] "RemoveContainer" containerID="3d401e3428a1ce8e8356bed74b80db9385676c7c9aacb672aa31984e20cdecda"
Jul 01 20:57:08.655661 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:57:08.655647    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d401e3428a1ce8e8356bed74b80db9385676c7c9aacb672aa31984e20cdecda\": container with ID starting with 3d401e3428a1ce8e8356bed74b80db9385676c7c9aacb672aa31984e20cdecda not found: ID does not exist" containerID="3d401e3428a1ce8e8356bed74b80db9385676c7c9aacb672aa31984e20cdecda"
Jul 01 20:57:08.655722 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.655668    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d401e3428a1ce8e8356bed74b80db9385676c7c9aacb672aa31984e20cdecda"} err="failed to get container status \"3d401e3428a1ce8e8356bed74b80db9385676c7c9aacb672aa31984e20cdecda\": rpc error: code = NotFound desc = could not find container \"3d401e3428a1ce8e8356bed74b80db9385676c7c9aacb672aa31984e20cdecda\": container with ID starting with 3d401e3428a1ce8e8356bed74b80db9385676c7c9aacb672aa31984e20cdecda not found: ID does not exist"
Jul 01 20:57:08.655722 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.655688    2564 scope.go:117] "RemoveContainer" containerID="9fde6b9870a9cef77f324602ad181211a171b23de25631cb9bbcc90b9ed43883"
Jul 01 20:57:08.655956 ip-10-0-131-198 kubenswrapper[2564]: E0701 20:57:08.655939    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fde6b9870a9cef77f324602ad181211a171b23de25631cb9bbcc90b9ed43883\": container with ID starting with 9fde6b9870a9cef77f324602ad181211a171b23de25631cb9bbcc90b9ed43883 not found: ID does not exist" containerID="9fde6b9870a9cef77f324602ad181211a171b23de25631cb9bbcc90b9ed43883"
Jul 01 20:57:08.656006 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.655962    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fde6b9870a9cef77f324602ad181211a171b23de25631cb9bbcc90b9ed43883"} err="failed to get container status \"9fde6b9870a9cef77f324602ad181211a171b23de25631cb9bbcc90b9ed43883\": rpc error: code = NotFound desc = could not find container \"9fde6b9870a9cef77f324602ad181211a171b23de25631cb9bbcc90b9ed43883\": container with ID starting with 9fde6b9870a9cef77f324602ad181211a171b23de25631cb9bbcc90b9ed43883 not found: ID does not exist"
Jul 01 20:57:08.670475 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.670459    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:57:08.670540 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.670477    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e28fcffa-cde8-44ce-9015-a1ad49645e7a-cabundle-cert\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:57:08.670540 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.670486    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e28fcffa-cde8-44ce-9015-a1ad49645e7a-proxy-tls\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:57:08.670540 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.670494    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-proxy-tls\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:57:08.670540 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.670502    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mfc2\" (UniqueName: \"kubernetes.io/projected/e28fcffa-cde8-44ce-9015-a1ad49645e7a-kube-api-access-8mfc2\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:57:08.670540 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.670509    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdb9h\" (UniqueName: \"kubernetes.io/projected/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4-kube-api-access-wdb9h\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:57:08.670540 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.670518    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e28fcffa-cde8-44ce-9015-a1ad49645e7a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 20:57:08.763610 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.763580    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="140bfc0f-a451-4a3b-9acc-1fdcde1702e5" path="/var/lib/kubelet/pods/140bfc0f-a451-4a3b-9acc-1fdcde1702e5/volumes"
Jul 01 20:57:08.764084 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.764070    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" path="/var/lib/kubelet/pods/603c3269-3a3d-4ff6-b52a-41bb9af0d5f4/volumes"
Jul 01 20:57:08.764866 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:08.764842    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e28fcffa-cde8-44ce-9015-a1ad49645e7a" path="/var/lib/kubelet/pods/e28fcffa-cde8-44ce-9015-a1ad49645e7a/volumes"
Jul 01 20:57:09.438538 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:09.438502    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4" podUID="447b6d3c-1be1-4053-91db-e8f11d74445f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 01 20:57:09.439108 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:09.439082    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7" podUID="72b059bc-a318-419f-b477-f0d1b982dcac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 01 20:57:09.584502 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:09.584422    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7"
Jul 01 20:57:09.584823 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:09.584793    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7" podUID="361e65bc-14bb-4657-a4ea-85dedccfc845" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 01 20:57:10.589590 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:10.589560    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2"
Jul 01 20:57:10.590009 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:10.589980    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2" podUID="e1392569-add7-437e-b0d7-ef1ad3b24949" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 01 20:57:19.438589 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:19.438544    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4" podUID="447b6d3c-1be1-4053-91db-e8f11d74445f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 01 20:57:19.439570 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:19.439547    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7"
Jul 01 20:57:19.584849 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:19.584806    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7" podUID="361e65bc-14bb-4657-a4ea-85dedccfc845" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 01 20:57:20.590545 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:20.590505    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2" podUID="e1392569-add7-437e-b0d7-ef1ad3b24949" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 01 20:57:29.439052 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:29.439021    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4"
Jul 01 20:57:29.585675 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:29.585632    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7" podUID="361e65bc-14bb-4657-a4ea-85dedccfc845" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 01 20:57:30.590015 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:30.589971    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2" podUID="e1392569-add7-437e-b0d7-ef1ad3b24949" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 01 20:57:39.585051 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:39.585012    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7" podUID="361e65bc-14bb-4657-a4ea-85dedccfc845" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 01 20:57:40.590910 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:40.590852    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2" podUID="e1392569-add7-437e-b0d7-ef1ad3b24949" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 01 20:57:49.585027 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:49.584994    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7"
Jul 01 20:57:50.590845 ip-10-0-131-198 kubenswrapper[2564]: I0701 20:57:50.590810    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2"
Jul 01 21:01:40.765079 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:01:40.765002    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/ovn-acl-logging/0.log"
Jul 01 21:01:40.768499 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:01:40.768478    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/ovn-acl-logging/0.log"
Jul 01 21:05:57.345813 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.345779    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7"]
Jul 01 21:05:57.348316 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.346174    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" containerName="kserve-container"
Jul 01 21:05:57.348316 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.346189    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" containerName="kserve-container"
Jul 01 21:05:57.348316 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.346201    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" containerName="kube-rbac-proxy"
Jul 01 21:05:57.348316 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.346208    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" containerName="kube-rbac-proxy"
Jul 01 21:05:57.348316 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.346218    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="140bfc0f-a451-4a3b-9acc-1fdcde1702e5" containerName="storage-initializer"
Jul 01 21:05:57.348316 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.346224    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="140bfc0f-a451-4a3b-9acc-1fdcde1702e5" containerName="storage-initializer"
Jul 01 21:05:57.348316 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.346236    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="140bfc0f-a451-4a3b-9acc-1fdcde1702e5" containerName="kube-rbac-proxy"
Jul 01 21:05:57.348316 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.346244    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="140bfc0f-a451-4a3b-9acc-1fdcde1702e5" containerName="kube-rbac-proxy"
Jul 01 21:05:57.348316 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.346256    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" containerName="storage-initializer"
Jul 01 21:05:57.348316 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.346262    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" containerName="storage-initializer"
Jul 01 21:05:57.348316 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.346272    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e28fcffa-cde8-44ce-9015-a1ad49645e7a" containerName="storage-initializer"
Jul 01 21:05:57.348316 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.346277    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e28fcffa-cde8-44ce-9015-a1ad49645e7a" containerName="storage-initializer"
Jul 01 21:05:57.348316 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.346283    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e28fcffa-cde8-44ce-9015-a1ad49645e7a" containerName="kserve-container"
Jul 01 21:05:57.348316 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.346288    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e28fcffa-cde8-44ce-9015-a1ad49645e7a" containerName="kserve-container"
Jul 01 21:05:57.348316 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.346293    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e28fcffa-cde8-44ce-9015-a1ad49645e7a" containerName="kube-rbac-proxy"
Jul 01 21:05:57.348316 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.346298    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e28fcffa-cde8-44ce-9015-a1ad49645e7a" containerName="kube-rbac-proxy"
Jul 01 21:05:57.348316 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.346304    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="140bfc0f-a451-4a3b-9acc-1fdcde1702e5" containerName="kserve-container"
Jul 01 21:05:57.348316 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.346308    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="140bfc0f-a451-4a3b-9acc-1fdcde1702e5" containerName="kserve-container"
Jul 01 21:05:57.348316 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.346355    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" containerName="kserve-container"
Jul 01 21:05:57.348316 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.346364    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="603c3269-3a3d-4ff6-b52a-41bb9af0d5f4" containerName="kube-rbac-proxy"
Jul 01 21:05:57.348316 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.346371    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e28fcffa-cde8-44ce-9015-a1ad49645e7a" containerName="kube-rbac-proxy"
Jul 01 21:05:57.348316 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.346377    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e28fcffa-cde8-44ce-9015-a1ad49645e7a" containerName="kserve-container"
Jul 01 21:05:57.348316 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.346383    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="140bfc0f-a451-4a3b-9acc-1fdcde1702e5" containerName="kserve-container"
Jul 01 21:05:57.348316 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.346397    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="140bfc0f-a451-4a3b-9acc-1fdcde1702e5" containerName="kube-rbac-proxy"
Jul 01 21:05:57.349258 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.349242    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7"
Jul 01 21:05:57.351908 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.351886    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f1d61-kube-rbac-proxy-sar-config\""
Jul 01 21:05:57.351908 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.351897    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f1d61-predictor-serving-cert\""
Jul 01 21:05:57.357789 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.357767    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7"]
Jul 01 21:05:57.360613 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.360593    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4"]
Jul 01 21:05:57.360894 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.360833    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4" podUID="447b6d3c-1be1-4053-91db-e8f11d74445f" containerName="kserve-container" containerID="cri-o://1d9d5f1b4f5f325efe6edffe1f018414d9996261a0fc4b4045b7d1f8d3c2f9ee" gracePeriod=30
Jul 01 21:05:57.360948 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.360905    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4" podUID="447b6d3c-1be1-4053-91db-e8f11d74445f" containerName="kube-rbac-proxy" containerID="cri-o://ee7eafdb90be56234d212e6fc44ec204d330761cf3e25401782dfdbf0e823e7b" gracePeriod=30
Jul 01 21:05:57.369987 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.369851    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8dv4\" (UniqueName: \"kubernetes.io/projected/0f7441a7-bb69-4d68-8073-98c87a63a34d-kube-api-access-l8dv4\") pod \"success-200-isvc-f1d61-predictor-69d9f6659c-rznj7\" (UID: \"0f7441a7-bb69-4d68-8073-98c87a63a34d\") " pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7"
Jul 01 21:05:57.370114 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.370092    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f7441a7-bb69-4d68-8073-98c87a63a34d-proxy-tls\") pod \"success-200-isvc-f1d61-predictor-69d9f6659c-rznj7\" (UID: \"0f7441a7-bb69-4d68-8073-98c87a63a34d\") " pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7"
Jul 01 21:05:57.370171 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.370124    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f7441a7-bb69-4d68-8073-98c87a63a34d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f1d61-predictor-69d9f6659c-rznj7\" (UID: \"0f7441a7-bb69-4d68-8073-98c87a63a34d\") " pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7"
Jul 01 21:05:57.421092 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.421070    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv"]
Jul 01 21:05:57.424406 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.424390    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv"
Jul 01 21:05:57.426938 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.426914    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f1d61-predictor-serving-cert\""
Jul 01 21:05:57.426938 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.426939    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f1d61-kube-rbac-proxy-sar-config\""
Jul 01 21:05:57.433825 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.433806    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv"]
Jul 01 21:05:57.461733 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.461706    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7"]
Jul 01 21:05:57.461945 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.461924    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7" podUID="72b059bc-a318-419f-b477-f0d1b982dcac" containerName="kserve-container" containerID="cri-o://249100f43230213f40fe82c0cf9c7ab485cdb5cf2a5458bc55575de3ab178f46" gracePeriod=30
Jul 01 21:05:57.462017 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.461950    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7" podUID="72b059bc-a318-419f-b477-f0d1b982dcac" containerName="kube-rbac-proxy" containerID="cri-o://9179fa5f317a0543c2e556631dde1a8859dae7d5bcbcfa9aaed87ea5eec6619d" gracePeriod=30
Jul 01 21:05:57.471133 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.471110    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f7441a7-bb69-4d68-8073-98c87a63a34d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f1d61-predictor-69d9f6659c-rznj7\" (UID: \"0f7441a7-bb69-4d68-8073-98c87a63a34d\") " pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7"
Jul 01 21:05:57.471241 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.471148    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fad15195-1882-4c1a-8731-cc759e2831db-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f1d61-predictor-7896dd4c76-6snkv\" (UID: \"fad15195-1882-4c1a-8731-cc759e2831db\") " pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv"
Jul 01 21:05:57.471241 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.471211    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8dv4\" (UniqueName: \"kubernetes.io/projected/0f7441a7-bb69-4d68-8073-98c87a63a34d-kube-api-access-l8dv4\") pod \"success-200-isvc-f1d61-predictor-69d9f6659c-rznj7\" (UID: \"0f7441a7-bb69-4d68-8073-98c87a63a34d\") " pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7"
Jul 01 21:05:57.471356 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.471255    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbrr2\" (UniqueName: \"kubernetes.io/projected/fad15195-1882-4c1a-8731-cc759e2831db-kube-api-access-bbrr2\") pod \"error-404-isvc-f1d61-predictor-7896dd4c76-6snkv\" (UID: \"fad15195-1882-4c1a-8731-cc759e2831db\") " pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv"
Jul 01 21:05:57.471416 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.471392    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fad15195-1882-4c1a-8731-cc759e2831db-proxy-tls\") pod \"error-404-isvc-f1d61-predictor-7896dd4c76-6snkv\" (UID: \"fad15195-1882-4c1a-8731-cc759e2831db\") " pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv"
Jul 01 21:05:57.471510 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.471494    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f7441a7-bb69-4d68-8073-98c87a63a34d-proxy-tls\") pod \"success-200-isvc-f1d61-predictor-69d9f6659c-rznj7\" (UID: \"0f7441a7-bb69-4d68-8073-98c87a63a34d\") " pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7"
Jul 01 21:05:57.471753 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.471734    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f7441a7-bb69-4d68-8073-98c87a63a34d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f1d61-predictor-69d9f6659c-rznj7\" (UID: \"0f7441a7-bb69-4d68-8073-98c87a63a34d\") " pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7"
Jul 01 21:05:57.474022 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.474000    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f7441a7-bb69-4d68-8073-98c87a63a34d-proxy-tls\") pod \"success-200-isvc-f1d61-predictor-69d9f6659c-rznj7\" (UID: \"0f7441a7-bb69-4d68-8073-98c87a63a34d\") " pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7"
Jul 01 21:05:57.482727 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.482709    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8dv4\" (UniqueName: \"kubernetes.io/projected/0f7441a7-bb69-4d68-8073-98c87a63a34d-kube-api-access-l8dv4\") pod \"success-200-isvc-f1d61-predictor-69d9f6659c-rznj7\" (UID: \"0f7441a7-bb69-4d68-8073-98c87a63a34d\") " pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7"
Jul 01 21:05:57.572196 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.572156    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fad15195-1882-4c1a-8731-cc759e2831db-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f1d61-predictor-7896dd4c76-6snkv\" (UID: \"fad15195-1882-4c1a-8731-cc759e2831db\") " pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv"
Jul 01 21:05:57.572366 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.572232    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbrr2\" (UniqueName: \"kubernetes.io/projected/fad15195-1882-4c1a-8731-cc759e2831db-kube-api-access-bbrr2\") pod \"error-404-isvc-f1d61-predictor-7896dd4c76-6snkv\" (UID: \"fad15195-1882-4c1a-8731-cc759e2831db\") " pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv"
Jul 01 21:05:57.572366 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.572262    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fad15195-1882-4c1a-8731-cc759e2831db-proxy-tls\") pod \"error-404-isvc-f1d61-predictor-7896dd4c76-6snkv\" (UID: \"fad15195-1882-4c1a-8731-cc759e2831db\") " pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv"
Jul 01 21:05:57.572366 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:05:57.572356    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-serving-cert: secret "error-404-isvc-f1d61-predictor-serving-cert" not found
Jul 01 21:05:57.572512 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:05:57.572431    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fad15195-1882-4c1a-8731-cc759e2831db-proxy-tls podName:fad15195-1882-4c1a-8731-cc759e2831db nodeName:}" failed. No retries permitted until 2026-07-01 21:05:58.072413018 +0000 UTC m=+1157.748428564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fad15195-1882-4c1a-8731-cc759e2831db-proxy-tls") pod "error-404-isvc-f1d61-predictor-7896dd4c76-6snkv" (UID: "fad15195-1882-4c1a-8731-cc759e2831db") : secret "error-404-isvc-f1d61-predictor-serving-cert" not found
Jul 01 21:05:57.572887 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.572853    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fad15195-1882-4c1a-8731-cc759e2831db-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f1d61-predictor-7896dd4c76-6snkv\" (UID: \"fad15195-1882-4c1a-8731-cc759e2831db\") " pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv"
Jul 01 21:05:57.582793 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.582769    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbrr2\" (UniqueName: \"kubernetes.io/projected/fad15195-1882-4c1a-8731-cc759e2831db-kube-api-access-bbrr2\") pod \"error-404-isvc-f1d61-predictor-7896dd4c76-6snkv\" (UID: \"fad15195-1882-4c1a-8731-cc759e2831db\") " pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv"
Jul 01 21:05:57.660621 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.660581    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7"
Jul 01 21:05:57.694004 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.693968    2564 generic.go:356] "Generic (PLEG): container finished" podID="447b6d3c-1be1-4053-91db-e8f11d74445f" containerID="ee7eafdb90be56234d212e6fc44ec204d330761cf3e25401782dfdbf0e823e7b" exitCode=2
Jul 01 21:05:57.694170 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.694044    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4" event={"ID":"447b6d3c-1be1-4053-91db-e8f11d74445f","Type":"ContainerDied","Data":"ee7eafdb90be56234d212e6fc44ec204d330761cf3e25401782dfdbf0e823e7b"}
Jul 01 21:05:57.696101 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.696067    2564 generic.go:356] "Generic (PLEG): container finished" podID="72b059bc-a318-419f-b477-f0d1b982dcac" containerID="9179fa5f317a0543c2e556631dde1a8859dae7d5bcbcfa9aaed87ea5eec6619d" exitCode=2
Jul 01 21:05:57.696233 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.696110    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7" event={"ID":"72b059bc-a318-419f-b477-f0d1b982dcac","Type":"ContainerDied","Data":"9179fa5f317a0543c2e556631dde1a8859dae7d5bcbcfa9aaed87ea5eec6619d"}
Jul 01 21:05:57.792194 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:57.792172    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7"]
Jul 01 21:05:57.794807 ip-10-0-131-198 kubenswrapper[2564]: W0701 21:05:57.794781    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f7441a7_bb69_4d68_8073_98c87a63a34d.slice/crio-751ac7ce3d14489cf4b8975ea5cdb7c8bde97a150771aa46a3a51ed5535aa8a1 WatchSource:0}: Error finding container 751ac7ce3d14489cf4b8975ea5cdb7c8bde97a150771aa46a3a51ed5535aa8a1: Status 404 returned error can't find the container with id 751ac7ce3d14489cf4b8975ea5cdb7c8bde97a150771aa46a3a51ed5535aa8a1
Jul 01 21:05:58.075581 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:58.075493    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fad15195-1882-4c1a-8731-cc759e2831db-proxy-tls\") pod \"error-404-isvc-f1d61-predictor-7896dd4c76-6snkv\" (UID: \"fad15195-1882-4c1a-8731-cc759e2831db\") " pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv"
Jul 01 21:05:58.077785 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:58.077767    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fad15195-1882-4c1a-8731-cc759e2831db-proxy-tls\") pod \"error-404-isvc-f1d61-predictor-7896dd4c76-6snkv\" (UID: \"fad15195-1882-4c1a-8731-cc759e2831db\") " pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv"
Jul 01 21:05:58.334350 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:58.334265    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv"
Jul 01 21:05:58.464255 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:58.463935    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv"]
Jul 01 21:05:58.466599 ip-10-0-131-198 kubenswrapper[2564]: W0701 21:05:58.466573    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad15195_1882_4c1a_8731_cc759e2831db.slice/crio-a1ae7dfa76fd845a801c589510e532fb2dd0224553aa233d838cb1b93256a684 WatchSource:0}: Error finding container a1ae7dfa76fd845a801c589510e532fb2dd0224553aa233d838cb1b93256a684: Status 404 returned error can't find the container with id a1ae7dfa76fd845a801c589510e532fb2dd0224553aa233d838cb1b93256a684
Jul 01 21:05:58.705116 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:58.705067    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7" event={"ID":"0f7441a7-bb69-4d68-8073-98c87a63a34d","Type":"ContainerStarted","Data":"1c6e42ee6db025b915957fa5ba00e535c8810edb360749727aa0f07249f4ac0c"}
Jul 01 21:05:58.705376 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:58.705349    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7"
Jul 01 21:05:58.705540 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:58.705528    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7"
Jul 01 21:05:58.705683 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:58.705667    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7" event={"ID":"0f7441a7-bb69-4d68-8073-98c87a63a34d","Type":"ContainerStarted","Data":"9f9c11efa0728cab31032bd14d178b62ceeb6fb1e8ccf520ae8464b495b3a66a"}
Jul 01 21:05:58.705812 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:58.705797    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7" event={"ID":"0f7441a7-bb69-4d68-8073-98c87a63a34d","Type":"ContainerStarted","Data":"751ac7ce3d14489cf4b8975ea5cdb7c8bde97a150771aa46a3a51ed5535aa8a1"}
Jul 01 21:05:58.706726 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:58.706702    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7" podUID="0f7441a7-bb69-4d68-8073-98c87a63a34d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 01 21:05:58.707442 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:58.707407    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv" event={"ID":"fad15195-1882-4c1a-8731-cc759e2831db","Type":"ContainerStarted","Data":"3305d2f7af26ea78ed9a562e937462413e091f09f8253e43d3a6faa283ff36c5"}
Jul 01 21:05:58.707563 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:58.707459    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv"
Jul 01 21:05:58.707563 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:58.707488    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv"
Jul 01 21:05:58.707563 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:58.707508    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv" event={"ID":"fad15195-1882-4c1a-8731-cc759e2831db","Type":"ContainerStarted","Data":"e66a394dadc0a8b57ec2024df01560e15f45dfc75f9b045c0681763dc2c9668c"}
Jul 01 21:05:58.707563 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:58.707526    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv" event={"ID":"fad15195-1882-4c1a-8731-cc759e2831db","Type":"ContainerStarted","Data":"a1ae7dfa76fd845a801c589510e532fb2dd0224553aa233d838cb1b93256a684"}
Jul 01 21:05:58.708569 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:58.708540    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv" podUID="fad15195-1882-4c1a-8731-cc759e2831db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 01 21:05:58.724335 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:58.724296    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7" podStartSLOduration=1.7242829039999998 podStartE2EDuration="1.724282904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:05:58.72297873 +0000 UTC m=+1158.398994297" watchObservedRunningTime="2026-07-01 21:05:58.724282904 +0000 UTC m=+1158.400298531"
Jul 01 21:05:58.740468 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:58.740429    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv" podStartSLOduration=1.740418357 podStartE2EDuration="1.740418357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:05:58.739432977 +0000 UTC m=+1158.415448545" watchObservedRunningTime="2026-07-01 21:05:58.740418357 +0000 UTC m=+1158.416433924"
Jul 01 21:05:59.434484 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:59.434443    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7" podUID="72b059bc-a318-419f-b477-f0d1b982dcac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 01 21:05:59.434667 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:59.434487    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4" podUID="447b6d3c-1be1-4053-91db-e8f11d74445f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 01 21:05:59.438834 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:59.438809    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4" podUID="447b6d3c-1be1-4053-91db-e8f11d74445f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 01 21:05:59.440036 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:59.440018    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7" podUID="72b059bc-a318-419f-b477-f0d1b982dcac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 01 21:05:59.712551 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:59.712445    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv" podUID="fad15195-1882-4c1a-8731-cc759e2831db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 01 21:05:59.713064 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:05:59.712668    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7" podUID="0f7441a7-bb69-4d68-8073-98c87a63a34d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 01 21:06:00.613779 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:00.613758    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4"
Jul 01 21:06:00.701996 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:00.701912    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/447b6d3c-1be1-4053-91db-e8f11d74445f-kube-rbac-proxy-sar-config\") pod \"447b6d3c-1be1-4053-91db-e8f11d74445f\" (UID: \"447b6d3c-1be1-4053-91db-e8f11d74445f\") "
Jul 01 21:06:00.701996 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:00.701951    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6b95\" (UniqueName: \"kubernetes.io/projected/447b6d3c-1be1-4053-91db-e8f11d74445f-kube-api-access-h6b95\") pod \"447b6d3c-1be1-4053-91db-e8f11d74445f\" (UID: \"447b6d3c-1be1-4053-91db-e8f11d74445f\") "
Jul 01 21:06:00.701996 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:00.701984    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/447b6d3c-1be1-4053-91db-e8f11d74445f-proxy-tls\") pod \"447b6d3c-1be1-4053-91db-e8f11d74445f\" (UID: \"447b6d3c-1be1-4053-91db-e8f11d74445f\") "
Jul 01 21:06:00.702358 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:00.702332    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/447b6d3c-1be1-4053-91db-e8f11d74445f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "447b6d3c-1be1-4053-91db-e8f11d74445f" (UID: "447b6d3c-1be1-4053-91db-e8f11d74445f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:06:00.704507 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:00.704467    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/447b6d3c-1be1-4053-91db-e8f11d74445f-kube-api-access-h6b95" (OuterVolumeSpecName: "kube-api-access-h6b95") pod "447b6d3c-1be1-4053-91db-e8f11d74445f" (UID: "447b6d3c-1be1-4053-91db-e8f11d74445f"). InnerVolumeSpecName "kube-api-access-h6b95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:06:00.704824 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:00.704794    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/447b6d3c-1be1-4053-91db-e8f11d74445f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "447b6d3c-1be1-4053-91db-e8f11d74445f" (UID: "447b6d3c-1be1-4053-91db-e8f11d74445f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:06:00.718368 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:00.718343    2564 generic.go:356] "Generic (PLEG): container finished" podID="447b6d3c-1be1-4053-91db-e8f11d74445f" containerID="1d9d5f1b4f5f325efe6edffe1f018414d9996261a0fc4b4045b7d1f8d3c2f9ee" exitCode=0
Jul 01 21:06:00.718657 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:00.718390    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4"
Jul 01 21:06:00.718657 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:00.718433    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4" event={"ID":"447b6d3c-1be1-4053-91db-e8f11d74445f","Type":"ContainerDied","Data":"1d9d5f1b4f5f325efe6edffe1f018414d9996261a0fc4b4045b7d1f8d3c2f9ee"}
Jul 01 21:06:00.718657 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:00.718499    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4" event={"ID":"447b6d3c-1be1-4053-91db-e8f11d74445f","Type":"ContainerDied","Data":"409ceb70ccd5b5e5e37b8000fa09cfe6a89dd963a4c437edb179f376af93cfbe"}
Jul 01 21:06:00.718657 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:00.718531    2564 scope.go:117] "RemoveContainer" containerID="ee7eafdb90be56234d212e6fc44ec204d330761cf3e25401782dfdbf0e823e7b"
Jul 01 21:06:00.719132 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:00.719090    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7" podUID="0f7441a7-bb69-4d68-8073-98c87a63a34d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 01 21:06:00.727594 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:00.727577    2564 scope.go:117] "RemoveContainer" containerID="1d9d5f1b4f5f325efe6edffe1f018414d9996261a0fc4b4045b7d1f8d3c2f9ee"
Jul 01 21:06:00.734516 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:00.734499    2564 scope.go:117] "RemoveContainer" containerID="ee7eafdb90be56234d212e6fc44ec204d330761cf3e25401782dfdbf0e823e7b"
Jul 01 21:06:00.734748 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:06:00.734731    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee7eafdb90be56234d212e6fc44ec204d330761cf3e25401782dfdbf0e823e7b\": container with ID starting with ee7eafdb90be56234d212e6fc44ec204d330761cf3e25401782dfdbf0e823e7b not found: ID does not exist" containerID="ee7eafdb90be56234d212e6fc44ec204d330761cf3e25401782dfdbf0e823e7b"
Jul 01 21:06:00.734794 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:00.734756    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee7eafdb90be56234d212e6fc44ec204d330761cf3e25401782dfdbf0e823e7b"} err="failed to get container status \"ee7eafdb90be56234d212e6fc44ec204d330761cf3e25401782dfdbf0e823e7b\": rpc error: code = NotFound desc = could not find container \"ee7eafdb90be56234d212e6fc44ec204d330761cf3e25401782dfdbf0e823e7b\": container with ID starting with ee7eafdb90be56234d212e6fc44ec204d330761cf3e25401782dfdbf0e823e7b not found: ID does not exist"
Jul 01 21:06:00.734794 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:00.734771    2564 scope.go:117] "RemoveContainer" containerID="1d9d5f1b4f5f325efe6edffe1f018414d9996261a0fc4b4045b7d1f8d3c2f9ee"
Jul 01 21:06:00.735022 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:06:00.735007    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d9d5f1b4f5f325efe6edffe1f018414d9996261a0fc4b4045b7d1f8d3c2f9ee\": container with ID starting with 1d9d5f1b4f5f325efe6edffe1f018414d9996261a0fc4b4045b7d1f8d3c2f9ee not found: ID does not exist" containerID="1d9d5f1b4f5f325efe6edffe1f018414d9996261a0fc4b4045b7d1f8d3c2f9ee"
Jul 01 21:06:00.735084 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:00.735027    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d9d5f1b4f5f325efe6edffe1f018414d9996261a0fc4b4045b7d1f8d3c2f9ee"} err="failed to get container status \"1d9d5f1b4f5f325efe6edffe1f018414d9996261a0fc4b4045b7d1f8d3c2f9ee\": rpc error: code = NotFound desc = could not find container \"1d9d5f1b4f5f325efe6edffe1f018414d9996261a0fc4b4045b7d1f8d3c2f9ee\": container with ID starting with 1d9d5f1b4f5f325efe6edffe1f018414d9996261a0fc4b4045b7d1f8d3c2f9ee not found: ID does not exist"
Jul 01 21:06:00.739777 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:00.739759    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4"]
Jul 01 21:06:00.743221 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:00.743202    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f347e-predictor-9c7897d59-775b4"]
Jul 01 21:06:00.758982 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:00.758962    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="447b6d3c-1be1-4053-91db-e8f11d74445f" path="/var/lib/kubelet/pods/447b6d3c-1be1-4053-91db-e8f11d74445f/volumes"
Jul 01 21:06:00.803554 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:00.803521    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/447b6d3c-1be1-4053-91db-e8f11d74445f-proxy-tls\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:06:00.803554 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:00.803548    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/447b6d3c-1be1-4053-91db-e8f11d74445f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:06:00.803554 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:00.803559    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6b95\" (UniqueName: \"kubernetes.io/projected/447b6d3c-1be1-4053-91db-e8f11d74445f-kube-api-access-h6b95\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:06:01.298320 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:01.298296    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7"
Jul 01 21:06:01.409724 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:01.409697    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72b059bc-a318-419f-b477-f0d1b982dcac-kube-rbac-proxy-sar-config\") pod \"72b059bc-a318-419f-b477-f0d1b982dcac\" (UID: \"72b059bc-a318-419f-b477-f0d1b982dcac\") "
Jul 01 21:06:01.409849 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:01.409796    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72b059bc-a318-419f-b477-f0d1b982dcac-proxy-tls\") pod \"72b059bc-a318-419f-b477-f0d1b982dcac\" (UID: \"72b059bc-a318-419f-b477-f0d1b982dcac\") "
Jul 01 21:06:01.409849 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:01.409825    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8c7v\" (UniqueName: \"kubernetes.io/projected/72b059bc-a318-419f-b477-f0d1b982dcac-kube-api-access-j8c7v\") pod \"72b059bc-a318-419f-b477-f0d1b982dcac\" (UID: \"72b059bc-a318-419f-b477-f0d1b982dcac\") "
Jul 01 21:06:01.410089 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:01.410063    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b059bc-a318-419f-b477-f0d1b982dcac-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "72b059bc-a318-419f-b477-f0d1b982dcac" (UID: "72b059bc-a318-419f-b477-f0d1b982dcac"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:06:01.411762 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:01.411737    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b059bc-a318-419f-b477-f0d1b982dcac-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "72b059bc-a318-419f-b477-f0d1b982dcac" (UID: "72b059bc-a318-419f-b477-f0d1b982dcac"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:06:01.411851 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:01.411828    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b059bc-a318-419f-b477-f0d1b982dcac-kube-api-access-j8c7v" (OuterVolumeSpecName: "kube-api-access-j8c7v") pod "72b059bc-a318-419f-b477-f0d1b982dcac" (UID: "72b059bc-a318-419f-b477-f0d1b982dcac"). InnerVolumeSpecName "kube-api-access-j8c7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:06:01.511298 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:01.511258    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72b059bc-a318-419f-b477-f0d1b982dcac-proxy-tls\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:06:01.511298 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:01.511295    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8c7v\" (UniqueName: \"kubernetes.io/projected/72b059bc-a318-419f-b477-f0d1b982dcac-kube-api-access-j8c7v\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:06:01.511491 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:01.511310    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72b059bc-a318-419f-b477-f0d1b982dcac-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:06:01.724981 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:01.724896    2564 generic.go:356] "Generic (PLEG): container finished" podID="72b059bc-a318-419f-b477-f0d1b982dcac" containerID="249100f43230213f40fe82c0cf9c7ab485cdb5cf2a5458bc55575de3ab178f46" exitCode=0
Jul 01 21:06:01.724981 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:01.724937    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7" event={"ID":"72b059bc-a318-419f-b477-f0d1b982dcac","Type":"ContainerDied","Data":"249100f43230213f40fe82c0cf9c7ab485cdb5cf2a5458bc55575de3ab178f46"}
Jul 01 21:06:01.725400 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:01.724986    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7" event={"ID":"72b059bc-a318-419f-b477-f0d1b982dcac","Type":"ContainerDied","Data":"c474f79df433a7ec83f362d011931eb51f4d0cf562c19be2564687d65bc6d66c"}
Jul 01 21:06:01.725400 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:01.725001    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7"
Jul 01 21:06:01.725400 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:01.725012    2564 scope.go:117] "RemoveContainer" containerID="9179fa5f317a0543c2e556631dde1a8859dae7d5bcbcfa9aaed87ea5eec6619d"
Jul 01 21:06:01.733478 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:01.733459    2564 scope.go:117] "RemoveContainer" containerID="249100f43230213f40fe82c0cf9c7ab485cdb5cf2a5458bc55575de3ab178f46"
Jul 01 21:06:01.740737 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:01.740719    2564 scope.go:117] "RemoveContainer" containerID="9179fa5f317a0543c2e556631dde1a8859dae7d5bcbcfa9aaed87ea5eec6619d"
Jul 01 21:06:01.740986 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:06:01.740969    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9179fa5f317a0543c2e556631dde1a8859dae7d5bcbcfa9aaed87ea5eec6619d\": container with ID starting with 9179fa5f317a0543c2e556631dde1a8859dae7d5bcbcfa9aaed87ea5eec6619d not found: ID does not exist" containerID="9179fa5f317a0543c2e556631dde1a8859dae7d5bcbcfa9aaed87ea5eec6619d"
Jul 01 21:06:01.741042 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:01.740992    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9179fa5f317a0543c2e556631dde1a8859dae7d5bcbcfa9aaed87ea5eec6619d"} err="failed to get container status \"9179fa5f317a0543c2e556631dde1a8859dae7d5bcbcfa9aaed87ea5eec6619d\": rpc error: code = NotFound desc = could not find container \"9179fa5f317a0543c2e556631dde1a8859dae7d5bcbcfa9aaed87ea5eec6619d\": container with ID starting with 9179fa5f317a0543c2e556631dde1a8859dae7d5bcbcfa9aaed87ea5eec6619d not found: ID does not exist"
Jul 01 21:06:01.741042 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:01.741007    2564 scope.go:117] "RemoveContainer" containerID="249100f43230213f40fe82c0cf9c7ab485cdb5cf2a5458bc55575de3ab178f46"
Jul 01 21:06:01.741235 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:06:01.741218    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"249100f43230213f40fe82c0cf9c7ab485cdb5cf2a5458bc55575de3ab178f46\": container with ID starting with 249100f43230213f40fe82c0cf9c7ab485cdb5cf2a5458bc55575de3ab178f46 not found: ID does not exist" containerID="249100f43230213f40fe82c0cf9c7ab485cdb5cf2a5458bc55575de3ab178f46"
Jul 01 21:06:01.741276 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:01.741241    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"249100f43230213f40fe82c0cf9c7ab485cdb5cf2a5458bc55575de3ab178f46"} err="failed to get container status \"249100f43230213f40fe82c0cf9c7ab485cdb5cf2a5458bc55575de3ab178f46\": rpc error: code = NotFound desc = could not find container \"249100f43230213f40fe82c0cf9c7ab485cdb5cf2a5458bc55575de3ab178f46\": container with ID starting with 249100f43230213f40fe82c0cf9c7ab485cdb5cf2a5458bc55575de3ab178f46 not found: ID does not exist"
Jul 01 21:06:01.746949 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:01.746929    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7"]
Jul 01 21:06:01.750903 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:01.750884    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f347e-predictor-5965cb5bdb-ztwl7"]
Jul 01 21:06:02.757218 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:02.757187    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72b059bc-a318-419f-b477-f0d1b982dcac" path="/var/lib/kubelet/pods/72b059bc-a318-419f-b477-f0d1b982dcac/volumes"
Jul 01 21:06:04.716703 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:04.716674    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv"
Jul 01 21:06:04.717129 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:04.717098    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv" podUID="fad15195-1882-4c1a-8731-cc759e2831db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 01 21:06:05.722821 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:05.722794    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7"
Jul 01 21:06:05.723340 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:05.723310    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7" podUID="0f7441a7-bb69-4d68-8073-98c87a63a34d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 01 21:06:14.717555 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:14.717517    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv" podUID="fad15195-1882-4c1a-8731-cc759e2831db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 01 21:06:15.724079 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:15.724042    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7" podUID="0f7441a7-bb69-4d68-8073-98c87a63a34d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 01 21:06:17.140021 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.139993    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7"]
Jul 01 21:06:17.140433 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.140208    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7" podUID="361e65bc-14bb-4657-a4ea-85dedccfc845" containerName="kserve-container" containerID="cri-o://5514eb0a6028a7606f6e8cfd04622e4f07f10184a788a98a48833e850bc269ae" gracePeriod=30
Jul 01 21:06:17.140433 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.140274    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7" podUID="361e65bc-14bb-4657-a4ea-85dedccfc845" containerName="kube-rbac-proxy" containerID="cri-o://8a6f12f9049e30159e73bc76a5824603664ee49e1285d60663536deac642e2ac" gracePeriod=30
Jul 01 21:06:17.202621 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.202585    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v"]
Jul 01 21:06:17.203160 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.203141    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="447b6d3c-1be1-4053-91db-e8f11d74445f" containerName="kserve-container"
Jul 01 21:06:17.203220 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.203165    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="447b6d3c-1be1-4053-91db-e8f11d74445f" containerName="kserve-container"
Jul 01 21:06:17.203220 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.203193    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72b059bc-a318-419f-b477-f0d1b982dcac" containerName="kube-rbac-proxy"
Jul 01 21:06:17.203220 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.203202    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="72b059bc-a318-419f-b477-f0d1b982dcac" containerName="kube-rbac-proxy"
Jul 01 21:06:17.203320 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.203222    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72b059bc-a318-419f-b477-f0d1b982dcac" containerName="kserve-container"
Jul 01 21:06:17.203320 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.203231    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="72b059bc-a318-419f-b477-f0d1b982dcac" containerName="kserve-container"
Jul 01 21:06:17.203320 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.203246    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="447b6d3c-1be1-4053-91db-e8f11d74445f" containerName="kube-rbac-proxy"
Jul 01 21:06:17.203320 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.203254    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="447b6d3c-1be1-4053-91db-e8f11d74445f" containerName="kube-rbac-proxy"
Jul 01 21:06:17.203438 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.203336    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="72b059bc-a318-419f-b477-f0d1b982dcac" containerName="kserve-container"
Jul 01 21:06:17.203438 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.203350    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="447b6d3c-1be1-4053-91db-e8f11d74445f" containerName="kube-rbac-proxy"
Jul 01 21:06:17.203438 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.203361    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="72b059bc-a318-419f-b477-f0d1b982dcac" containerName="kube-rbac-proxy"
Jul 01 21:06:17.203438 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.203370    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="447b6d3c-1be1-4053-91db-e8f11d74445f" containerName="kserve-container"
Jul 01 21:06:17.206898 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.206879    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v"
Jul 01 21:06:17.210061 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.210033    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7cf3f-predictor-serving-cert\""
Jul 01 21:06:17.210230 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.210212    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7cf3f-kube-rbac-proxy-sar-config\""
Jul 01 21:06:17.219202 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.219172    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v"]
Jul 01 21:06:17.273925 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.273888    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2"]
Jul 01 21:06:17.274156 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.274128    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2" podUID="e1392569-add7-437e-b0d7-ef1ad3b24949" containerName="kserve-container" containerID="cri-o://c76f237315f1588e82542f92a0a401e0c9fc646abbca85be4bdb5020f1b6a95b" gracePeriod=30
Jul 01 21:06:17.274271 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.274128    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2" podUID="e1392569-add7-437e-b0d7-ef1ad3b24949" containerName="kube-rbac-proxy" containerID="cri-o://153c044199852dc9923bfa5e7ab0a8558aa0fdc217ea0878083245cd67bb43cd" gracePeriod=30
Jul 01 21:06:17.305972 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.305938    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58"]
Jul 01 21:06:17.309482 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.309466    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58"
Jul 01 21:06:17.312078 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.312057    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7cf3f-predictor-serving-cert\""
Jul 01 21:06:17.312190 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.312078    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7cf3f-kube-rbac-proxy-sar-config\""
Jul 01 21:06:17.318622 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.318598    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58"]
Jul 01 21:06:17.347419 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.347385    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c74f80f2-e534-4b04-af47-e3425926839f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v\" (UID: \"c74f80f2-e534-4b04-af47-e3425926839f\") " pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v"
Jul 01 21:06:17.347577 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.347457    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c74f80f2-e534-4b04-af47-e3425926839f-proxy-tls\") pod \"success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v\" (UID: \"c74f80f2-e534-4b04-af47-e3425926839f\") " pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v"
Jul 01 21:06:17.347577 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.347512    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtkz8\" (UniqueName: \"kubernetes.io/projected/c74f80f2-e534-4b04-af47-e3425926839f-kube-api-access-mtkz8\") pod \"success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v\" (UID: \"c74f80f2-e534-4b04-af47-e3425926839f\") " pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v"
Jul 01 21:06:17.448365 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.448329    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6993df91-d98d-4a74-ba21-84fa315d343e-proxy-tls\") pod \"error-404-isvc-7cf3f-predictor-575f8587c8-67j58\" (UID: \"6993df91-d98d-4a74-ba21-84fa315d343e\") " pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58"
Jul 01 21:06:17.448571 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.448383    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtkz8\" (UniqueName: \"kubernetes.io/projected/c74f80f2-e534-4b04-af47-e3425926839f-kube-api-access-mtkz8\") pod \"success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v\" (UID: \"c74f80f2-e534-4b04-af47-e3425926839f\") " pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v"
Jul 01 21:06:17.448571 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.448403    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6993df91-d98d-4a74-ba21-84fa315d343e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7cf3f-predictor-575f8587c8-67j58\" (UID: \"6993df91-d98d-4a74-ba21-84fa315d343e\") " pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58"
Jul 01 21:06:17.448571 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.448429    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c74f80f2-e534-4b04-af47-e3425926839f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v\" (UID: \"c74f80f2-e534-4b04-af47-e3425926839f\") " pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v"
Jul 01 21:06:17.448571 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.448465    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh6dh\" (UniqueName: \"kubernetes.io/projected/6993df91-d98d-4a74-ba21-84fa315d343e-kube-api-access-fh6dh\") pod \"error-404-isvc-7cf3f-predictor-575f8587c8-67j58\" (UID: \"6993df91-d98d-4a74-ba21-84fa315d343e\") " pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58"
Jul 01 21:06:17.448571 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.448485    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c74f80f2-e534-4b04-af47-e3425926839f-proxy-tls\") pod \"success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v\" (UID: \"c74f80f2-e534-4b04-af47-e3425926839f\") " pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v"
Jul 01 21:06:17.449096 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.449078    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c74f80f2-e534-4b04-af47-e3425926839f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v\" (UID: \"c74f80f2-e534-4b04-af47-e3425926839f\") " pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v"
Jul 01 21:06:17.450937 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.450916    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c74f80f2-e534-4b04-af47-e3425926839f-proxy-tls\") pod \"success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v\" (UID: \"c74f80f2-e534-4b04-af47-e3425926839f\") " pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v"
Jul 01 21:06:17.456010 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.455992    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtkz8\" (UniqueName: \"kubernetes.io/projected/c74f80f2-e534-4b04-af47-e3425926839f-kube-api-access-mtkz8\") pod \"success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v\" (UID: \"c74f80f2-e534-4b04-af47-e3425926839f\") " pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v"
Jul 01 21:06:17.521635 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.521595    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v"
Jul 01 21:06:17.549051 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.549020    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh6dh\" (UniqueName: \"kubernetes.io/projected/6993df91-d98d-4a74-ba21-84fa315d343e-kube-api-access-fh6dh\") pod \"error-404-isvc-7cf3f-predictor-575f8587c8-67j58\" (UID: \"6993df91-d98d-4a74-ba21-84fa315d343e\") " pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58"
Jul 01 21:06:17.549206 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.549084    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6993df91-d98d-4a74-ba21-84fa315d343e-proxy-tls\") pod \"error-404-isvc-7cf3f-predictor-575f8587c8-67j58\" (UID: \"6993df91-d98d-4a74-ba21-84fa315d343e\") " pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58"
Jul 01 21:06:17.549206 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.549125    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6993df91-d98d-4a74-ba21-84fa315d343e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7cf3f-predictor-575f8587c8-67j58\" (UID: \"6993df91-d98d-4a74-ba21-84fa315d343e\") " pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58"
Jul 01 21:06:17.549735 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.549716    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6993df91-d98d-4a74-ba21-84fa315d343e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7cf3f-predictor-575f8587c8-67j58\" (UID: \"6993df91-d98d-4a74-ba21-84fa315d343e\") " pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58"
Jul 01 21:06:17.551392 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.551342    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6993df91-d98d-4a74-ba21-84fa315d343e-proxy-tls\") pod \"error-404-isvc-7cf3f-predictor-575f8587c8-67j58\" (UID: \"6993df91-d98d-4a74-ba21-84fa315d343e\") " pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58"
Jul 01 21:06:17.557153 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.557132    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh6dh\" (UniqueName: \"kubernetes.io/projected/6993df91-d98d-4a74-ba21-84fa315d343e-kube-api-access-fh6dh\") pod \"error-404-isvc-7cf3f-predictor-575f8587c8-67j58\" (UID: \"6993df91-d98d-4a74-ba21-84fa315d343e\") " pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58"
Jul 01 21:06:17.622269 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.622223    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58"
Jul 01 21:06:17.654268 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.654193    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v"]
Jul 01 21:06:17.657110 ip-10-0-131-198 kubenswrapper[2564]: W0701 21:06:17.657077    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc74f80f2_e534_4b04_af47_e3425926839f.slice/crio-d5a613e5d8af2e25c8756fb8eec32900be9093e9115efc77524f3f9a5bdb0ade WatchSource:0}: Error finding container d5a613e5d8af2e25c8756fb8eec32900be9093e9115efc77524f3f9a5bdb0ade: Status 404 returned error can't find the container with id d5a613e5d8af2e25c8756fb8eec32900be9093e9115efc77524f3f9a5bdb0ade
Jul 01 21:06:17.764674 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.764633    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58"]
Jul 01 21:06:17.766568 ip-10-0-131-198 kubenswrapper[2564]: W0701 21:06:17.766544    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6993df91_d98d_4a74_ba21_84fa315d343e.slice/crio-d230822557dbbfe465b2a533e9258ecc765ce9e0e18899c6ff5d746c0a4d616d WatchSource:0}: Error finding container d230822557dbbfe465b2a533e9258ecc765ce9e0e18899c6ff5d746c0a4d616d: Status 404 returned error can't find the container with id d230822557dbbfe465b2a533e9258ecc765ce9e0e18899c6ff5d746c0a4d616d
Jul 01 21:06:17.788231 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.788189    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v" event={"ID":"c74f80f2-e534-4b04-af47-e3425926839f","Type":"ContainerStarted","Data":"e0d3aeda9ee342defc5abc19fb7e98e6eababd14988ea20467a6e468b19779a4"}
Jul 01 21:06:17.788348 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.788254    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v" event={"ID":"c74f80f2-e534-4b04-af47-e3425926839f","Type":"ContainerStarted","Data":"11e39449474b48b9835a9d39d98d52ef6fdd4ec5e23abb8946fbbadf776958cd"}
Jul 01 21:06:17.788348 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.788292    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v"
Jul 01 21:06:17.788467 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.788357    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v" event={"ID":"c74f80f2-e534-4b04-af47-e3425926839f","Type":"ContainerStarted","Data":"d5a613e5d8af2e25c8756fb8eec32900be9093e9115efc77524f3f9a5bdb0ade"}
Jul 01 21:06:17.788467 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.788431    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v"
Jul 01 21:06:17.789474 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.789455    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58" event={"ID":"6993df91-d98d-4a74-ba21-84fa315d343e","Type":"ContainerStarted","Data":"d230822557dbbfe465b2a533e9258ecc765ce9e0e18899c6ff5d746c0a4d616d"}
Jul 01 21:06:17.789791 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.789771    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v" podUID="c74f80f2-e534-4b04-af47-e3425926839f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 21:06:17.791088 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.791068    2564 generic.go:356] "Generic (PLEG): container finished" podID="361e65bc-14bb-4657-a4ea-85dedccfc845" containerID="8a6f12f9049e30159e73bc76a5824603664ee49e1285d60663536deac642e2ac" exitCode=2
Jul 01 21:06:17.791187 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.791126    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7" event={"ID":"361e65bc-14bb-4657-a4ea-85dedccfc845","Type":"ContainerDied","Data":"8a6f12f9049e30159e73bc76a5824603664ee49e1285d60663536deac642e2ac"}
Jul 01 21:06:17.792709 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.792689    2564 generic.go:356] "Generic (PLEG): container finished" podID="e1392569-add7-437e-b0d7-ef1ad3b24949" containerID="153c044199852dc9923bfa5e7ab0a8558aa0fdc217ea0878083245cd67bb43cd" exitCode=2
Jul 01 21:06:17.792790 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.792733    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2" event={"ID":"e1392569-add7-437e-b0d7-ef1ad3b24949","Type":"ContainerDied","Data":"153c044199852dc9923bfa5e7ab0a8558aa0fdc217ea0878083245cd67bb43cd"}
Jul 01 21:06:17.808140 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:17.808097    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v" podStartSLOduration=0.808082234 podStartE2EDuration="808.082234ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:06:17.806200615 +0000 UTC m=+1177.482216196" watchObservedRunningTime="2026-07-01 21:06:17.808082234 +0000 UTC m=+1177.484097801"
Jul 01 21:06:18.799922 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:18.799885    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58" event={"ID":"6993df91-d98d-4a74-ba21-84fa315d343e","Type":"ContainerStarted","Data":"379dcef24e957fc74c6210e800231c735000972c91114a6c81848202861ed04a"}
Jul 01 21:06:18.800535 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:18.800488    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58" event={"ID":"6993df91-d98d-4a74-ba21-84fa315d343e","Type":"ContainerStarted","Data":"2f56d26cd91e09e4ac49d33686ea8ce25a1540a3e692bc2cafcd018094863479"}
Jul 01 21:06:18.801064 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:18.801014    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v" podUID="c74f80f2-e534-4b04-af47-e3425926839f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 21:06:18.820232 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:18.820176    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58" podStartSLOduration=1.8201599929999999 podStartE2EDuration="1.820159993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:06:18.819200368 +0000 UTC m=+1178.495215937" watchObservedRunningTime="2026-07-01 21:06:18.820159993 +0000 UTC m=+1178.496175559"
Jul 01 21:06:19.581221 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:19.581180    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7" podUID="361e65bc-14bb-4657-a4ea-85dedccfc845" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 01 21:06:19.585016 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:19.584989    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7" podUID="361e65bc-14bb-4657-a4ea-85dedccfc845" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 01 21:06:19.806708 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:19.806672    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58"
Jul 01 21:06:19.807098 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:19.806734    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58"
Jul 01 21:06:19.807872 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:19.807829    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58" podUID="6993df91-d98d-4a74-ba21-84fa315d343e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 01 21:06:20.585331 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:20.585298    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2" podUID="e1392569-add7-437e-b0d7-ef1ad3b24949" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 01 21:06:20.591001 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:20.590967    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2" podUID="e1392569-add7-437e-b0d7-ef1ad3b24949" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 01 21:06:20.591108 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:20.591013    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7"
Jul 01 21:06:20.676911 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:20.676826    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xd7xj\" (UniqueName: \"kubernetes.io/projected/361e65bc-14bb-4657-a4ea-85dedccfc845-kube-api-access-xd7xj\") pod \"361e65bc-14bb-4657-a4ea-85dedccfc845\" (UID: \"361e65bc-14bb-4657-a4ea-85dedccfc845\") "
Jul 01 21:06:20.677058 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:20.676924    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/361e65bc-14bb-4657-a4ea-85dedccfc845-proxy-tls\") pod \"361e65bc-14bb-4657-a4ea-85dedccfc845\" (UID: \"361e65bc-14bb-4657-a4ea-85dedccfc845\") "
Jul 01 21:06:20.677058 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:20.676996    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/361e65bc-14bb-4657-a4ea-85dedccfc845-kube-rbac-proxy-sar-config\") pod \"361e65bc-14bb-4657-a4ea-85dedccfc845\" (UID: \"361e65bc-14bb-4657-a4ea-85dedccfc845\") "
Jul 01 21:06:20.677360 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:20.677334    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/361e65bc-14bb-4657-a4ea-85dedccfc845-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "361e65bc-14bb-4657-a4ea-85dedccfc845" (UID: "361e65bc-14bb-4657-a4ea-85dedccfc845"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:06:20.678850 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:20.678818    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/361e65bc-14bb-4657-a4ea-85dedccfc845-kube-api-access-xd7xj" (OuterVolumeSpecName: "kube-api-access-xd7xj") pod "361e65bc-14bb-4657-a4ea-85dedccfc845" (UID: "361e65bc-14bb-4657-a4ea-85dedccfc845"). InnerVolumeSpecName "kube-api-access-xd7xj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:06:20.678980 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:20.678917    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/361e65bc-14bb-4657-a4ea-85dedccfc845-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "361e65bc-14bb-4657-a4ea-85dedccfc845" (UID: "361e65bc-14bb-4657-a4ea-85dedccfc845"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:06:20.778008 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:20.777984    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/361e65bc-14bb-4657-a4ea-85dedccfc845-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:06:20.778008 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:20.778006    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd7xj\" (UniqueName: \"kubernetes.io/projected/361e65bc-14bb-4657-a4ea-85dedccfc845-kube-api-access-xd7xj\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:06:20.778148 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:20.778015    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/361e65bc-14bb-4657-a4ea-85dedccfc845-proxy-tls\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:06:20.817787 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:20.817757    2564 generic.go:356] "Generic (PLEG): container finished" podID="361e65bc-14bb-4657-a4ea-85dedccfc845" containerID="5514eb0a6028a7606f6e8cfd04622e4f07f10184a788a98a48833e850bc269ae" exitCode=0
Jul 01 21:06:20.818173 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:20.817822    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7"
Jul 01 21:06:20.818173 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:20.817843    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7" event={"ID":"361e65bc-14bb-4657-a4ea-85dedccfc845","Type":"ContainerDied","Data":"5514eb0a6028a7606f6e8cfd04622e4f07f10184a788a98a48833e850bc269ae"}
Jul 01 21:06:20.818173 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:20.817934    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7" event={"ID":"361e65bc-14bb-4657-a4ea-85dedccfc845","Type":"ContainerDied","Data":"69d351f9c86bf0426e37ed0b8fb6972de046e9a63a5456497a47c0e4acf4139c"}
Jul 01 21:06:20.818173 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:20.817962    2564 scope.go:117] "RemoveContainer" containerID="8a6f12f9049e30159e73bc76a5824603664ee49e1285d60663536deac642e2ac"
Jul 01 21:06:20.818510 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:20.818478    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58" podUID="6993df91-d98d-4a74-ba21-84fa315d343e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 01 21:06:20.826325 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:20.826308    2564 scope.go:117] "RemoveContainer" containerID="5514eb0a6028a7606f6e8cfd04622e4f07f10184a788a98a48833e850bc269ae"
Jul 01 21:06:20.833608 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:20.833593    2564 scope.go:117] "RemoveContainer" containerID="8a6f12f9049e30159e73bc76a5824603664ee49e1285d60663536deac642e2ac"
Jul 01 21:06:20.833852 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:06:20.833833    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a6f12f9049e30159e73bc76a5824603664ee49e1285d60663536deac642e2ac\": container with ID starting with 8a6f12f9049e30159e73bc76a5824603664ee49e1285d60663536deac642e2ac not found: ID does not exist" containerID="8a6f12f9049e30159e73bc76a5824603664ee49e1285d60663536deac642e2ac"
Jul 01 21:06:20.833951 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:20.833881    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a6f12f9049e30159e73bc76a5824603664ee49e1285d60663536deac642e2ac"} err="failed to get container status \"8a6f12f9049e30159e73bc76a5824603664ee49e1285d60663536deac642e2ac\": rpc error: code = NotFound desc = could not find container \"8a6f12f9049e30159e73bc76a5824603664ee49e1285d60663536deac642e2ac\": container with ID starting with 8a6f12f9049e30159e73bc76a5824603664ee49e1285d60663536deac642e2ac not found: ID does not exist"
Jul 01 21:06:20.833951 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:20.833904    2564 scope.go:117] "RemoveContainer" containerID="5514eb0a6028a7606f6e8cfd04622e4f07f10184a788a98a48833e850bc269ae"
Jul 01 21:06:20.834150 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:06:20.834134    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5514eb0a6028a7606f6e8cfd04622e4f07f10184a788a98a48833e850bc269ae\": container with ID starting with 5514eb0a6028a7606f6e8cfd04622e4f07f10184a788a98a48833e850bc269ae not found: ID does not exist" containerID="5514eb0a6028a7606f6e8cfd04622e4f07f10184a788a98a48833e850bc269ae"
Jul 01 21:06:20.834205 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:20.834157    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5514eb0a6028a7606f6e8cfd04622e4f07f10184a788a98a48833e850bc269ae"} err="failed to get container status \"5514eb0a6028a7606f6e8cfd04622e4f07f10184a788a98a48833e850bc269ae\": rpc error: code = NotFound desc = could not find container \"5514eb0a6028a7606f6e8cfd04622e4f07f10184a788a98a48833e850bc269ae\": container with ID starting with 5514eb0a6028a7606f6e8cfd04622e4f07f10184a788a98a48833e850bc269ae not found: ID does not exist"
Jul 01 21:06:20.836812 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:20.836791    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7"]
Jul 01 21:06:20.841110 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:20.841091    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a93e1-predictor-85bdd6c986-f5nv7"]
Jul 01 21:06:21.430296 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:21.430274    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2"
Jul 01 21:06:21.584791 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:21.584715    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-869xr\" (UniqueName: \"kubernetes.io/projected/e1392569-add7-437e-b0d7-ef1ad3b24949-kube-api-access-869xr\") pod \"e1392569-add7-437e-b0d7-ef1ad3b24949\" (UID: \"e1392569-add7-437e-b0d7-ef1ad3b24949\") "
Jul 01 21:06:21.584958 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:21.584815    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1392569-add7-437e-b0d7-ef1ad3b24949-proxy-tls\") pod \"e1392569-add7-437e-b0d7-ef1ad3b24949\" (UID: \"e1392569-add7-437e-b0d7-ef1ad3b24949\") "
Jul 01 21:06:21.584958 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:21.584842    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1392569-add7-437e-b0d7-ef1ad3b24949-kube-rbac-proxy-sar-config\") pod \"e1392569-add7-437e-b0d7-ef1ad3b24949\" (UID: \"e1392569-add7-437e-b0d7-ef1ad3b24949\") "
Jul 01 21:06:21.585238 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:21.585215    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1392569-add7-437e-b0d7-ef1ad3b24949-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e1392569-add7-437e-b0d7-ef1ad3b24949" (UID: "e1392569-add7-437e-b0d7-ef1ad3b24949"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:06:21.586916 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:21.586892    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1392569-add7-437e-b0d7-ef1ad3b24949-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e1392569-add7-437e-b0d7-ef1ad3b24949" (UID: "e1392569-add7-437e-b0d7-ef1ad3b24949"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:06:21.587012 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:21.586928    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1392569-add7-437e-b0d7-ef1ad3b24949-kube-api-access-869xr" (OuterVolumeSpecName: "kube-api-access-869xr") pod "e1392569-add7-437e-b0d7-ef1ad3b24949" (UID: "e1392569-add7-437e-b0d7-ef1ad3b24949"). InnerVolumeSpecName "kube-api-access-869xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:06:21.685728 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:21.685695    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1392569-add7-437e-b0d7-ef1ad3b24949-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:06:21.685728 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:21.685722    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-869xr\" (UniqueName: \"kubernetes.io/projected/e1392569-add7-437e-b0d7-ef1ad3b24949-kube-api-access-869xr\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:06:21.685728 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:21.685732    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1392569-add7-437e-b0d7-ef1ad3b24949-proxy-tls\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:06:21.824267 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:21.824233    2564 generic.go:356] "Generic (PLEG): container finished" podID="e1392569-add7-437e-b0d7-ef1ad3b24949" containerID="c76f237315f1588e82542f92a0a401e0c9fc646abbca85be4bdb5020f1b6a95b" exitCode=0
Jul 01 21:06:21.824709 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:21.824295    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2"
Jul 01 21:06:21.824709 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:21.824324    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2" event={"ID":"e1392569-add7-437e-b0d7-ef1ad3b24949","Type":"ContainerDied","Data":"c76f237315f1588e82542f92a0a401e0c9fc646abbca85be4bdb5020f1b6a95b"}
Jul 01 21:06:21.824709 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:21.824393    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2" event={"ID":"e1392569-add7-437e-b0d7-ef1ad3b24949","Type":"ContainerDied","Data":"6c273678bfbfa630cf21d58f8dac03629b32d1dff08097333559627adb8c7fcb"}
Jul 01 21:06:21.824709 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:21.824433    2564 scope.go:117] "RemoveContainer" containerID="153c044199852dc9923bfa5e7ab0a8558aa0fdc217ea0878083245cd67bb43cd"
Jul 01 21:06:21.833023 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:21.833004    2564 scope.go:117] "RemoveContainer" containerID="c76f237315f1588e82542f92a0a401e0c9fc646abbca85be4bdb5020f1b6a95b"
Jul 01 21:06:21.840084 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:21.840072    2564 scope.go:117] "RemoveContainer" containerID="153c044199852dc9923bfa5e7ab0a8558aa0fdc217ea0878083245cd67bb43cd"
Jul 01 21:06:21.840344 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:06:21.840323    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"153c044199852dc9923bfa5e7ab0a8558aa0fdc217ea0878083245cd67bb43cd\": container with ID starting with 153c044199852dc9923bfa5e7ab0a8558aa0fdc217ea0878083245cd67bb43cd not found: ID does not exist" containerID="153c044199852dc9923bfa5e7ab0a8558aa0fdc217ea0878083245cd67bb43cd"
Jul 01 21:06:21.840410 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:21.840351    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"153c044199852dc9923bfa5e7ab0a8558aa0fdc217ea0878083245cd67bb43cd"} err="failed to get container status \"153c044199852dc9923bfa5e7ab0a8558aa0fdc217ea0878083245cd67bb43cd\": rpc error: code = NotFound desc = could not find container \"153c044199852dc9923bfa5e7ab0a8558aa0fdc217ea0878083245cd67bb43cd\": container with ID starting with 153c044199852dc9923bfa5e7ab0a8558aa0fdc217ea0878083245cd67bb43cd not found: ID does not exist"
Jul 01 21:06:21.840410 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:21.840367    2564 scope.go:117] "RemoveContainer" containerID="c76f237315f1588e82542f92a0a401e0c9fc646abbca85be4bdb5020f1b6a95b"
Jul 01 21:06:21.840598 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:06:21.840580    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c76f237315f1588e82542f92a0a401e0c9fc646abbca85be4bdb5020f1b6a95b\": container with ID starting with c76f237315f1588e82542f92a0a401e0c9fc646abbca85be4bdb5020f1b6a95b not found: ID does not exist" containerID="c76f237315f1588e82542f92a0a401e0c9fc646abbca85be4bdb5020f1b6a95b"
Jul 01 21:06:21.840635 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:21.840604    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c76f237315f1588e82542f92a0a401e0c9fc646abbca85be4bdb5020f1b6a95b"} err="failed to get container status \"c76f237315f1588e82542f92a0a401e0c9fc646abbca85be4bdb5020f1b6a95b\": rpc error: code = NotFound desc = could not find container \"c76f237315f1588e82542f92a0a401e0c9fc646abbca85be4bdb5020f1b6a95b\": container with ID starting with c76f237315f1588e82542f92a0a401e0c9fc646abbca85be4bdb5020f1b6a95b not found: ID does not exist"
Jul 01 21:06:21.845960 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:21.845940    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2"]
Jul 01 21:06:21.852236 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:21.852218    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a93e1-predictor-657768547-67mn2"]
Jul 01 21:06:22.758229 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:22.758201    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="361e65bc-14bb-4657-a4ea-85dedccfc845" path="/var/lib/kubelet/pods/361e65bc-14bb-4657-a4ea-85dedccfc845/volumes"
Jul 01 21:06:22.758616 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:22.758603    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1392569-add7-437e-b0d7-ef1ad3b24949" path="/var/lib/kubelet/pods/e1392569-add7-437e-b0d7-ef1ad3b24949/volumes"
Jul 01 21:06:23.807406 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:23.807378    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v"
Jul 01 21:06:23.807833 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:23.807808    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v" podUID="c74f80f2-e534-4b04-af47-e3425926839f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 21:06:24.717750 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:24.717712    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv" podUID="fad15195-1882-4c1a-8731-cc759e2831db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 01 21:06:25.723385 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:25.723349    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7" podUID="0f7441a7-bb69-4d68-8073-98c87a63a34d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 01 21:06:25.822442 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:25.822412    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58"
Jul 01 21:06:25.822961 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:25.822930    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58" podUID="6993df91-d98d-4a74-ba21-84fa315d343e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 01 21:06:33.808628 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:33.808591    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v" podUID="c74f80f2-e534-4b04-af47-e3425926839f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 21:06:34.717227 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:34.717192    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv" podUID="fad15195-1882-4c1a-8731-cc759e2831db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 01 21:06:35.723361 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:35.723323    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7" podUID="0f7441a7-bb69-4d68-8073-98c87a63a34d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 01 21:06:35.823701 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:35.823667    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58" podUID="6993df91-d98d-4a74-ba21-84fa315d343e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 01 21:06:40.792411 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:40.792384    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/ovn-acl-logging/0.log"
Jul 01 21:06:40.793536 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:40.793519    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/ovn-acl-logging/0.log"
Jul 01 21:06:43.808638 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:43.808596    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v" podUID="c74f80f2-e534-4b04-af47-e3425926839f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 21:06:44.717901 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:44.717850    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv" podUID="fad15195-1882-4c1a-8731-cc759e2831db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 01 21:06:45.723587 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:45.723550    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7" podUID="0f7441a7-bb69-4d68-8073-98c87a63a34d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 01 21:06:45.823487 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:45.823454    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58" podUID="6993df91-d98d-4a74-ba21-84fa315d343e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 01 21:06:53.808049 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:53.808010    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v" podUID="c74f80f2-e534-4b04-af47-e3425926839f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 21:06:54.717950 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:54.717915    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv"
Jul 01 21:06:55.724011 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:55.723980    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7"
Jul 01 21:06:55.823598 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:06:55.823565    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58" podUID="6993df91-d98d-4a74-ba21-84fa315d343e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 01 21:07:03.808216 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:03.808166    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v" podUID="c74f80f2-e534-4b04-af47-e3425926839f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 21:07:05.824005 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:05.823975    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58"
Jul 01 21:07:13.808988 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:13.808959    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v"
Jul 01 21:07:17.566946 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.566914    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7"]
Jul 01 21:07:17.567351 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.567160    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7" podUID="0f7441a7-bb69-4d68-8073-98c87a63a34d" containerName="kserve-container" containerID="cri-o://9f9c11efa0728cab31032bd14d178b62ceeb6fb1e8ccf520ae8464b495b3a66a" gracePeriod=30
Jul 01 21:07:17.567351 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.567184    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7" podUID="0f7441a7-bb69-4d68-8073-98c87a63a34d" containerName="kube-rbac-proxy" containerID="cri-o://1c6e42ee6db025b915957fa5ba00e535c8810edb360749727aa0f07249f4ac0c" gracePeriod=30
Jul 01 21:07:17.583471 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.583448    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm"]
Jul 01 21:07:17.583793 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.583780    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="361e65bc-14bb-4657-a4ea-85dedccfc845" containerName="kube-rbac-proxy"
Jul 01 21:07:17.583836 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.583794    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="361e65bc-14bb-4657-a4ea-85dedccfc845" containerName="kube-rbac-proxy"
Jul 01 21:07:17.583836 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.583805    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1392569-add7-437e-b0d7-ef1ad3b24949" containerName="kserve-container"
Jul 01 21:07:17.583836 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.583810    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1392569-add7-437e-b0d7-ef1ad3b24949" containerName="kserve-container"
Jul 01 21:07:17.583836 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.583831    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="361e65bc-14bb-4657-a4ea-85dedccfc845" containerName="kserve-container"
Jul 01 21:07:17.583836 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.583836    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="361e65bc-14bb-4657-a4ea-85dedccfc845" containerName="kserve-container"
Jul 01 21:07:17.584008 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.583845    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1392569-add7-437e-b0d7-ef1ad3b24949" containerName="kube-rbac-proxy"
Jul 01 21:07:17.584008 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.583853    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1392569-add7-437e-b0d7-ef1ad3b24949" containerName="kube-rbac-proxy"
Jul 01 21:07:17.584008 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.583926    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1392569-add7-437e-b0d7-ef1ad3b24949" containerName="kserve-container"
Jul 01 21:07:17.584008 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.583935    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="361e65bc-14bb-4657-a4ea-85dedccfc845" containerName="kube-rbac-proxy"
Jul 01 21:07:17.584008 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.583942    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1392569-add7-437e-b0d7-ef1ad3b24949" containerName="kube-rbac-proxy"
Jul 01 21:07:17.584008 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.583950    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="361e65bc-14bb-4657-a4ea-85dedccfc845" containerName="kserve-container"
Jul 01 21:07:17.586844 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.586826    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm"
Jul 01 21:07:17.589580 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.589562    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-86dc6-predictor-serving-cert\""
Jul 01 21:07:17.589680 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.589579    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-86dc6-kube-rbac-proxy-sar-config\""
Jul 01 21:07:17.596843 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.596823    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm"]
Jul 01 21:07:17.680016 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.679987    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc"]
Jul 01 21:07:17.683372 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.683356    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc"
Jul 01 21:07:17.686772 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.686753    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-86dc6-predictor-serving-cert\""
Jul 01 21:07:17.687199 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.687182    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-86dc6-kube-rbac-proxy-sar-config\""
Jul 01 21:07:17.692833 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.692813    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc"]
Jul 01 21:07:17.731782 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.731757    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv"]
Jul 01 21:07:17.732059 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.732019    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv" podUID="fad15195-1882-4c1a-8731-cc759e2831db" containerName="kserve-container" containerID="cri-o://e66a394dadc0a8b57ec2024df01560e15f45dfc75f9b045c0681763dc2c9668c" gracePeriod=30
Jul 01 21:07:17.732488 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.732060    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv" podUID="fad15195-1882-4c1a-8731-cc759e2831db" containerName="kube-rbac-proxy" containerID="cri-o://3305d2f7af26ea78ed9a562e937462413e091f09f8253e43d3a6faa283ff36c5" gracePeriod=30
Jul 01 21:07:17.751488 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.751464    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wdpm\" (UniqueName: \"kubernetes.io/projected/a591d25f-cdbd-4368-b760-6b9a71a763dc-kube-api-access-5wdpm\") pod \"success-200-isvc-86dc6-predictor-6758b686f5-zdnwm\" (UID: \"a591d25f-cdbd-4368-b760-6b9a71a763dc\") " pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm"
Jul 01 21:07:17.751578 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.751503    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f281e07-2bbe-4193-9472-ac2de18a483d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-86dc6-predictor-6869d476d6-pldxc\" (UID: \"5f281e07-2bbe-4193-9472-ac2de18a483d\") " pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc"
Jul 01 21:07:17.751578 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.751521    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czgl2\" (UniqueName: \"kubernetes.io/projected/5f281e07-2bbe-4193-9472-ac2de18a483d-kube-api-access-czgl2\") pod \"error-404-isvc-86dc6-predictor-6869d476d6-pldxc\" (UID: \"5f281e07-2bbe-4193-9472-ac2de18a483d\") " pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc"
Jul 01 21:07:17.751578 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.751543    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a591d25f-cdbd-4368-b760-6b9a71a763dc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-86dc6-predictor-6758b686f5-zdnwm\" (UID: \"a591d25f-cdbd-4368-b760-6b9a71a763dc\") " pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm"
Jul 01 21:07:17.751694 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.751604    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f281e07-2bbe-4193-9472-ac2de18a483d-proxy-tls\") pod \"error-404-isvc-86dc6-predictor-6869d476d6-pldxc\" (UID: \"5f281e07-2bbe-4193-9472-ac2de18a483d\") " pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc"
Jul 01 21:07:17.751694 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.751636    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a591d25f-cdbd-4368-b760-6b9a71a763dc-proxy-tls\") pod \"success-200-isvc-86dc6-predictor-6758b686f5-zdnwm\" (UID: \"a591d25f-cdbd-4368-b760-6b9a71a763dc\") " pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm"
Jul 01 21:07:17.852269 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.852246    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f281e07-2bbe-4193-9472-ac2de18a483d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-86dc6-predictor-6869d476d6-pldxc\" (UID: \"5f281e07-2bbe-4193-9472-ac2de18a483d\") " pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc"
Jul 01 21:07:17.852369 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.852278    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czgl2\" (UniqueName: \"kubernetes.io/projected/5f281e07-2bbe-4193-9472-ac2de18a483d-kube-api-access-czgl2\") pod \"error-404-isvc-86dc6-predictor-6869d476d6-pldxc\" (UID: \"5f281e07-2bbe-4193-9472-ac2de18a483d\") " pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc"
Jul 01 21:07:17.852369 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.852302    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a591d25f-cdbd-4368-b760-6b9a71a763dc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-86dc6-predictor-6758b686f5-zdnwm\" (UID: \"a591d25f-cdbd-4368-b760-6b9a71a763dc\") " pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm"
Jul 01 21:07:17.852369 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.852339    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f281e07-2bbe-4193-9472-ac2de18a483d-proxy-tls\") pod \"error-404-isvc-86dc6-predictor-6869d476d6-pldxc\" (UID: \"5f281e07-2bbe-4193-9472-ac2de18a483d\") " pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc"
Jul 01 21:07:17.852369 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.852359    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a591d25f-cdbd-4368-b760-6b9a71a763dc-proxy-tls\") pod \"success-200-isvc-86dc6-predictor-6758b686f5-zdnwm\" (UID: \"a591d25f-cdbd-4368-b760-6b9a71a763dc\") " pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm"
Jul 01 21:07:17.852566 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.852435    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wdpm\" (UniqueName: \"kubernetes.io/projected/a591d25f-cdbd-4368-b760-6b9a71a763dc-kube-api-access-5wdpm\") pod \"success-200-isvc-86dc6-predictor-6758b686f5-zdnwm\" (UID: \"a591d25f-cdbd-4368-b760-6b9a71a763dc\") " pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm"
Jul 01 21:07:17.852566 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:07:17.852493    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-serving-cert: secret "error-404-isvc-86dc6-predictor-serving-cert" not found
Jul 01 21:07:17.852566 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:07:17.852565    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f281e07-2bbe-4193-9472-ac2de18a483d-proxy-tls podName:5f281e07-2bbe-4193-9472-ac2de18a483d nodeName:}" failed. No retries permitted until 2026-07-01 21:07:18.35254232 +0000 UTC m=+1238.028557887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5f281e07-2bbe-4193-9472-ac2de18a483d-proxy-tls") pod "error-404-isvc-86dc6-predictor-6869d476d6-pldxc" (UID: "5f281e07-2bbe-4193-9472-ac2de18a483d") : secret "error-404-isvc-86dc6-predictor-serving-cert" not found
Jul 01 21:07:17.852945 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.852926    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f281e07-2bbe-4193-9472-ac2de18a483d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-86dc6-predictor-6869d476d6-pldxc\" (UID: \"5f281e07-2bbe-4193-9472-ac2de18a483d\") " pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc"
Jul 01 21:07:17.853140 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.853120    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a591d25f-cdbd-4368-b760-6b9a71a763dc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-86dc6-predictor-6758b686f5-zdnwm\" (UID: \"a591d25f-cdbd-4368-b760-6b9a71a763dc\") " pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm"
Jul 01 21:07:17.854792 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.854775    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a591d25f-cdbd-4368-b760-6b9a71a763dc-proxy-tls\") pod \"success-200-isvc-86dc6-predictor-6758b686f5-zdnwm\" (UID: \"a591d25f-cdbd-4368-b760-6b9a71a763dc\") " pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm"
Jul 01 21:07:17.860806 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.860779    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wdpm\" (UniqueName: \"kubernetes.io/projected/a591d25f-cdbd-4368-b760-6b9a71a763dc-kube-api-access-5wdpm\") pod \"success-200-isvc-86dc6-predictor-6758b686f5-zdnwm\" (UID: \"a591d25f-cdbd-4368-b760-6b9a71a763dc\") " pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm"
Jul 01 21:07:17.861022 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.861006    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czgl2\" (UniqueName: \"kubernetes.io/projected/5f281e07-2bbe-4193-9472-ac2de18a483d-kube-api-access-czgl2\") pod \"error-404-isvc-86dc6-predictor-6869d476d6-pldxc\" (UID: \"5f281e07-2bbe-4193-9472-ac2de18a483d\") " pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc"
Jul 01 21:07:17.897451 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:17.897425    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm"
Jul 01 21:07:18.026202 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:18.026179    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm"]
Jul 01 21:07:18.028291 ip-10-0-131-198 kubenswrapper[2564]: W0701 21:07:18.028263    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda591d25f_cdbd_4368_b760_6b9a71a763dc.slice/crio-c2ec968667b51dfadd432040713f4a02bef927cbe962876ac2d253557fc562e6 WatchSource:0}: Error finding container c2ec968667b51dfadd432040713f4a02bef927cbe962876ac2d253557fc562e6: Status 404 returned error can't find the container with id c2ec968667b51dfadd432040713f4a02bef927cbe962876ac2d253557fc562e6
Jul 01 21:07:18.054325 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:18.054300    2564 generic.go:356] "Generic (PLEG): container finished" podID="fad15195-1882-4c1a-8731-cc759e2831db" containerID="3305d2f7af26ea78ed9a562e937462413e091f09f8253e43d3a6faa283ff36c5" exitCode=2
Jul 01 21:07:18.054405 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:18.054369    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv" event={"ID":"fad15195-1882-4c1a-8731-cc759e2831db","Type":"ContainerDied","Data":"3305d2f7af26ea78ed9a562e937462413e091f09f8253e43d3a6faa283ff36c5"}
Jul 01 21:07:18.055342 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:18.055314    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm" event={"ID":"a591d25f-cdbd-4368-b760-6b9a71a763dc","Type":"ContainerStarted","Data":"c2ec968667b51dfadd432040713f4a02bef927cbe962876ac2d253557fc562e6"}
Jul 01 21:07:18.056793 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:18.056776    2564 generic.go:356] "Generic (PLEG): container finished" podID="0f7441a7-bb69-4d68-8073-98c87a63a34d" containerID="1c6e42ee6db025b915957fa5ba00e535c8810edb360749727aa0f07249f4ac0c" exitCode=2
Jul 01 21:07:18.056897 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:18.056799    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7" event={"ID":"0f7441a7-bb69-4d68-8073-98c87a63a34d","Type":"ContainerDied","Data":"1c6e42ee6db025b915957fa5ba00e535c8810edb360749727aa0f07249f4ac0c"}
Jul 01 21:07:18.354914 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:18.354881    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f281e07-2bbe-4193-9472-ac2de18a483d-proxy-tls\") pod \"error-404-isvc-86dc6-predictor-6869d476d6-pldxc\" (UID: \"5f281e07-2bbe-4193-9472-ac2de18a483d\") " pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc"
Jul 01 21:07:18.357150 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:18.357120    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f281e07-2bbe-4193-9472-ac2de18a483d-proxy-tls\") pod \"error-404-isvc-86dc6-predictor-6869d476d6-pldxc\" (UID: \"5f281e07-2bbe-4193-9472-ac2de18a483d\") " pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc"
Jul 01 21:07:18.593022 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:18.592976    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc"
Jul 01 21:07:18.724649 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:18.724622    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc"]
Jul 01 21:07:18.727415 ip-10-0-131-198 kubenswrapper[2564]: W0701 21:07:18.727387    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f281e07_2bbe_4193_9472_ac2de18a483d.slice/crio-84e41bdee11c3760894164a461d99e94acc8694ff43b14fc1687c1b1174220f5 WatchSource:0}: Error finding container 84e41bdee11c3760894164a461d99e94acc8694ff43b14fc1687c1b1174220f5: Status 404 returned error can't find the container with id 84e41bdee11c3760894164a461d99e94acc8694ff43b14fc1687c1b1174220f5
Jul 01 21:07:19.063150 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:19.063113    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc" event={"ID":"5f281e07-2bbe-4193-9472-ac2de18a483d","Type":"ContainerStarted","Data":"5e1c17b1f4495961809e731142c1ff16fcad223a0f0c4763072f6519bc1e6463"}
Jul 01 21:07:19.063352 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:19.063328    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc"
Jul 01 21:07:19.063422 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:19.063384    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc"
Jul 01 21:07:19.063422 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:19.063414    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc" event={"ID":"5f281e07-2bbe-4193-9472-ac2de18a483d","Type":"ContainerStarted","Data":"583e537ec92717cd2ebce2739b79f6d1995aca9c1fc6ad5845d8f98c86a923b6"}
Jul 01 21:07:19.063534 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:19.063448    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc" event={"ID":"5f281e07-2bbe-4193-9472-ac2de18a483d","Type":"ContainerStarted","Data":"84e41bdee11c3760894164a461d99e94acc8694ff43b14fc1687c1b1174220f5"}
Jul 01 21:07:19.064535 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:19.064499    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc" podUID="5f281e07-2bbe-4193-9472-ac2de18a483d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 21:07:19.065055 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:19.065036    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm" event={"ID":"a591d25f-cdbd-4368-b760-6b9a71a763dc","Type":"ContainerStarted","Data":"fb7b1a899a17d1093cac77fadc5e11285b1572f914ef98d38f82bace59aa7bd5"}
Jul 01 21:07:19.065104 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:19.065078    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm" event={"ID":"a591d25f-cdbd-4368-b760-6b9a71a763dc","Type":"ContainerStarted","Data":"4eda7edbda204a8c765325fa824770ba01a93876ef9b9f91ddbb5f1b07d2b4f9"}
Jul 01 21:07:19.065136 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:19.065109    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm"
Jul 01 21:07:19.065178 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:19.065137    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm"
Jul 01 21:07:19.066117 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:19.066097    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm" podUID="a591d25f-cdbd-4368-b760-6b9a71a763dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 21:07:19.083118 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:19.083074    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc" podStartSLOduration=2.08306163 podStartE2EDuration="2.08306163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:07:19.080791224 +0000 UTC m=+1238.756806834" watchObservedRunningTime="2026-07-01 21:07:19.08306163 +0000 UTC m=+1238.759077196"
Jul 01 21:07:19.098047 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:19.097992    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm" podStartSLOduration=2.097974453 podStartE2EDuration="2.097974453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:07:19.096747357 +0000 UTC m=+1238.772762926" watchObservedRunningTime="2026-07-01 21:07:19.097974453 +0000 UTC m=+1238.773990023"
Jul 01 21:07:19.712852 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:19.712811    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv" podUID="fad15195-1882-4c1a-8731-cc759e2831db" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 01 21:07:20.069560 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:20.069463    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc" podUID="5f281e07-2bbe-4193-9472-ac2de18a483d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 21:07:20.069560 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:20.069538    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm" podUID="a591d25f-cdbd-4368-b760-6b9a71a763dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 21:07:20.719295 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:20.719252    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7" podUID="0f7441a7-bb69-4d68-8073-98c87a63a34d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 01 21:07:20.878417 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:20.878390    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7"
Jul 01 21:07:20.976305 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:20.976283    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f7441a7-bb69-4d68-8073-98c87a63a34d-proxy-tls\") pod \"0f7441a7-bb69-4d68-8073-98c87a63a34d\" (UID: \"0f7441a7-bb69-4d68-8073-98c87a63a34d\") "
Jul 01 21:07:20.976420 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:20.976339    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f7441a7-bb69-4d68-8073-98c87a63a34d-kube-rbac-proxy-sar-config\") pod \"0f7441a7-bb69-4d68-8073-98c87a63a34d\" (UID: \"0f7441a7-bb69-4d68-8073-98c87a63a34d\") "
Jul 01 21:07:20.976420 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:20.976382    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8dv4\" (UniqueName: \"kubernetes.io/projected/0f7441a7-bb69-4d68-8073-98c87a63a34d-kube-api-access-l8dv4\") pod \"0f7441a7-bb69-4d68-8073-98c87a63a34d\" (UID: \"0f7441a7-bb69-4d68-8073-98c87a63a34d\") "
Jul 01 21:07:20.976717 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:20.976696    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f7441a7-bb69-4d68-8073-98c87a63a34d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0f7441a7-bb69-4d68-8073-98c87a63a34d" (UID: "0f7441a7-bb69-4d68-8073-98c87a63a34d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:07:20.978442 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:20.978414    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f7441a7-bb69-4d68-8073-98c87a63a34d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0f7441a7-bb69-4d68-8073-98c87a63a34d" (UID: "0f7441a7-bb69-4d68-8073-98c87a63a34d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:07:20.978586 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:20.978564    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f7441a7-bb69-4d68-8073-98c87a63a34d-kube-api-access-l8dv4" (OuterVolumeSpecName: "kube-api-access-l8dv4") pod "0f7441a7-bb69-4d68-8073-98c87a63a34d" (UID: "0f7441a7-bb69-4d68-8073-98c87a63a34d"). InnerVolumeSpecName "kube-api-access-l8dv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:07:20.982284 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:20.982266    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv"
Jul 01 21:07:21.076572 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.076540    2564 generic.go:356] "Generic (PLEG): container finished" podID="0f7441a7-bb69-4d68-8073-98c87a63a34d" containerID="9f9c11efa0728cab31032bd14d178b62ceeb6fb1e8ccf520ae8464b495b3a66a" exitCode=0
Jul 01 21:07:21.076731 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.076600    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7"
Jul 01 21:07:21.076731 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.076623    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7" event={"ID":"0f7441a7-bb69-4d68-8073-98c87a63a34d","Type":"ContainerDied","Data":"9f9c11efa0728cab31032bd14d178b62ceeb6fb1e8ccf520ae8464b495b3a66a"}
Jul 01 21:07:21.076731 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.076727    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7" event={"ID":"0f7441a7-bb69-4d68-8073-98c87a63a34d","Type":"ContainerDied","Data":"751ac7ce3d14489cf4b8975ea5cdb7c8bde97a150771aa46a3a51ed5535aa8a1"}
Jul 01 21:07:21.076964 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.076755    2564 scope.go:117] "RemoveContainer" containerID="1c6e42ee6db025b915957fa5ba00e535c8810edb360749727aa0f07249f4ac0c"
Jul 01 21:07:21.076964 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.076836    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fad15195-1882-4c1a-8731-cc759e2831db-proxy-tls\") pod \"fad15195-1882-4c1a-8731-cc759e2831db\" (UID: \"fad15195-1882-4c1a-8731-cc759e2831db\") "
Jul 01 21:07:21.076964 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.076937    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fad15195-1882-4c1a-8731-cc759e2831db-kube-rbac-proxy-sar-config\") pod \"fad15195-1882-4c1a-8731-cc759e2831db\" (UID: \"fad15195-1882-4c1a-8731-cc759e2831db\") "
Jul 01 21:07:21.077122 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.076973    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbrr2\" (UniqueName: \"kubernetes.io/projected/fad15195-1882-4c1a-8731-cc759e2831db-kube-api-access-bbrr2\") pod \"fad15195-1882-4c1a-8731-cc759e2831db\" (UID: \"fad15195-1882-4c1a-8731-cc759e2831db\") "
Jul 01 21:07:21.077179 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.077162    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f7441a7-bb69-4d68-8073-98c87a63a34d-proxy-tls\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:07:21.077238 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.077178    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f7441a7-bb69-4d68-8073-98c87a63a34d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:07:21.077238 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.077191    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8dv4\" (UniqueName: \"kubernetes.io/projected/0f7441a7-bb69-4d68-8073-98c87a63a34d-kube-api-access-l8dv4\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:07:21.077385 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.077361    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fad15195-1882-4c1a-8731-cc759e2831db-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fad15195-1882-4c1a-8731-cc759e2831db" (UID: "fad15195-1882-4c1a-8731-cc759e2831db"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:07:21.078550 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.078524    2564 generic.go:356] "Generic (PLEG): container finished" podID="fad15195-1882-4c1a-8731-cc759e2831db" containerID="e66a394dadc0a8b57ec2024df01560e15f45dfc75f9b045c0681763dc2c9668c" exitCode=0
Jul 01 21:07:21.078732 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.078558    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv" event={"ID":"fad15195-1882-4c1a-8731-cc759e2831db","Type":"ContainerDied","Data":"e66a394dadc0a8b57ec2024df01560e15f45dfc75f9b045c0681763dc2c9668c"}
Jul 01 21:07:21.078885 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.078777    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv" event={"ID":"fad15195-1882-4c1a-8731-cc759e2831db","Type":"ContainerDied","Data":"a1ae7dfa76fd845a801c589510e532fb2dd0224553aa233d838cb1b93256a684"}
Jul 01 21:07:21.078885 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.078608    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv"
Jul 01 21:07:21.079252 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.079227    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fad15195-1882-4c1a-8731-cc759e2831db-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fad15195-1882-4c1a-8731-cc759e2831db" (UID: "fad15195-1882-4c1a-8731-cc759e2831db"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:07:21.079379 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.079355    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad15195-1882-4c1a-8731-cc759e2831db-kube-api-access-bbrr2" (OuterVolumeSpecName: "kube-api-access-bbrr2") pod "fad15195-1882-4c1a-8731-cc759e2831db" (UID: "fad15195-1882-4c1a-8731-cc759e2831db"). InnerVolumeSpecName "kube-api-access-bbrr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:07:21.085280 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.085265    2564 scope.go:117] "RemoveContainer" containerID="9f9c11efa0728cab31032bd14d178b62ceeb6fb1e8ccf520ae8464b495b3a66a"
Jul 01 21:07:21.093223 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.093206    2564 scope.go:117] "RemoveContainer" containerID="1c6e42ee6db025b915957fa5ba00e535c8810edb360749727aa0f07249f4ac0c"
Jul 01 21:07:21.093494 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:07:21.093473    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c6e42ee6db025b915957fa5ba00e535c8810edb360749727aa0f07249f4ac0c\": container with ID starting with 1c6e42ee6db025b915957fa5ba00e535c8810edb360749727aa0f07249f4ac0c not found: ID does not exist" containerID="1c6e42ee6db025b915957fa5ba00e535c8810edb360749727aa0f07249f4ac0c"
Jul 01 21:07:21.093572 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.093500    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c6e42ee6db025b915957fa5ba00e535c8810edb360749727aa0f07249f4ac0c"} err="failed to get container status \"1c6e42ee6db025b915957fa5ba00e535c8810edb360749727aa0f07249f4ac0c\": rpc error: code = NotFound desc = could not find container \"1c6e42ee6db025b915957fa5ba00e535c8810edb360749727aa0f07249f4ac0c\": container with ID starting with 1c6e42ee6db025b915957fa5ba00e535c8810edb360749727aa0f07249f4ac0c not found: ID does not exist"
Jul 01 21:07:21.093572 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.093518    2564 scope.go:117] "RemoveContainer" containerID="9f9c11efa0728cab31032bd14d178b62ceeb6fb1e8ccf520ae8464b495b3a66a"
Jul 01 21:07:21.093767 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:07:21.093750    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f9c11efa0728cab31032bd14d178b62ceeb6fb1e8ccf520ae8464b495b3a66a\": container with ID starting with 9f9c11efa0728cab31032bd14d178b62ceeb6fb1e8ccf520ae8464b495b3a66a not found: ID does not exist" containerID="9f9c11efa0728cab31032bd14d178b62ceeb6fb1e8ccf520ae8464b495b3a66a"
Jul 01 21:07:21.093836 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.093773    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f9c11efa0728cab31032bd14d178b62ceeb6fb1e8ccf520ae8464b495b3a66a"} err="failed to get container status \"9f9c11efa0728cab31032bd14d178b62ceeb6fb1e8ccf520ae8464b495b3a66a\": rpc error: code = NotFound desc = could not find container \"9f9c11efa0728cab31032bd14d178b62ceeb6fb1e8ccf520ae8464b495b3a66a\": container with ID starting with 9f9c11efa0728cab31032bd14d178b62ceeb6fb1e8ccf520ae8464b495b3a66a not found: ID does not exist"
Jul 01 21:07:21.093836 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.093789    2564 scope.go:117] "RemoveContainer" containerID="3305d2f7af26ea78ed9a562e937462413e091f09f8253e43d3a6faa283ff36c5"
Jul 01 21:07:21.099393 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.099370    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7"]
Jul 01 21:07:21.103081 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.103061    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f1d61-predictor-69d9f6659c-rznj7"]
Jul 01 21:07:21.104290 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.104267    2564 scope.go:117] "RemoveContainer" containerID="e66a394dadc0a8b57ec2024df01560e15f45dfc75f9b045c0681763dc2c9668c"
Jul 01 21:07:21.112012 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.111995    2564 scope.go:117] "RemoveContainer" containerID="3305d2f7af26ea78ed9a562e937462413e091f09f8253e43d3a6faa283ff36c5"
Jul 01 21:07:21.112249 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:07:21.112230    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3305d2f7af26ea78ed9a562e937462413e091f09f8253e43d3a6faa283ff36c5\": container with ID starting with 3305d2f7af26ea78ed9a562e937462413e091f09f8253e43d3a6faa283ff36c5 not found: ID does not exist" containerID="3305d2f7af26ea78ed9a562e937462413e091f09f8253e43d3a6faa283ff36c5"
Jul 01 21:07:21.112329 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.112257    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3305d2f7af26ea78ed9a562e937462413e091f09f8253e43d3a6faa283ff36c5"} err="failed to get container status \"3305d2f7af26ea78ed9a562e937462413e091f09f8253e43d3a6faa283ff36c5\": rpc error: code = NotFound desc = could not find container \"3305d2f7af26ea78ed9a562e937462413e091f09f8253e43d3a6faa283ff36c5\": container with ID starting with 3305d2f7af26ea78ed9a562e937462413e091f09f8253e43d3a6faa283ff36c5 not found: ID does not exist"
Jul 01 21:07:21.112329 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.112279    2564 scope.go:117] "RemoveContainer" containerID="e66a394dadc0a8b57ec2024df01560e15f45dfc75f9b045c0681763dc2c9668c"
Jul 01 21:07:21.112516 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:07:21.112497    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e66a394dadc0a8b57ec2024df01560e15f45dfc75f9b045c0681763dc2c9668c\": container with ID starting with e66a394dadc0a8b57ec2024df01560e15f45dfc75f9b045c0681763dc2c9668c not found: ID does not exist" containerID="e66a394dadc0a8b57ec2024df01560e15f45dfc75f9b045c0681763dc2c9668c"
Jul 01 21:07:21.112516 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.112522    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e66a394dadc0a8b57ec2024df01560e15f45dfc75f9b045c0681763dc2c9668c"} err="failed to get container status \"e66a394dadc0a8b57ec2024df01560e15f45dfc75f9b045c0681763dc2c9668c\": rpc error: code = NotFound desc = could not find container \"e66a394dadc0a8b57ec2024df01560e15f45dfc75f9b045c0681763dc2c9668c\": container with ID starting with e66a394dadc0a8b57ec2024df01560e15f45dfc75f9b045c0681763dc2c9668c not found: ID does not exist"
Jul 01 21:07:21.177780 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.177750    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fad15195-1882-4c1a-8731-cc759e2831db-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:07:21.177780 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.177775    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbrr2\" (UniqueName: \"kubernetes.io/projected/fad15195-1882-4c1a-8731-cc759e2831db-kube-api-access-bbrr2\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:07:21.177780 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.177786    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fad15195-1882-4c1a-8731-cc759e2831db-proxy-tls\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:07:21.401367 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.401330    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv"]
Jul 01 21:07:21.405819 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:21.405792    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f1d61-predictor-7896dd4c76-6snkv"]
Jul 01 21:07:22.758405 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:22.758374    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f7441a7-bb69-4d68-8073-98c87a63a34d" path="/var/lib/kubelet/pods/0f7441a7-bb69-4d68-8073-98c87a63a34d/volumes"
Jul 01 21:07:22.758788 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:22.758776    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fad15195-1882-4c1a-8731-cc759e2831db" path="/var/lib/kubelet/pods/fad15195-1882-4c1a-8731-cc759e2831db/volumes"
Jul 01 21:07:25.073891 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:25.073846    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc"
Jul 01 21:07:25.074312 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:25.073946    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm"
Jul 01 21:07:25.074398 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:25.074304    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc" podUID="5f281e07-2bbe-4193-9472-ac2de18a483d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 21:07:25.074463 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:25.074392    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm" podUID="a591d25f-cdbd-4368-b760-6b9a71a763dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 21:07:35.075096 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:35.075013    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm" podUID="a591d25f-cdbd-4368-b760-6b9a71a763dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 21:07:35.075096 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:35.075017    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc" podUID="5f281e07-2bbe-4193-9472-ac2de18a483d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 21:07:37.381581 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.381549    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58"]
Jul 01 21:07:37.382176 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.381965    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58" podUID="6993df91-d98d-4a74-ba21-84fa315d343e" containerName="kube-rbac-proxy" containerID="cri-o://379dcef24e957fc74c6210e800231c735000972c91114a6c81848202861ed04a" gracePeriod=30
Jul 01 21:07:37.382176 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.382126    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58" podUID="6993df91-d98d-4a74-ba21-84fa315d343e" containerName="kserve-container" containerID="cri-o://2f56d26cd91e09e4ac49d33686ea8ce25a1540a3e692bc2cafcd018094863479" gracePeriod=30
Jul 01 21:07:37.442388 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.442357    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc"]
Jul 01 21:07:37.442730 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.442715    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fad15195-1882-4c1a-8731-cc759e2831db" containerName="kube-rbac-proxy"
Jul 01 21:07:37.442817 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.442732    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fad15195-1882-4c1a-8731-cc759e2831db" containerName="kube-rbac-proxy"
Jul 01 21:07:37.442817 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.442750    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f7441a7-bb69-4d68-8073-98c87a63a34d" containerName="kserve-container"
Jul 01 21:07:37.442817 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.442755    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f7441a7-bb69-4d68-8073-98c87a63a34d" containerName="kserve-container"
Jul 01 21:07:37.442817 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.442762    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f7441a7-bb69-4d68-8073-98c87a63a34d" containerName="kube-rbac-proxy"
Jul 01 21:07:37.442817 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.442768    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f7441a7-bb69-4d68-8073-98c87a63a34d" containerName="kube-rbac-proxy"
Jul 01 21:07:37.442817 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.442785    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fad15195-1882-4c1a-8731-cc759e2831db" containerName="kserve-container"
Jul 01 21:07:37.442817 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.442791    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fad15195-1882-4c1a-8731-cc759e2831db" containerName="kserve-container"
Jul 01 21:07:37.443248 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.442842    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f7441a7-bb69-4d68-8073-98c87a63a34d" containerName="kserve-container"
Jul 01 21:07:37.443248 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.442853    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f7441a7-bb69-4d68-8073-98c87a63a34d" containerName="kube-rbac-proxy"
Jul 01 21:07:37.443248 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.442873    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fad15195-1882-4c1a-8731-cc759e2831db" containerName="kube-rbac-proxy"
Jul 01 21:07:37.443248 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.442882    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fad15195-1882-4c1a-8731-cc759e2831db" containerName="kserve-container"
Jul 01 21:07:37.447370 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.447351    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v"]
Jul 01 21:07:37.447548 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.447526    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v" podUID="c74f80f2-e534-4b04-af47-e3425926839f" containerName="kserve-container" containerID="cri-o://11e39449474b48b9835a9d39d98d52ef6fdd4ec5e23abb8946fbbadf776958cd" gracePeriod=30
Jul 01 21:07:37.447625 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.447542    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc"
Jul 01 21:07:37.447625 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.447554    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v" podUID="c74f80f2-e534-4b04-af47-e3425926839f" containerName="kube-rbac-proxy" containerID="cri-o://e0d3aeda9ee342defc5abc19fb7e98e6eababd14988ea20467a6e468b19779a4" gracePeriod=30
Jul 01 21:07:37.450144 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.450057    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c5814-kube-rbac-proxy-sar-config\""
Jul 01 21:07:37.450369 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.450277    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c5814-predictor-serving-cert\""
Jul 01 21:07:37.456518 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.456498    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc"]
Jul 01 21:07:37.528742 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.528713    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb"]
Jul 01 21:07:37.532026 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.532011    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb"
Jul 01 21:07:37.534749 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.534729    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c5814-predictor-serving-cert\""
Jul 01 21:07:37.534939 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.534737    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c5814-kube-rbac-proxy-sar-config\""
Jul 01 21:07:37.540000 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.539981    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb"]
Jul 01 21:07:37.613159 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.613121    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cdfc78e7-3714-46d2-b056-5feb3f503126-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c5814-predictor-84c478fd7-87szc\" (UID: \"cdfc78e7-3714-46d2-b056-5feb3f503126\") " pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc"
Jul 01 21:07:37.613289 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.613227    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s68ds\" (UniqueName: \"kubernetes.io/projected/cdfc78e7-3714-46d2-b056-5feb3f503126-kube-api-access-s68ds\") pod \"success-200-isvc-c5814-predictor-84c478fd7-87szc\" (UID: \"cdfc78e7-3714-46d2-b056-5feb3f503126\") " pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc"
Jul 01 21:07:37.613289 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.613265    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdfc78e7-3714-46d2-b056-5feb3f503126-proxy-tls\") pod \"success-200-isvc-c5814-predictor-84c478fd7-87szc\" (UID: \"cdfc78e7-3714-46d2-b056-5feb3f503126\") " pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc"
Jul 01 21:07:37.714526 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.714494    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cdfc78e7-3714-46d2-b056-5feb3f503126-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c5814-predictor-84c478fd7-87szc\" (UID: \"cdfc78e7-3714-46d2-b056-5feb3f503126\") " pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc"
Jul 01 21:07:37.714665 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.714553    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b-proxy-tls\") pod \"error-404-isvc-c5814-predictor-5c7686f9f4-mclpb\" (UID: \"76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b\") " pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb"
Jul 01 21:07:37.714665 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.714572    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c5814-predictor-5c7686f9f4-mclpb\" (UID: \"76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b\") " pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb"
Jul 01 21:07:37.714665 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.714605    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s68ds\" (UniqueName: \"kubernetes.io/projected/cdfc78e7-3714-46d2-b056-5feb3f503126-kube-api-access-s68ds\") pod \"success-200-isvc-c5814-predictor-84c478fd7-87szc\" (UID: \"cdfc78e7-3714-46d2-b056-5feb3f503126\") " pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc"
Jul 01 21:07:37.714665 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.714621    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b8jb\" (UniqueName: \"kubernetes.io/projected/76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b-kube-api-access-8b8jb\") pod \"error-404-isvc-c5814-predictor-5c7686f9f4-mclpb\" (UID: \"76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b\") " pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb"
Jul 01 21:07:37.714665 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.714648    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdfc78e7-3714-46d2-b056-5feb3f503126-proxy-tls\") pod \"success-200-isvc-c5814-predictor-84c478fd7-87szc\" (UID: \"cdfc78e7-3714-46d2-b056-5feb3f503126\") " pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc"
Jul 01 21:07:37.714947 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:07:37.714743    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-c5814-predictor-serving-cert: secret "success-200-isvc-c5814-predictor-serving-cert" not found
Jul 01 21:07:37.714947 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:07:37.714793    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdfc78e7-3714-46d2-b056-5feb3f503126-proxy-tls podName:cdfc78e7-3714-46d2-b056-5feb3f503126 nodeName:}" failed. No retries permitted until 2026-07-01 21:07:38.214778685 +0000 UTC m=+1257.890794229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cdfc78e7-3714-46d2-b056-5feb3f503126-proxy-tls") pod "success-200-isvc-c5814-predictor-84c478fd7-87szc" (UID: "cdfc78e7-3714-46d2-b056-5feb3f503126") : secret "success-200-isvc-c5814-predictor-serving-cert" not found
Jul 01 21:07:37.715173 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.715152    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cdfc78e7-3714-46d2-b056-5feb3f503126-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c5814-predictor-84c478fd7-87szc\" (UID: \"cdfc78e7-3714-46d2-b056-5feb3f503126\") " pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc"
Jul 01 21:07:37.724559 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.724536    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s68ds\" (UniqueName: \"kubernetes.io/projected/cdfc78e7-3714-46d2-b056-5feb3f503126-kube-api-access-s68ds\") pod \"success-200-isvc-c5814-predictor-84c478fd7-87szc\" (UID: \"cdfc78e7-3714-46d2-b056-5feb3f503126\") " pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc"
Jul 01 21:07:37.815519 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.815482    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b8jb\" (UniqueName: \"kubernetes.io/projected/76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b-kube-api-access-8b8jb\") pod \"error-404-isvc-c5814-predictor-5c7686f9f4-mclpb\" (UID: \"76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b\") " pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb"
Jul 01 21:07:37.815681 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.815583    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b-proxy-tls\") pod \"error-404-isvc-c5814-predictor-5c7686f9f4-mclpb\" (UID: \"76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b\") " pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb"
Jul 01 21:07:37.815681 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.815606    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c5814-predictor-5c7686f9f4-mclpb\" (UID: \"76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b\") " pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb"
Jul 01 21:07:37.816326 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.816307    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c5814-predictor-5c7686f9f4-mclpb\" (UID: \"76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b\") " pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb"
Jul 01 21:07:37.818172 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.818151    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b-proxy-tls\") pod \"error-404-isvc-c5814-predictor-5c7686f9f4-mclpb\" (UID: \"76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b\") " pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb"
Jul 01 21:07:37.825641 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.825615    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b8jb\" (UniqueName: \"kubernetes.io/projected/76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b-kube-api-access-8b8jb\") pod \"error-404-isvc-c5814-predictor-5c7686f9f4-mclpb\" (UID: \"76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b\") " pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb"
Jul 01 21:07:37.842503 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.842481    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb"
Jul 01 21:07:37.970916 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:37.970732    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb"]
Jul 01 21:07:38.151941 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:38.151911    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb" event={"ID":"76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b","Type":"ContainerStarted","Data":"3a87e888a4cb846a5200c502bb6ba9af86c70128ce455658db81e1d7e7974943"}
Jul 01 21:07:38.152123 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:38.151968    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb" event={"ID":"76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b","Type":"ContainerStarted","Data":"5be87c547641b8a60f151f315b06ec436ac9d711962b6125c71c6f9e8e99ff16"}
Jul 01 21:07:38.152123 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:38.151999    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb" event={"ID":"76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b","Type":"ContainerStarted","Data":"01ea7d455fc91e4ac87e5e3d7332b7c8116a8f3d42de2c01b354b352ec762106"}
Jul 01 21:07:38.152123 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:38.152037    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb"
Jul 01 21:07:38.153646 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:38.153624    2564 generic.go:356] "Generic (PLEG): container finished" podID="c74f80f2-e534-4b04-af47-e3425926839f" containerID="e0d3aeda9ee342defc5abc19fb7e98e6eababd14988ea20467a6e468b19779a4" exitCode=2
Jul 01 21:07:38.153768 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:38.153693    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v" event={"ID":"c74f80f2-e534-4b04-af47-e3425926839f","Type":"ContainerDied","Data":"e0d3aeda9ee342defc5abc19fb7e98e6eababd14988ea20467a6e468b19779a4"}
Jul 01 21:07:38.155371 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:38.155349    2564 generic.go:356] "Generic (PLEG): container finished" podID="6993df91-d98d-4a74-ba21-84fa315d343e" containerID="379dcef24e957fc74c6210e800231c735000972c91114a6c81848202861ed04a" exitCode=2
Jul 01 21:07:38.155454 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:38.155390    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58" event={"ID":"6993df91-d98d-4a74-ba21-84fa315d343e","Type":"ContainerDied","Data":"379dcef24e957fc74c6210e800231c735000972c91114a6c81848202861ed04a"}
Jul 01 21:07:38.170263 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:38.170224    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb" podStartSLOduration=1.170210446 podStartE2EDuration="1.170210446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:07:38.16868244 +0000 UTC m=+1257.844698019" watchObservedRunningTime="2026-07-01 21:07:38.170210446 +0000 UTC m=+1257.846226085"
Jul 01 21:07:38.219715 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:38.219684    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdfc78e7-3714-46d2-b056-5feb3f503126-proxy-tls\") pod \"success-200-isvc-c5814-predictor-84c478fd7-87szc\" (UID: \"cdfc78e7-3714-46d2-b056-5feb3f503126\") " pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc"
Jul 01 21:07:38.222044 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:38.221986    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdfc78e7-3714-46d2-b056-5feb3f503126-proxy-tls\") pod \"success-200-isvc-c5814-predictor-84c478fd7-87szc\" (UID: \"cdfc78e7-3714-46d2-b056-5feb3f503126\") " pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc"
Jul 01 21:07:38.366367 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:38.366339    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc"
Jul 01 21:07:38.495896 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:38.495798    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc"]
Jul 01 21:07:38.801643 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:38.801527    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v" podUID="c74f80f2-e534-4b04-af47-e3425926839f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 01 21:07:39.166200 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:39.166161    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc" event={"ID":"cdfc78e7-3714-46d2-b056-5feb3f503126","Type":"ContainerStarted","Data":"6673d33c499de2fde57bceae44841fb49e7a869b0793064ad0b8b3d8d9e60422"}
Jul 01 21:07:39.166461 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:39.166444    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb"
Jul 01 21:07:39.166554 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:39.166496    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc"
Jul 01 21:07:39.166554 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:39.166518    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc" event={"ID":"cdfc78e7-3714-46d2-b056-5feb3f503126","Type":"ContainerStarted","Data":"7bb92b6e2ea2bcfb8e84cdf825435530933dd1d378ff516213c61dfd3fb3fe80"}
Jul 01 21:07:39.166554 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:39.166538    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc" event={"ID":"cdfc78e7-3714-46d2-b056-5feb3f503126","Type":"ContainerStarted","Data":"e93ff6b99a751e707b56fe25f08d748584e1cfa0a29250677e0cda731d167aab"}
Jul 01 21:07:39.166727 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:39.166558    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc"
Jul 01 21:07:39.167610 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:39.167585    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb" podUID="76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 01 21:07:39.167679 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:39.167592    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc" podUID="cdfc78e7-3714-46d2-b056-5feb3f503126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 01 21:07:39.202712 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:39.202654    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc" podStartSLOduration=2.202633928 podStartE2EDuration="2.202633928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:07:39.200254065 +0000 UTC m=+1258.876269632" watchObservedRunningTime="2026-07-01 21:07:39.202633928 +0000 UTC m=+1258.878649498"
Jul 01 21:07:40.170165 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:40.170125    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc" podUID="cdfc78e7-3714-46d2-b056-5feb3f503126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 01 21:07:40.170623 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:40.170125    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb" podUID="76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 01 21:07:40.819564 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:40.819526    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58" podUID="6993df91-d98d-4a74-ba21-84fa315d343e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 01 21:07:41.175238 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:41.175199    2564 generic.go:356] "Generic (PLEG): container finished" podID="c74f80f2-e534-4b04-af47-e3425926839f" containerID="11e39449474b48b9835a9d39d98d52ef6fdd4ec5e23abb8946fbbadf776958cd" exitCode=0
Jul 01 21:07:41.175623 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:41.175271    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v" event={"ID":"c74f80f2-e534-4b04-af47-e3425926839f","Type":"ContainerDied","Data":"11e39449474b48b9835a9d39d98d52ef6fdd4ec5e23abb8946fbbadf776958cd"}
Jul 01 21:07:41.293779 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:41.293756    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v"
Jul 01 21:07:41.342250 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:41.342177    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c74f80f2-e534-4b04-af47-e3425926839f-proxy-tls\") pod \"c74f80f2-e534-4b04-af47-e3425926839f\" (UID: \"c74f80f2-e534-4b04-af47-e3425926839f\") "
Jul 01 21:07:41.342382 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:41.342270    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c74f80f2-e534-4b04-af47-e3425926839f-kube-rbac-proxy-sar-config\") pod \"c74f80f2-e534-4b04-af47-e3425926839f\" (UID: \"c74f80f2-e534-4b04-af47-e3425926839f\") "
Jul 01 21:07:41.342382 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:41.342296    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtkz8\" (UniqueName: \"kubernetes.io/projected/c74f80f2-e534-4b04-af47-e3425926839f-kube-api-access-mtkz8\") pod \"c74f80f2-e534-4b04-af47-e3425926839f\" (UID: \"c74f80f2-e534-4b04-af47-e3425926839f\") "
Jul 01 21:07:41.342578 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:41.342556    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c74f80f2-e534-4b04-af47-e3425926839f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c74f80f2-e534-4b04-af47-e3425926839f" (UID: "c74f80f2-e534-4b04-af47-e3425926839f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:07:41.344127 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:41.344108    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c74f80f2-e534-4b04-af47-e3425926839f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c74f80f2-e534-4b04-af47-e3425926839f" (UID: "c74f80f2-e534-4b04-af47-e3425926839f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:07:41.344189 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:41.344129    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c74f80f2-e534-4b04-af47-e3425926839f-kube-api-access-mtkz8" (OuterVolumeSpecName: "kube-api-access-mtkz8") pod "c74f80f2-e534-4b04-af47-e3425926839f" (UID: "c74f80f2-e534-4b04-af47-e3425926839f"). InnerVolumeSpecName "kube-api-access-mtkz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:07:41.443174 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:41.443139    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c74f80f2-e534-4b04-af47-e3425926839f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:07:41.443174 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:41.443169    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtkz8\" (UniqueName: \"kubernetes.io/projected/c74f80f2-e534-4b04-af47-e3425926839f-kube-api-access-mtkz8\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:07:41.443174 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:41.443183    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c74f80f2-e534-4b04-af47-e3425926839f-proxy-tls\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:07:41.722507 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:41.722481    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58"
Jul 01 21:07:41.744678 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:41.744655    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6993df91-d98d-4a74-ba21-84fa315d343e-kube-rbac-proxy-sar-config\") pod \"6993df91-d98d-4a74-ba21-84fa315d343e\" (UID: \"6993df91-d98d-4a74-ba21-84fa315d343e\") "
Jul 01 21:07:41.744814 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:41.744695    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6993df91-d98d-4a74-ba21-84fa315d343e-proxy-tls\") pod \"6993df91-d98d-4a74-ba21-84fa315d343e\" (UID: \"6993df91-d98d-4a74-ba21-84fa315d343e\") "
Jul 01 21:07:41.744814 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:41.744747    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fh6dh\" (UniqueName: \"kubernetes.io/projected/6993df91-d98d-4a74-ba21-84fa315d343e-kube-api-access-fh6dh\") pod \"6993df91-d98d-4a74-ba21-84fa315d343e\" (UID: \"6993df91-d98d-4a74-ba21-84fa315d343e\") "
Jul 01 21:07:41.745098 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:41.745070    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6993df91-d98d-4a74-ba21-84fa315d343e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6993df91-d98d-4a74-ba21-84fa315d343e" (UID: "6993df91-d98d-4a74-ba21-84fa315d343e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:07:41.746800 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:41.746775    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6993df91-d98d-4a74-ba21-84fa315d343e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6993df91-d98d-4a74-ba21-84fa315d343e" (UID: "6993df91-d98d-4a74-ba21-84fa315d343e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:07:41.746944 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:41.746920    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6993df91-d98d-4a74-ba21-84fa315d343e-kube-api-access-fh6dh" (OuterVolumeSpecName: "kube-api-access-fh6dh") pod "6993df91-d98d-4a74-ba21-84fa315d343e" (UID: "6993df91-d98d-4a74-ba21-84fa315d343e"). InnerVolumeSpecName "kube-api-access-fh6dh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:07:41.846212 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:41.846140    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6993df91-d98d-4a74-ba21-84fa315d343e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:07:41.846212 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:41.846164    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6993df91-d98d-4a74-ba21-84fa315d343e-proxy-tls\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:07:41.846212 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:41.846173    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh6dh\" (UniqueName: \"kubernetes.io/projected/6993df91-d98d-4a74-ba21-84fa315d343e-kube-api-access-fh6dh\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:07:42.180561 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:42.180532    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v"
Jul 01 21:07:42.180561 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:42.180545    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v" event={"ID":"c74f80f2-e534-4b04-af47-e3425926839f","Type":"ContainerDied","Data":"d5a613e5d8af2e25c8756fb8eec32900be9093e9115efc77524f3f9a5bdb0ade"}
Jul 01 21:07:42.181080 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:42.180621    2564 scope.go:117] "RemoveContainer" containerID="e0d3aeda9ee342defc5abc19fb7e98e6eababd14988ea20467a6e468b19779a4"
Jul 01 21:07:42.182342 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:42.182319    2564 generic.go:356] "Generic (PLEG): container finished" podID="6993df91-d98d-4a74-ba21-84fa315d343e" containerID="2f56d26cd91e09e4ac49d33686ea8ce25a1540a3e692bc2cafcd018094863479" exitCode=0
Jul 01 21:07:42.182425 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:42.182371    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58"
Jul 01 21:07:42.182425 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:42.182399    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58" event={"ID":"6993df91-d98d-4a74-ba21-84fa315d343e","Type":"ContainerDied","Data":"2f56d26cd91e09e4ac49d33686ea8ce25a1540a3e692bc2cafcd018094863479"}
Jul 01 21:07:42.182499 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:42.182472    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58" event={"ID":"6993df91-d98d-4a74-ba21-84fa315d343e","Type":"ContainerDied","Data":"d230822557dbbfe465b2a533e9258ecc765ce9e0e18899c6ff5d746c0a4d616d"}
Jul 01 21:07:42.189078 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:42.189060    2564 scope.go:117] "RemoveContainer" containerID="11e39449474b48b9835a9d39d98d52ef6fdd4ec5e23abb8946fbbadf776958cd"
Jul 01 21:07:42.200493 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:42.198732    2564 scope.go:117] "RemoveContainer" containerID="379dcef24e957fc74c6210e800231c735000972c91114a6c81848202861ed04a"
Jul 01 21:07:42.206629 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:42.206604    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v"]
Jul 01 21:07:42.208953 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:42.208933    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7cf3f-predictor-5899db67f5-kbg5v"]
Jul 01 21:07:42.210793 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:42.210774    2564 scope.go:117] "RemoveContainer" containerID="2f56d26cd91e09e4ac49d33686ea8ce25a1540a3e692bc2cafcd018094863479"
Jul 01 21:07:42.217568 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:42.217550    2564 scope.go:117] "RemoveContainer" containerID="379dcef24e957fc74c6210e800231c735000972c91114a6c81848202861ed04a"
Jul 01 21:07:42.217797 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:07:42.217780    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"379dcef24e957fc74c6210e800231c735000972c91114a6c81848202861ed04a\": container with ID starting with 379dcef24e957fc74c6210e800231c735000972c91114a6c81848202861ed04a not found: ID does not exist" containerID="379dcef24e957fc74c6210e800231c735000972c91114a6c81848202861ed04a"
Jul 01 21:07:42.217889 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:42.217804    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"379dcef24e957fc74c6210e800231c735000972c91114a6c81848202861ed04a"} err="failed to get container status \"379dcef24e957fc74c6210e800231c735000972c91114a6c81848202861ed04a\": rpc error: code = NotFound desc = could not find container \"379dcef24e957fc74c6210e800231c735000972c91114a6c81848202861ed04a\": container with ID starting with 379dcef24e957fc74c6210e800231c735000972c91114a6c81848202861ed04a not found: ID does not exist"
Jul 01 21:07:42.217889 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:42.217825    2564 scope.go:117] "RemoveContainer" containerID="2f56d26cd91e09e4ac49d33686ea8ce25a1540a3e692bc2cafcd018094863479"
Jul 01 21:07:42.218091 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:07:42.218073    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f56d26cd91e09e4ac49d33686ea8ce25a1540a3e692bc2cafcd018094863479\": container with ID starting with 2f56d26cd91e09e4ac49d33686ea8ce25a1540a3e692bc2cafcd018094863479 not found: ID does not exist" containerID="2f56d26cd91e09e4ac49d33686ea8ce25a1540a3e692bc2cafcd018094863479"
Jul 01 21:07:42.218146 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:42.218097    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f56d26cd91e09e4ac49d33686ea8ce25a1540a3e692bc2cafcd018094863479"} err="failed to get container status \"2f56d26cd91e09e4ac49d33686ea8ce25a1540a3e692bc2cafcd018094863479\": rpc error: code = NotFound desc = could not find container \"2f56d26cd91e09e4ac49d33686ea8ce25a1540a3e692bc2cafcd018094863479\": container with ID starting with 2f56d26cd91e09e4ac49d33686ea8ce25a1540a3e692bc2cafcd018094863479 not found: ID does not exist"
Jul 01 21:07:42.219783 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:42.219764    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58"]
Jul 01 21:07:42.227715 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:42.227695    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7cf3f-predictor-575f8587c8-67j58"]
Jul 01 21:07:42.758014 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:42.757984    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6993df91-d98d-4a74-ba21-84fa315d343e" path="/var/lib/kubelet/pods/6993df91-d98d-4a74-ba21-84fa315d343e/volumes"
Jul 01 21:07:42.758437 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:42.758424    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c74f80f2-e534-4b04-af47-e3425926839f" path="/var/lib/kubelet/pods/c74f80f2-e534-4b04-af47-e3425926839f/volumes"
Jul 01 21:07:45.074852 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:45.074815    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm" podUID="a591d25f-cdbd-4368-b760-6b9a71a763dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 21:07:45.074852 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:45.074821    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc" podUID="5f281e07-2bbe-4193-9472-ac2de18a483d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 21:07:45.174397 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:45.174370    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb"
Jul 01 21:07:45.174696 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:45.174674    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc"
Jul 01 21:07:45.174873 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:45.174832    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb" podUID="76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 01 21:07:45.175296 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:45.175269    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc" podUID="cdfc78e7-3714-46d2-b056-5feb3f503126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 01 21:07:55.074723 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:55.074678    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc" podUID="5f281e07-2bbe-4193-9472-ac2de18a483d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 21:07:55.075103 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:55.074681    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm" podUID="a591d25f-cdbd-4368-b760-6b9a71a763dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 21:07:55.175005 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:55.174961    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb" podUID="76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 01 21:07:55.175287 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:07:55.175260    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc" podUID="cdfc78e7-3714-46d2-b056-5feb3f503126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 01 21:08:05.074989 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:08:05.074961    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc"
Jul 01 21:08:05.075351 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:08:05.075040    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm"
Jul 01 21:08:05.175363 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:08:05.175319    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc" podUID="cdfc78e7-3714-46d2-b056-5feb3f503126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 01 21:08:05.175495 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:08:05.175319    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb" podUID="76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 01 21:08:15.175411 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:08:15.175378    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb" podUID="76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 01 21:08:15.175876 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:08:15.175381    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc" podUID="cdfc78e7-3714-46d2-b056-5feb3f503126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 01 21:08:25.175550 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:08:25.175521    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb"
Jul 01 21:08:25.176392 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:08:25.176373    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc"
Jul 01 21:11:40.822987 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:11:40.822960    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/ovn-acl-logging/0.log"
Jul 01 21:11:40.823496 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:11:40.823164    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/ovn-acl-logging/0.log"
Jul 01 21:16:32.669781 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.669709    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm"]
Jul 01 21:16:32.672332 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.669938    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm" podUID="a591d25f-cdbd-4368-b760-6b9a71a763dc" containerName="kserve-container" containerID="cri-o://4eda7edbda204a8c765325fa824770ba01a93876ef9b9f91ddbb5f1b07d2b4f9" gracePeriod=30
Jul 01 21:16:32.672332 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.669995    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm" podUID="a591d25f-cdbd-4368-b760-6b9a71a763dc" containerName="kube-rbac-proxy" containerID="cri-o://fb7b1a899a17d1093cac77fadc5e11285b1572f914ef98d38f82bace59aa7bd5" gracePeriod=30
Jul 01 21:16:32.676293 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.676270    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg"]
Jul 01 21:16:32.676631 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.676614    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c74f80f2-e534-4b04-af47-e3425926839f" containerName="kserve-container"
Jul 01 21:16:32.676631 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.676631    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c74f80f2-e534-4b04-af47-e3425926839f" containerName="kserve-container"
Jul 01 21:16:32.676769 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.676641    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6993df91-d98d-4a74-ba21-84fa315d343e" containerName="kserve-container"
Jul 01 21:16:32.676769 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.676647    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6993df91-d98d-4a74-ba21-84fa315d343e" containerName="kserve-container"
Jul 01 21:16:32.676769 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.676667    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6993df91-d98d-4a74-ba21-84fa315d343e" containerName="kube-rbac-proxy"
Jul 01 21:16:32.676769 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.676673    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6993df91-d98d-4a74-ba21-84fa315d343e" containerName="kube-rbac-proxy"
Jul 01 21:16:32.676769 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.676687    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c74f80f2-e534-4b04-af47-e3425926839f" containerName="kube-rbac-proxy"
Jul 01 21:16:32.676769 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.676695    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c74f80f2-e534-4b04-af47-e3425926839f" containerName="kube-rbac-proxy"
Jul 01 21:16:32.676769 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.676745    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c74f80f2-e534-4b04-af47-e3425926839f" containerName="kserve-container"
Jul 01 21:16:32.676769 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.676754    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c74f80f2-e534-4b04-af47-e3425926839f" containerName="kube-rbac-proxy"
Jul 01 21:16:32.676769 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.676763    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6993df91-d98d-4a74-ba21-84fa315d343e" containerName="kube-rbac-proxy"
Jul 01 21:16:32.676769 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.676772    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6993df91-d98d-4a74-ba21-84fa315d343e" containerName="kserve-container"
Jul 01 21:16:32.680124 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.680109    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg"
Jul 01 21:16:32.683338 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.683319    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3b58c-predictor-serving-cert\""
Jul 01 21:16:32.683435 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.683323    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3b58c-kube-rbac-proxy-sar-config\""
Jul 01 21:16:32.690276 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.690254    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg"]
Jul 01 21:16:32.725200 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.725166    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08d15b65-39b4-4b32-8c82-a4535bba8ee1-proxy-tls\") pod \"success-200-isvc-3b58c-predictor-66bc46db55-g6zxg\" (UID: \"08d15b65-39b4-4b32-8c82-a4535bba8ee1\") " pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg"
Jul 01 21:16:32.725362 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.725212    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08d15b65-39b4-4b32-8c82-a4535bba8ee1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3b58c-predictor-66bc46db55-g6zxg\" (UID: \"08d15b65-39b4-4b32-8c82-a4535bba8ee1\") " pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg"
Jul 01 21:16:32.725362 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.725265    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwvrw\" (UniqueName: \"kubernetes.io/projected/08d15b65-39b4-4b32-8c82-a4535bba8ee1-kube-api-access-fwvrw\") pod \"success-200-isvc-3b58c-predictor-66bc46db55-g6zxg\" (UID: \"08d15b65-39b4-4b32-8c82-a4535bba8ee1\") " pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg"
Jul 01 21:16:32.790960 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.790933    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw"]
Jul 01 21:16:32.794160 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.794138    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw"
Jul 01 21:16:32.797326 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.797308    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3b58c-predictor-serving-cert\""
Jul 01 21:16:32.797426 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.797334    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3b58c-kube-rbac-proxy-sar-config\""
Jul 01 21:16:32.807269 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.807247    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw"]
Jul 01 21:16:32.826187 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.826075    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/facd928f-2243-4d2e-addf-d9c156d21db4-proxy-tls\") pod \"error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw\" (UID: \"facd928f-2243-4d2e-addf-d9c156d21db4\") " pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw"
Jul 01 21:16:32.826187 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.826119    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6x6c\" (UniqueName: \"kubernetes.io/projected/facd928f-2243-4d2e-addf-d9c156d21db4-kube-api-access-k6x6c\") pod \"error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw\" (UID: \"facd928f-2243-4d2e-addf-d9c156d21db4\") " pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw"
Jul 01 21:16:32.826187 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.826152    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/facd928f-2243-4d2e-addf-d9c156d21db4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw\" (UID: \"facd928f-2243-4d2e-addf-d9c156d21db4\") " pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw"
Jul 01 21:16:32.826426 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.826208    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08d15b65-39b4-4b32-8c82-a4535bba8ee1-proxy-tls\") pod \"success-200-isvc-3b58c-predictor-66bc46db55-g6zxg\" (UID: \"08d15b65-39b4-4b32-8c82-a4535bba8ee1\") " pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg"
Jul 01 21:16:32.826426 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.826235    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08d15b65-39b4-4b32-8c82-a4535bba8ee1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3b58c-predictor-66bc46db55-g6zxg\" (UID: \"08d15b65-39b4-4b32-8c82-a4535bba8ee1\") " pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg"
Jul 01 21:16:32.826426 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.826271    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwvrw\" (UniqueName: \"kubernetes.io/projected/08d15b65-39b4-4b32-8c82-a4535bba8ee1-kube-api-access-fwvrw\") pod \"success-200-isvc-3b58c-predictor-66bc46db55-g6zxg\" (UID: \"08d15b65-39b4-4b32-8c82-a4535bba8ee1\") " pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg"
Jul 01 21:16:32.826576 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:16:32.826529    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-serving-cert: secret "success-200-isvc-3b58c-predictor-serving-cert" not found
Jul 01 21:16:32.826638 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:16:32.826613    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08d15b65-39b4-4b32-8c82-a4535bba8ee1-proxy-tls podName:08d15b65-39b4-4b32-8c82-a4535bba8ee1 nodeName:}" failed. No retries permitted until 2026-07-01 21:16:33.326592192 +0000 UTC m=+1793.002607741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/08d15b65-39b4-4b32-8c82-a4535bba8ee1-proxy-tls") pod "success-200-isvc-3b58c-predictor-66bc46db55-g6zxg" (UID: "08d15b65-39b4-4b32-8c82-a4535bba8ee1") : secret "success-200-isvc-3b58c-predictor-serving-cert" not found
Jul 01 21:16:32.827165 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.827142    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08d15b65-39b4-4b32-8c82-a4535bba8ee1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3b58c-predictor-66bc46db55-g6zxg\" (UID: \"08d15b65-39b4-4b32-8c82-a4535bba8ee1\") " pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg"
Jul 01 21:16:32.837331 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.837290    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwvrw\" (UniqueName: \"kubernetes.io/projected/08d15b65-39b4-4b32-8c82-a4535bba8ee1-kube-api-access-fwvrw\") pod \"success-200-isvc-3b58c-predictor-66bc46db55-g6zxg\" (UID: \"08d15b65-39b4-4b32-8c82-a4535bba8ee1\") " pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg"
Jul 01 21:16:32.839396 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.839374    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc"]
Jul 01 21:16:32.839607 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.839562    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc" podUID="5f281e07-2bbe-4193-9472-ac2de18a483d" containerName="kserve-container" containerID="cri-o://583e537ec92717cd2ebce2739b79f6d1995aca9c1fc6ad5845d8f98c86a923b6" gracePeriod=30
Jul 01 21:16:32.839607 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.839583    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc" podUID="5f281e07-2bbe-4193-9472-ac2de18a483d" containerName="kube-rbac-proxy" containerID="cri-o://5e1c17b1f4495961809e731142c1ff16fcad223a0f0c4763072f6519bc1e6463" gracePeriod=30
Jul 01 21:16:32.927411 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.927341    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/facd928f-2243-4d2e-addf-d9c156d21db4-proxy-tls\") pod \"error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw\" (UID: \"facd928f-2243-4d2e-addf-d9c156d21db4\") " pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw"
Jul 01 21:16:32.927411 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.927377    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6x6c\" (UniqueName: \"kubernetes.io/projected/facd928f-2243-4d2e-addf-d9c156d21db4-kube-api-access-k6x6c\") pod \"error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw\" (UID: \"facd928f-2243-4d2e-addf-d9c156d21db4\") " pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw"
Jul 01 21:16:32.927411 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.927398    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/facd928f-2243-4d2e-addf-d9c156d21db4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw\" (UID: \"facd928f-2243-4d2e-addf-d9c156d21db4\") " pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw"
Jul 01 21:16:32.928060 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.928042    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/facd928f-2243-4d2e-addf-d9c156d21db4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw\" (UID: \"facd928f-2243-4d2e-addf-d9c156d21db4\") " pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw"
Jul 01 21:16:32.929726 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.929705    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/facd928f-2243-4d2e-addf-d9c156d21db4-proxy-tls\") pod \"error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw\" (UID: \"facd928f-2243-4d2e-addf-d9c156d21db4\") " pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw"
Jul 01 21:16:32.936908 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:32.936887    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6x6c\" (UniqueName: \"kubernetes.io/projected/facd928f-2243-4d2e-addf-d9c156d21db4-kube-api-access-k6x6c\") pod \"error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw\" (UID: \"facd928f-2243-4d2e-addf-d9c156d21db4\") " pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw"
Jul 01 21:16:33.104400 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:33.104359    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw"
Jul 01 21:16:33.235211 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:33.235181    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw"]
Jul 01 21:16:33.238125 ip-10-0-131-198 kubenswrapper[2564]: W0701 21:16:33.238100    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfacd928f_2243_4d2e_addf_d9c156d21db4.slice/crio-e5b26777c4412af0a1b2c889a07886c96da51cd4c03910886e7b45cf1e003b0e WatchSource:0}: Error finding container e5b26777c4412af0a1b2c889a07886c96da51cd4c03910886e7b45cf1e003b0e: Status 404 returned error can't find the container with id e5b26777c4412af0a1b2c889a07886c96da51cd4c03910886e7b45cf1e003b0e
Jul 01 21:16:33.257193 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:33.257165    2564 generic.go:356] "Generic (PLEG): container finished" podID="5f281e07-2bbe-4193-9472-ac2de18a483d" containerID="5e1c17b1f4495961809e731142c1ff16fcad223a0f0c4763072f6519bc1e6463" exitCode=2
Jul 01 21:16:33.257283 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:33.257239    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc" event={"ID":"5f281e07-2bbe-4193-9472-ac2de18a483d","Type":"ContainerDied","Data":"5e1c17b1f4495961809e731142c1ff16fcad223a0f0c4763072f6519bc1e6463"}
Jul 01 21:16:33.258305 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:33.258283    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw" event={"ID":"facd928f-2243-4d2e-addf-d9c156d21db4","Type":"ContainerStarted","Data":"e5b26777c4412af0a1b2c889a07886c96da51cd4c03910886e7b45cf1e003b0e"}
Jul 01 21:16:33.260037 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:33.260015    2564 generic.go:356] "Generic (PLEG): container finished" podID="a591d25f-cdbd-4368-b760-6b9a71a763dc" containerID="fb7b1a899a17d1093cac77fadc5e11285b1572f914ef98d38f82bace59aa7bd5" exitCode=2
Jul 01 21:16:33.260128 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:33.260051    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm" event={"ID":"a591d25f-cdbd-4368-b760-6b9a71a763dc","Type":"ContainerDied","Data":"fb7b1a899a17d1093cac77fadc5e11285b1572f914ef98d38f82bace59aa7bd5"}
Jul 01 21:16:33.329955 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:33.329916    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08d15b65-39b4-4b32-8c82-a4535bba8ee1-proxy-tls\") pod \"success-200-isvc-3b58c-predictor-66bc46db55-g6zxg\" (UID: \"08d15b65-39b4-4b32-8c82-a4535bba8ee1\") " pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg"
Jul 01 21:16:33.331994 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:33.331966    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08d15b65-39b4-4b32-8c82-a4535bba8ee1-proxy-tls\") pod \"success-200-isvc-3b58c-predictor-66bc46db55-g6zxg\" (UID: \"08d15b65-39b4-4b32-8c82-a4535bba8ee1\") " pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg"
Jul 01 21:16:33.591779 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:33.591695    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg"
Jul 01 21:16:33.720881 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:33.720838    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg"]
Jul 01 21:16:33.723815 ip-10-0-131-198 kubenswrapper[2564]: W0701 21:16:33.723787    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d15b65_39b4_4b32_8c82_a4535bba8ee1.slice/crio-73fc2d8f73e55f316acd270baa0c5dcf9d046765fdcefad9360093214537af92 WatchSource:0}: Error finding container 73fc2d8f73e55f316acd270baa0c5dcf9d046765fdcefad9360093214537af92: Status 404 returned error can't find the container with id 73fc2d8f73e55f316acd270baa0c5dcf9d046765fdcefad9360093214537af92
Jul 01 21:16:34.265493 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:34.265457    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw" event={"ID":"facd928f-2243-4d2e-addf-d9c156d21db4","Type":"ContainerStarted","Data":"63cebe879e3535d2687d5cb5ad022f033a96f5fad61d08a889251428a2413a79"}
Jul 01 21:16:34.265700 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:34.265530    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw" event={"ID":"facd928f-2243-4d2e-addf-d9c156d21db4","Type":"ContainerStarted","Data":"5faa8d16cea3f0fbb1c0177964439239074654473bb7b43074df137256c6ea90"}
Jul 01 21:16:34.265700 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:34.265564    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw"
Jul 01 21:16:34.265700 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:34.265667    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw"
Jul 01 21:16:34.267110 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:34.267076    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw" podUID="facd928f-2243-4d2e-addf-d9c156d21db4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 01 21:16:34.267445 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:34.267423    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg" event={"ID":"08d15b65-39b4-4b32-8c82-a4535bba8ee1","Type":"ContainerStarted","Data":"3a771138c3cc240d9644560ad4ea4f32e47688779d5e6996341628286aa5fb69"}
Jul 01 21:16:34.267549 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:34.267475    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg" event={"ID":"08d15b65-39b4-4b32-8c82-a4535bba8ee1","Type":"ContainerStarted","Data":"aa32221886ce78d063ae04e03a7519d47e5b155a4374332d78a2cfa7f150363c"}
Jul 01 21:16:34.267549 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:34.267499    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg"
Jul 01 21:16:34.267549 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:34.267518    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg"
Jul 01 21:16:34.267549 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:34.267539    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg" event={"ID":"08d15b65-39b4-4b32-8c82-a4535bba8ee1","Type":"ContainerStarted","Data":"73fc2d8f73e55f316acd270baa0c5dcf9d046765fdcefad9360093214537af92"}
Jul 01 21:16:34.268456 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:34.268435    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg" podUID="08d15b65-39b4-4b32-8c82-a4535bba8ee1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 01 21:16:34.285839 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:34.285793    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw" podStartSLOduration=2.285778433 podStartE2EDuration="2.285778433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:16:34.284990573 +0000 UTC m=+1793.961006140" watchObservedRunningTime="2026-07-01 21:16:34.285778433 +0000 UTC m=+1793.961794001"
Jul 01 21:16:35.070153 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:35.070103    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc" podUID="5f281e07-2bbe-4193-9472-ac2de18a483d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 01 21:16:35.070539 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:35.070104    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm" podUID="a591d25f-cdbd-4368-b760-6b9a71a763dc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 01 21:16:35.074399 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:35.074363    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm" podUID="a591d25f-cdbd-4368-b760-6b9a71a763dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 21:16:35.074520 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:35.074361    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc" podUID="5f281e07-2bbe-4193-9472-ac2de18a483d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 21:16:35.271832 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:35.271784    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg" podUID="08d15b65-39b4-4b32-8c82-a4535bba8ee1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 01 21:16:35.272028 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:35.271788    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw" podUID="facd928f-2243-4d2e-addf-d9c156d21db4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 01 21:16:35.918188 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:35.918168    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm"
Jul 01 21:16:35.942417 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:35.942367    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg" podStartSLOduration=3.942348357 podStartE2EDuration="3.942348357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:16:34.322308073 +0000 UTC m=+1793.998323894" watchObservedRunningTime="2026-07-01 21:16:35.942348357 +0000 UTC m=+1795.618363924"
Jul 01 21:16:36.054714 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.054635    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a591d25f-cdbd-4368-b760-6b9a71a763dc-kube-rbac-proxy-sar-config\") pod \"a591d25f-cdbd-4368-b760-6b9a71a763dc\" (UID: \"a591d25f-cdbd-4368-b760-6b9a71a763dc\") "
Jul 01 21:16:36.054714 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.054668    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a591d25f-cdbd-4368-b760-6b9a71a763dc-proxy-tls\") pod \"a591d25f-cdbd-4368-b760-6b9a71a763dc\" (UID: \"a591d25f-cdbd-4368-b760-6b9a71a763dc\") "
Jul 01 21:16:36.054928 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.054777    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wdpm\" (UniqueName: \"kubernetes.io/projected/a591d25f-cdbd-4368-b760-6b9a71a763dc-kube-api-access-5wdpm\") pod \"a591d25f-cdbd-4368-b760-6b9a71a763dc\" (UID: \"a591d25f-cdbd-4368-b760-6b9a71a763dc\") "
Jul 01 21:16:36.055018 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.054988    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a591d25f-cdbd-4368-b760-6b9a71a763dc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a591d25f-cdbd-4368-b760-6b9a71a763dc" (UID: "a591d25f-cdbd-4368-b760-6b9a71a763dc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:16:36.055126 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.055108    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a591d25f-cdbd-4368-b760-6b9a71a763dc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:16:36.056822 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.056784    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a591d25f-cdbd-4368-b760-6b9a71a763dc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a591d25f-cdbd-4368-b760-6b9a71a763dc" (UID: "a591d25f-cdbd-4368-b760-6b9a71a763dc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:16:36.057023 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.056994    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a591d25f-cdbd-4368-b760-6b9a71a763dc-kube-api-access-5wdpm" (OuterVolumeSpecName: "kube-api-access-5wdpm") pod "a591d25f-cdbd-4368-b760-6b9a71a763dc" (UID: "a591d25f-cdbd-4368-b760-6b9a71a763dc"). InnerVolumeSpecName "kube-api-access-5wdpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:16:36.156268 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.156237    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a591d25f-cdbd-4368-b760-6b9a71a763dc-proxy-tls\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:16:36.156268 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.156265    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wdpm\" (UniqueName: \"kubernetes.io/projected/a591d25f-cdbd-4368-b760-6b9a71a763dc-kube-api-access-5wdpm\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:16:36.276907 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.276874    2564 generic.go:356] "Generic (PLEG): container finished" podID="a591d25f-cdbd-4368-b760-6b9a71a763dc" containerID="4eda7edbda204a8c765325fa824770ba01a93876ef9b9f91ddbb5f1b07d2b4f9" exitCode=0
Jul 01 21:16:36.277064 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.276919    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm" event={"ID":"a591d25f-cdbd-4368-b760-6b9a71a763dc","Type":"ContainerDied","Data":"4eda7edbda204a8c765325fa824770ba01a93876ef9b9f91ddbb5f1b07d2b4f9"}
Jul 01 21:16:36.277064 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.276963    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm" event={"ID":"a591d25f-cdbd-4368-b760-6b9a71a763dc","Type":"ContainerDied","Data":"c2ec968667b51dfadd432040713f4a02bef927cbe962876ac2d253557fc562e6"}
Jul 01 21:16:36.277064 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.276920    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm"
Jul 01 21:16:36.277064 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.276989    2564 scope.go:117] "RemoveContainer" containerID="fb7b1a899a17d1093cac77fadc5e11285b1572f914ef98d38f82bace59aa7bd5"
Jul 01 21:16:36.285746 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.285729    2564 scope.go:117] "RemoveContainer" containerID="4eda7edbda204a8c765325fa824770ba01a93876ef9b9f91ddbb5f1b07d2b4f9"
Jul 01 21:16:36.292724 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.292707    2564 scope.go:117] "RemoveContainer" containerID="fb7b1a899a17d1093cac77fadc5e11285b1572f914ef98d38f82bace59aa7bd5"
Jul 01 21:16:36.293026 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:16:36.293009    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb7b1a899a17d1093cac77fadc5e11285b1572f914ef98d38f82bace59aa7bd5\": container with ID starting with fb7b1a899a17d1093cac77fadc5e11285b1572f914ef98d38f82bace59aa7bd5 not found: ID does not exist" containerID="fb7b1a899a17d1093cac77fadc5e11285b1572f914ef98d38f82bace59aa7bd5"
Jul 01 21:16:36.293097 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.293033    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb7b1a899a17d1093cac77fadc5e11285b1572f914ef98d38f82bace59aa7bd5"} err="failed to get container status \"fb7b1a899a17d1093cac77fadc5e11285b1572f914ef98d38f82bace59aa7bd5\": rpc error: code = NotFound desc = could not find container \"fb7b1a899a17d1093cac77fadc5e11285b1572f914ef98d38f82bace59aa7bd5\": container with ID starting with fb7b1a899a17d1093cac77fadc5e11285b1572f914ef98d38f82bace59aa7bd5 not found: ID does not exist"
Jul 01 21:16:36.293097 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.293048    2564 scope.go:117] "RemoveContainer" containerID="4eda7edbda204a8c765325fa824770ba01a93876ef9b9f91ddbb5f1b07d2b4f9"
Jul 01 21:16:36.293265 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:16:36.293250    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eda7edbda204a8c765325fa824770ba01a93876ef9b9f91ddbb5f1b07d2b4f9\": container with ID starting with 4eda7edbda204a8c765325fa824770ba01a93876ef9b9f91ddbb5f1b07d2b4f9 not found: ID does not exist" containerID="4eda7edbda204a8c765325fa824770ba01a93876ef9b9f91ddbb5f1b07d2b4f9"
Jul 01 21:16:36.293306 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.293269    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eda7edbda204a8c765325fa824770ba01a93876ef9b9f91ddbb5f1b07d2b4f9"} err="failed to get container status \"4eda7edbda204a8c765325fa824770ba01a93876ef9b9f91ddbb5f1b07d2b4f9\": rpc error: code = NotFound desc = could not find container \"4eda7edbda204a8c765325fa824770ba01a93876ef9b9f91ddbb5f1b07d2b4f9\": container with ID starting with 4eda7edbda204a8c765325fa824770ba01a93876ef9b9f91ddbb5f1b07d2b4f9 not found: ID does not exist"
Jul 01 21:16:36.303962 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.303938    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm"]
Jul 01 21:16:36.308145 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.308078    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-86dc6-predictor-6758b686f5-zdnwm"]
Jul 01 21:16:36.567225 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.567202    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc"
Jul 01 21:16:36.659213 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.659173    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f281e07-2bbe-4193-9472-ac2de18a483d-proxy-tls\") pod \"5f281e07-2bbe-4193-9472-ac2de18a483d\" (UID: \"5f281e07-2bbe-4193-9472-ac2de18a483d\") "
Jul 01 21:16:36.659368 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.659277    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f281e07-2bbe-4193-9472-ac2de18a483d-kube-rbac-proxy-sar-config\") pod \"5f281e07-2bbe-4193-9472-ac2de18a483d\" (UID: \"5f281e07-2bbe-4193-9472-ac2de18a483d\") "
Jul 01 21:16:36.659368 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.659309    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czgl2\" (UniqueName: \"kubernetes.io/projected/5f281e07-2bbe-4193-9472-ac2de18a483d-kube-api-access-czgl2\") pod \"5f281e07-2bbe-4193-9472-ac2de18a483d\" (UID: \"5f281e07-2bbe-4193-9472-ac2de18a483d\") "
Jul 01 21:16:36.659613 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.659594    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f281e07-2bbe-4193-9472-ac2de18a483d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5f281e07-2bbe-4193-9472-ac2de18a483d" (UID: "5f281e07-2bbe-4193-9472-ac2de18a483d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:16:36.661238 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.661216    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f281e07-2bbe-4193-9472-ac2de18a483d-kube-api-access-czgl2" (OuterVolumeSpecName: "kube-api-access-czgl2") pod "5f281e07-2bbe-4193-9472-ac2de18a483d" (UID: "5f281e07-2bbe-4193-9472-ac2de18a483d"). InnerVolumeSpecName "kube-api-access-czgl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:16:36.661288 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.661261    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f281e07-2bbe-4193-9472-ac2de18a483d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5f281e07-2bbe-4193-9472-ac2de18a483d" (UID: "5f281e07-2bbe-4193-9472-ac2de18a483d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:16:36.757676 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.757642    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a591d25f-cdbd-4368-b760-6b9a71a763dc" path="/var/lib/kubelet/pods/a591d25f-cdbd-4368-b760-6b9a71a763dc/volumes"
Jul 01 21:16:36.760078 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.760060    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f281e07-2bbe-4193-9472-ac2de18a483d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:16:36.760167 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.760081    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czgl2\" (UniqueName: \"kubernetes.io/projected/5f281e07-2bbe-4193-9472-ac2de18a483d-kube-api-access-czgl2\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:16:36.760167 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:36.760095    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f281e07-2bbe-4193-9472-ac2de18a483d-proxy-tls\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:16:37.283183 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:37.283148    2564 generic.go:356] "Generic (PLEG): container finished" podID="5f281e07-2bbe-4193-9472-ac2de18a483d" containerID="583e537ec92717cd2ebce2739b79f6d1995aca9c1fc6ad5845d8f98c86a923b6" exitCode=0
Jul 01 21:16:37.283567 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:37.283195    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc" event={"ID":"5f281e07-2bbe-4193-9472-ac2de18a483d","Type":"ContainerDied","Data":"583e537ec92717cd2ebce2739b79f6d1995aca9c1fc6ad5845d8f98c86a923b6"}
Jul 01 21:16:37.283567 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:37.283204    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc"
Jul 01 21:16:37.283567 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:37.283255    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc" event={"ID":"5f281e07-2bbe-4193-9472-ac2de18a483d","Type":"ContainerDied","Data":"84e41bdee11c3760894164a461d99e94acc8694ff43b14fc1687c1b1174220f5"}
Jul 01 21:16:37.283567 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:37.283292    2564 scope.go:117] "RemoveContainer" containerID="5e1c17b1f4495961809e731142c1ff16fcad223a0f0c4763072f6519bc1e6463"
Jul 01 21:16:37.291198 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:37.291180    2564 scope.go:117] "RemoveContainer" containerID="583e537ec92717cd2ebce2739b79f6d1995aca9c1fc6ad5845d8f98c86a923b6"
Jul 01 21:16:37.298753 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:37.298730    2564 scope.go:117] "RemoveContainer" containerID="5e1c17b1f4495961809e731142c1ff16fcad223a0f0c4763072f6519bc1e6463"
Jul 01 21:16:37.299102 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:16:37.299084    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e1c17b1f4495961809e731142c1ff16fcad223a0f0c4763072f6519bc1e6463\": container with ID starting with 5e1c17b1f4495961809e731142c1ff16fcad223a0f0c4763072f6519bc1e6463 not found: ID does not exist" containerID="5e1c17b1f4495961809e731142c1ff16fcad223a0f0c4763072f6519bc1e6463"
Jul 01 21:16:37.299174 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:37.299115    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e1c17b1f4495961809e731142c1ff16fcad223a0f0c4763072f6519bc1e6463"} err="failed to get container status \"5e1c17b1f4495961809e731142c1ff16fcad223a0f0c4763072f6519bc1e6463\": rpc error: code = NotFound desc = could not find container \"5e1c17b1f4495961809e731142c1ff16fcad223a0f0c4763072f6519bc1e6463\": container with ID starting with 5e1c17b1f4495961809e731142c1ff16fcad223a0f0c4763072f6519bc1e6463 not found: ID does not exist"
Jul 01 21:16:37.299174 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:37.299132    2564 scope.go:117] "RemoveContainer" containerID="583e537ec92717cd2ebce2739b79f6d1995aca9c1fc6ad5845d8f98c86a923b6"
Jul 01 21:16:37.299403 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:16:37.299388    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"583e537ec92717cd2ebce2739b79f6d1995aca9c1fc6ad5845d8f98c86a923b6\": container with ID starting with 583e537ec92717cd2ebce2739b79f6d1995aca9c1fc6ad5845d8f98c86a923b6 not found: ID does not exist" containerID="583e537ec92717cd2ebce2739b79f6d1995aca9c1fc6ad5845d8f98c86a923b6"
Jul 01 21:16:37.299455 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:37.299405    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"583e537ec92717cd2ebce2739b79f6d1995aca9c1fc6ad5845d8f98c86a923b6"} err="failed to get container status \"583e537ec92717cd2ebce2739b79f6d1995aca9c1fc6ad5845d8f98c86a923b6\": rpc error: code = NotFound desc = could not find container \"583e537ec92717cd2ebce2739b79f6d1995aca9c1fc6ad5845d8f98c86a923b6\": container with ID starting with 583e537ec92717cd2ebce2739b79f6d1995aca9c1fc6ad5845d8f98c86a923b6 not found: ID does not exist"
Jul 01 21:16:37.308826 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:37.308746    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc"]
Jul 01 21:16:37.310677 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:37.310659    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-86dc6-predictor-6869d476d6-pldxc"]
Jul 01 21:16:38.757985 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:38.757955    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f281e07-2bbe-4193-9472-ac2de18a483d" path="/var/lib/kubelet/pods/5f281e07-2bbe-4193-9472-ac2de18a483d/volumes"
Jul 01 21:16:40.276222 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:40.276186    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg"
Jul 01 21:16:40.276672 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:40.276655    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw"
Jul 01 21:16:40.276729 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:40.276712    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg" podUID="08d15b65-39b4-4b32-8c82-a4535bba8ee1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 01 21:16:40.277058 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:40.277024    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw" podUID="facd928f-2243-4d2e-addf-d9c156d21db4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 01 21:16:40.847019 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:40.846993    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/ovn-acl-logging/0.log"
Jul 01 21:16:40.847888 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:40.847870    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/ovn-acl-logging/0.log"
Jul 01 21:16:50.277360 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:50.277316    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg" podUID="08d15b65-39b4-4b32-8c82-a4535bba8ee1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 01 21:16:50.277714 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:50.277315    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw" podUID="facd928f-2243-4d2e-addf-d9c156d21db4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 01 21:16:52.283266 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.283236    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc"]
Jul 01 21:16:52.283669 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.283480    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc" podUID="cdfc78e7-3714-46d2-b056-5feb3f503126" containerName="kserve-container" containerID="cri-o://7bb92b6e2ea2bcfb8e84cdf825435530933dd1d378ff516213c61dfd3fb3fe80" gracePeriod=30
Jul 01 21:16:52.283669 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.283496    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc" podUID="cdfc78e7-3714-46d2-b056-5feb3f503126" containerName="kube-rbac-proxy" containerID="cri-o://6673d33c499de2fde57bceae44841fb49e7a869b0793064ad0b8b3d8d9e60422" gracePeriod=30
Jul 01 21:16:52.293981 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.293957    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2"]
Jul 01 21:16:52.294348 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.294333    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a591d25f-cdbd-4368-b760-6b9a71a763dc" containerName="kserve-container"
Jul 01 21:16:52.294435 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.294352    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a591d25f-cdbd-4368-b760-6b9a71a763dc" containerName="kserve-container"
Jul 01 21:16:52.294435 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.294366    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f281e07-2bbe-4193-9472-ac2de18a483d" containerName="kube-rbac-proxy"
Jul 01 21:16:52.294435 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.294374    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f281e07-2bbe-4193-9472-ac2de18a483d" containerName="kube-rbac-proxy"
Jul 01 21:16:52.294435 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.294405    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f281e07-2bbe-4193-9472-ac2de18a483d" containerName="kserve-container"
Jul 01 21:16:52.294435 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.294414    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f281e07-2bbe-4193-9472-ac2de18a483d" containerName="kserve-container"
Jul 01 21:16:52.294435 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.294434    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a591d25f-cdbd-4368-b760-6b9a71a763dc" containerName="kube-rbac-proxy"
Jul 01 21:16:52.294817 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.294442    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a591d25f-cdbd-4368-b760-6b9a71a763dc" containerName="kube-rbac-proxy"
Jul 01 21:16:52.294817 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.294520    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a591d25f-cdbd-4368-b760-6b9a71a763dc" containerName="kserve-container"
Jul 01 21:16:52.294817 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.294533    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f281e07-2bbe-4193-9472-ac2de18a483d" containerName="kube-rbac-proxy"
Jul 01 21:16:52.294817 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.294546    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a591d25f-cdbd-4368-b760-6b9a71a763dc" containerName="kube-rbac-proxy"
Jul 01 21:16:52.294817 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.294555    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f281e07-2bbe-4193-9472-ac2de18a483d" containerName="kserve-container"
Jul 01 21:16:52.306254 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.306204    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2"
Jul 01 21:16:52.310104 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.310075    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ce993-predictor-serving-cert\""
Jul 01 21:16:52.310228 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.310179    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2"]
Jul 01 21:16:52.310508 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.310490    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ce993-kube-rbac-proxy-sar-config\""
Jul 01 21:16:52.387495 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.387458    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m"]
Jul 01 21:16:52.391092 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.391073    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m"
Jul 01 21:16:52.393768 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.393748    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ce993-predictor-serving-cert\""
Jul 01 21:16:52.393768 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.393759    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ce993-kube-rbac-proxy-sar-config\""
Jul 01 21:16:52.401466 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.401222    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8475069f-ef34-4979-9c40-9dcb2723997b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ce993-predictor-6d75865fd7-pt9r2\" (UID: \"8475069f-ef34-4979-9c40-9dcb2723997b\") " pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2"
Jul 01 21:16:52.401466 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.401316    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb8pc\" (UniqueName: \"kubernetes.io/projected/8475069f-ef34-4979-9c40-9dcb2723997b-kube-api-access-tb8pc\") pod \"success-200-isvc-ce993-predictor-6d75865fd7-pt9r2\" (UID: \"8475069f-ef34-4979-9c40-9dcb2723997b\") " pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2"
Jul 01 21:16:52.401466 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.401429    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8475069f-ef34-4979-9c40-9dcb2723997b-proxy-tls\") pod \"success-200-isvc-ce993-predictor-6d75865fd7-pt9r2\" (UID: \"8475069f-ef34-4979-9c40-9dcb2723997b\") " pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2"
Jul 01 21:16:52.401804 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.401776    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m"]
Jul 01 21:16:52.452960 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.452928    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb"]
Jul 01 21:16:52.453247 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.453196    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb" podUID="76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b" containerName="kserve-container" containerID="cri-o://5be87c547641b8a60f151f315b06ec436ac9d711962b6125c71c6f9e8e99ff16" gracePeriod=30
Jul 01 21:16:52.453436 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.453279    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb" podUID="76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b" containerName="kube-rbac-proxy" containerID="cri-o://3a87e888a4cb846a5200c502bb6ba9af86c70128ce455658db81e1d7e7974943" gracePeriod=30
Jul 01 21:16:52.502494 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.502461    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8475069f-ef34-4979-9c40-9dcb2723997b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ce993-predictor-6d75865fd7-pt9r2\" (UID: \"8475069f-ef34-4979-9c40-9dcb2723997b\") " pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2"
Jul 01 21:16:52.502665 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.502501    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb8pc\" (UniqueName: \"kubernetes.io/projected/8475069f-ef34-4979-9c40-9dcb2723997b-kube-api-access-tb8pc\") pod \"success-200-isvc-ce993-predictor-6d75865fd7-pt9r2\" (UID: \"8475069f-ef34-4979-9c40-9dcb2723997b\") " pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2"
Jul 01 21:16:52.502665 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.502563    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8475069f-ef34-4979-9c40-9dcb2723997b-proxy-tls\") pod \"success-200-isvc-ce993-predictor-6d75865fd7-pt9r2\" (UID: \"8475069f-ef34-4979-9c40-9dcb2723997b\") " pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2"
Jul 01 21:16:52.502665 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.502583    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d60c38a1-a61b-47d6-8da7-8584afb9c4b9-proxy-tls\") pod \"error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m\" (UID: \"d60c38a1-a61b-47d6-8da7-8584afb9c4b9\") " pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m"
Jul 01 21:16:52.502665 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.502600    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d60c38a1-a61b-47d6-8da7-8584afb9c4b9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m\" (UID: \"d60c38a1-a61b-47d6-8da7-8584afb9c4b9\") " pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m"
Jul 01 21:16:52.502665 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.502632    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb5qd\" (UniqueName: \"kubernetes.io/projected/d60c38a1-a61b-47d6-8da7-8584afb9c4b9-kube-api-access-cb5qd\") pod \"error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m\" (UID: \"d60c38a1-a61b-47d6-8da7-8584afb9c4b9\") " pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m"
Jul 01 21:16:52.503342 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.503315    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8475069f-ef34-4979-9c40-9dcb2723997b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ce993-predictor-6d75865fd7-pt9r2\" (UID: \"8475069f-ef34-4979-9c40-9dcb2723997b\") " pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2"
Jul 01 21:16:52.504987 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.504967    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8475069f-ef34-4979-9c40-9dcb2723997b-proxy-tls\") pod \"success-200-isvc-ce993-predictor-6d75865fd7-pt9r2\" (UID: \"8475069f-ef34-4979-9c40-9dcb2723997b\") " pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2"
Jul 01 21:16:52.511870 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.511832    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb8pc\" (UniqueName: \"kubernetes.io/projected/8475069f-ef34-4979-9c40-9dcb2723997b-kube-api-access-tb8pc\") pod \"success-200-isvc-ce993-predictor-6d75865fd7-pt9r2\" (UID: \"8475069f-ef34-4979-9c40-9dcb2723997b\") " pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2"
Jul 01 21:16:52.604088 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.604043    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d60c38a1-a61b-47d6-8da7-8584afb9c4b9-proxy-tls\") pod \"error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m\" (UID: \"d60c38a1-a61b-47d6-8da7-8584afb9c4b9\") " pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m"
Jul 01 21:16:52.604236 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.604092    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d60c38a1-a61b-47d6-8da7-8584afb9c4b9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m\" (UID: \"d60c38a1-a61b-47d6-8da7-8584afb9c4b9\") " pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m"
Jul 01 21:16:52.604236 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.604133    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb5qd\" (UniqueName: \"kubernetes.io/projected/d60c38a1-a61b-47d6-8da7-8584afb9c4b9-kube-api-access-cb5qd\") pod \"error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m\" (UID: \"d60c38a1-a61b-47d6-8da7-8584afb9c4b9\") " pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m"
Jul 01 21:16:52.604914 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.604880    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d60c38a1-a61b-47d6-8da7-8584afb9c4b9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m\" (UID: \"d60c38a1-a61b-47d6-8da7-8584afb9c4b9\") " pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m"
Jul 01 21:16:52.606686 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.606666    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d60c38a1-a61b-47d6-8da7-8584afb9c4b9-proxy-tls\") pod \"error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m\" (UID: \"d60c38a1-a61b-47d6-8da7-8584afb9c4b9\") " pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m"
Jul 01 21:16:52.613816 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.613795    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb5qd\" (UniqueName: \"kubernetes.io/projected/d60c38a1-a61b-47d6-8da7-8584afb9c4b9-kube-api-access-cb5qd\") pod \"error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m\" (UID: \"d60c38a1-a61b-47d6-8da7-8584afb9c4b9\") " pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m"
Jul 01 21:16:52.618602 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.618583    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2"
Jul 01 21:16:52.706632 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.706004    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m"
Jul 01 21:16:52.758681 ip-10-0-131-198 kubenswrapper[2564]: W0701 21:16:52.758654    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8475069f_ef34_4979_9c40_9dcb2723997b.slice/crio-6e73e922a64a14abe22ef4791543b4748772a03526677dd2768a6bbbac7b310f WatchSource:0}: Error finding container 6e73e922a64a14abe22ef4791543b4748772a03526677dd2768a6bbbac7b310f: Status 404 returned error can't find the container with id 6e73e922a64a14abe22ef4791543b4748772a03526677dd2768a6bbbac7b310f
Jul 01 21:16:52.760230 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.760199    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2"]
Jul 01 21:16:52.860020 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:52.859995    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m"]
Jul 01 21:16:52.862427 ip-10-0-131-198 kubenswrapper[2564]: W0701 21:16:52.862391    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60c38a1_a61b_47d6_8da7_8584afb9c4b9.slice/crio-2db226583b8561f222784ed39dfc260813b04ca8fcd8a255039cd253b86ad651 WatchSource:0}: Error finding container 2db226583b8561f222784ed39dfc260813b04ca8fcd8a255039cd253b86ad651: Status 404 returned error can't find the container with id 2db226583b8561f222784ed39dfc260813b04ca8fcd8a255039cd253b86ad651
Jul 01 21:16:53.352157 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:53.352120    2564 generic.go:356] "Generic (PLEG): container finished" podID="76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b" containerID="3a87e888a4cb846a5200c502bb6ba9af86c70128ce455658db81e1d7e7974943" exitCode=2
Jul 01 21:16:53.352602 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:53.352198    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb" event={"ID":"76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b","Type":"ContainerDied","Data":"3a87e888a4cb846a5200c502bb6ba9af86c70128ce455658db81e1d7e7974943"}
Jul 01 21:16:53.354024 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:53.353995    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2" event={"ID":"8475069f-ef34-4979-9c40-9dcb2723997b","Type":"ContainerStarted","Data":"2587094e3cbf813666dc9fe7188355fa989cdb2bdebdce0bbea06ef74bec148b"}
Jul 01 21:16:53.354150 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:53.354052    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2" event={"ID":"8475069f-ef34-4979-9c40-9dcb2723997b","Type":"ContainerStarted","Data":"93bd7d4e3ab30f1c627d39344feca687951e653618d60563a3003f1287a8e2a5"}
Jul 01 21:16:53.354150 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:53.354082    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2" event={"ID":"8475069f-ef34-4979-9c40-9dcb2723997b","Type":"ContainerStarted","Data":"6e73e922a64a14abe22ef4791543b4748772a03526677dd2768a6bbbac7b310f"}
Jul 01 21:16:53.354150 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:53.354112    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2"
Jul 01 21:16:53.354150 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:53.354141    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2"
Jul 01 21:16:53.355301 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:53.355277    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2" podUID="8475069f-ef34-4979-9c40-9dcb2723997b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 01 21:16:53.355746 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:53.355727    2564 generic.go:356] "Generic (PLEG): container finished" podID="cdfc78e7-3714-46d2-b056-5feb3f503126" containerID="6673d33c499de2fde57bceae44841fb49e7a869b0793064ad0b8b3d8d9e60422" exitCode=2
Jul 01 21:16:53.355836 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:53.355791    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc" event={"ID":"cdfc78e7-3714-46d2-b056-5feb3f503126","Type":"ContainerDied","Data":"6673d33c499de2fde57bceae44841fb49e7a869b0793064ad0b8b3d8d9e60422"}
Jul 01 21:16:53.357312 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:53.357292    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m" event={"ID":"d60c38a1-a61b-47d6-8da7-8584afb9c4b9","Type":"ContainerStarted","Data":"59123239647b734b876976c22041f087190a5664c07198563242d9a284a8720d"}
Jul 01 21:16:53.357395 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:53.357338    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m" event={"ID":"d60c38a1-a61b-47d6-8da7-8584afb9c4b9","Type":"ContainerStarted","Data":"d5a90d746799165656c18cd82f1884c687dd7b11e2d2b15365eabc624f7c7c1d"}
Jul 01 21:16:53.357395 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:53.357357    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m" event={"ID":"d60c38a1-a61b-47d6-8da7-8584afb9c4b9","Type":"ContainerStarted","Data":"2db226583b8561f222784ed39dfc260813b04ca8fcd8a255039cd253b86ad651"}
Jul 01 21:16:53.357469 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:53.357435    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m"
Jul 01 21:16:53.376647 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:53.376604    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2" podStartSLOduration=1.376589045 podStartE2EDuration="1.376589045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:16:53.373482411 +0000 UTC m=+1813.049497990" watchObservedRunningTime="2026-07-01 21:16:53.376589045 +0000 UTC m=+1813.052604611"
Jul 01 21:16:53.395723 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:53.395685    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m" podStartSLOduration=1.395670632 podStartE2EDuration="1.395670632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:16:53.394710577 +0000 UTC m=+1813.070726155" watchObservedRunningTime="2026-07-01 21:16:53.395670632 +0000 UTC m=+1813.071686198"
Jul 01 21:16:54.362156 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:54.362117    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2" podUID="8475069f-ef34-4979-9c40-9dcb2723997b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 01 21:16:54.362596 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:54.362236    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m"
Jul 01 21:16:54.363401 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:54.363376    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m" podUID="d60c38a1-a61b-47d6-8da7-8584afb9c4b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 01 21:16:55.170746 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:55.170707    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb" podUID="76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Jul 01 21:16:55.170922 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:55.170710    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc" podUID="cdfc78e7-3714-46d2-b056-5feb3f503126" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 01 21:16:55.175156 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:55.175128    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb" podUID="76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 01 21:16:55.175282 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:55.175261    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc" podUID="cdfc78e7-3714-46d2-b056-5feb3f503126" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 01 21:16:55.371425 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:55.371384    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m" podUID="d60c38a1-a61b-47d6-8da7-8584afb9c4b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 01 21:16:55.743306 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:55.743283    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc"
Jul 01 21:16:55.830519 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:55.830432    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s68ds\" (UniqueName: \"kubernetes.io/projected/cdfc78e7-3714-46d2-b056-5feb3f503126-kube-api-access-s68ds\") pod \"cdfc78e7-3714-46d2-b056-5feb3f503126\" (UID: \"cdfc78e7-3714-46d2-b056-5feb3f503126\") "
Jul 01 21:16:55.830700 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:55.830523    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cdfc78e7-3714-46d2-b056-5feb3f503126-kube-rbac-proxy-sar-config\") pod \"cdfc78e7-3714-46d2-b056-5feb3f503126\" (UID: \"cdfc78e7-3714-46d2-b056-5feb3f503126\") "
Jul 01 21:16:55.830700 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:55.830570    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdfc78e7-3714-46d2-b056-5feb3f503126-proxy-tls\") pod \"cdfc78e7-3714-46d2-b056-5feb3f503126\" (UID: \"cdfc78e7-3714-46d2-b056-5feb3f503126\") "
Jul 01 21:16:55.830952 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:55.830929    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdfc78e7-3714-46d2-b056-5feb3f503126-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cdfc78e7-3714-46d2-b056-5feb3f503126" (UID: "cdfc78e7-3714-46d2-b056-5feb3f503126"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:16:55.832571 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:55.832550    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdfc78e7-3714-46d2-b056-5feb3f503126-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cdfc78e7-3714-46d2-b056-5feb3f503126" (UID: "cdfc78e7-3714-46d2-b056-5feb3f503126"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:16:55.832700 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:55.832679    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdfc78e7-3714-46d2-b056-5feb3f503126-kube-api-access-s68ds" (OuterVolumeSpecName: "kube-api-access-s68ds") pod "cdfc78e7-3714-46d2-b056-5feb3f503126" (UID: "cdfc78e7-3714-46d2-b056-5feb3f503126"). InnerVolumeSpecName "kube-api-access-s68ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:16:55.932331 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:55.932279    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cdfc78e7-3714-46d2-b056-5feb3f503126-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:16:55.932539 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:55.932426    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdfc78e7-3714-46d2-b056-5feb3f503126-proxy-tls\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:16:55.932539 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:55.932451    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s68ds\" (UniqueName: \"kubernetes.io/projected/cdfc78e7-3714-46d2-b056-5feb3f503126-kube-api-access-s68ds\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:16:56.376256 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:56.376223    2564 generic.go:356] "Generic (PLEG): container finished" podID="cdfc78e7-3714-46d2-b056-5feb3f503126" containerID="7bb92b6e2ea2bcfb8e84cdf825435530933dd1d378ff516213c61dfd3fb3fe80" exitCode=0
Jul 01 21:16:56.376656 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:56.376268    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc" event={"ID":"cdfc78e7-3714-46d2-b056-5feb3f503126","Type":"ContainerDied","Data":"7bb92b6e2ea2bcfb8e84cdf825435530933dd1d378ff516213c61dfd3fb3fe80"}
Jul 01 21:16:56.376656 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:56.376316    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc" event={"ID":"cdfc78e7-3714-46d2-b056-5feb3f503126","Type":"ContainerDied","Data":"e93ff6b99a751e707b56fe25f08d748584e1cfa0a29250677e0cda731d167aab"}
Jul 01 21:16:56.376656 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:56.376271    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc"
Jul 01 21:16:56.376656 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:56.376341    2564 scope.go:117] "RemoveContainer" containerID="6673d33c499de2fde57bceae44841fb49e7a869b0793064ad0b8b3d8d9e60422"
Jul 01 21:16:56.384502 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:56.384479    2564 scope.go:117] "RemoveContainer" containerID="7bb92b6e2ea2bcfb8e84cdf825435530933dd1d378ff516213c61dfd3fb3fe80"
Jul 01 21:16:56.391582 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:56.391565    2564 scope.go:117] "RemoveContainer" containerID="6673d33c499de2fde57bceae44841fb49e7a869b0793064ad0b8b3d8d9e60422"
Jul 01 21:16:56.391822 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:16:56.391804    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6673d33c499de2fde57bceae44841fb49e7a869b0793064ad0b8b3d8d9e60422\": container with ID starting with 6673d33c499de2fde57bceae44841fb49e7a869b0793064ad0b8b3d8d9e60422 not found: ID does not exist" containerID="6673d33c499de2fde57bceae44841fb49e7a869b0793064ad0b8b3d8d9e60422"
Jul 01 21:16:56.391882 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:56.391831    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6673d33c499de2fde57bceae44841fb49e7a869b0793064ad0b8b3d8d9e60422"} err="failed to get container status \"6673d33c499de2fde57bceae44841fb49e7a869b0793064ad0b8b3d8d9e60422\": rpc error: code = NotFound desc = could not find container \"6673d33c499de2fde57bceae44841fb49e7a869b0793064ad0b8b3d8d9e60422\": container with ID starting with 6673d33c499de2fde57bceae44841fb49e7a869b0793064ad0b8b3d8d9e60422 not found: ID does not exist"
Jul 01 21:16:56.391882 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:56.391847    2564 scope.go:117] "RemoveContainer" containerID="7bb92b6e2ea2bcfb8e84cdf825435530933dd1d378ff516213c61dfd3fb3fe80"
Jul 01 21:16:56.392100 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:16:56.392082    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bb92b6e2ea2bcfb8e84cdf825435530933dd1d378ff516213c61dfd3fb3fe80\": container with ID starting with 7bb92b6e2ea2bcfb8e84cdf825435530933dd1d378ff516213c61dfd3fb3fe80 not found: ID does not exist" containerID="7bb92b6e2ea2bcfb8e84cdf825435530933dd1d378ff516213c61dfd3fb3fe80"
Jul 01 21:16:56.392158 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:56.392109    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bb92b6e2ea2bcfb8e84cdf825435530933dd1d378ff516213c61dfd3fb3fe80"} err="failed to get container status \"7bb92b6e2ea2bcfb8e84cdf825435530933dd1d378ff516213c61dfd3fb3fe80\": rpc error: code = NotFound desc = could not find container \"7bb92b6e2ea2bcfb8e84cdf825435530933dd1d378ff516213c61dfd3fb3fe80\": container with ID starting with 7bb92b6e2ea2bcfb8e84cdf825435530933dd1d378ff516213c61dfd3fb3fe80 not found: ID does not exist"
Jul 01 21:16:56.399088 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:56.399060    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc"]
Jul 01 21:16:56.402922 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:56.402826    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c5814-predictor-84c478fd7-87szc"]
Jul 01 21:16:56.599832 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:56.599802    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb"
Jul 01 21:16:56.739285 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:56.739249    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b-proxy-tls\") pod \"76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b\" (UID: \"76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b\") "
Jul 01 21:16:56.739442 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:56.739306    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b-kube-rbac-proxy-sar-config\") pod \"76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b\" (UID: \"76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b\") "
Jul 01 21:16:56.739442 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:56.739412    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8b8jb\" (UniqueName: \"kubernetes.io/projected/76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b-kube-api-access-8b8jb\") pod \"76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b\" (UID: \"76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b\") "
Jul 01 21:16:56.739624 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:56.739602    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b" (UID: "76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:16:56.739725 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:56.739707    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:16:56.741404 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:56.741380    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b" (UID: "76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:16:56.741522 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:56.741493    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b-kube-api-access-8b8jb" (OuterVolumeSpecName: "kube-api-access-8b8jb") pod "76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b" (UID: "76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b"). InnerVolumeSpecName "kube-api-access-8b8jb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:16:56.757656 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:56.757629    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdfc78e7-3714-46d2-b056-5feb3f503126" path="/var/lib/kubelet/pods/cdfc78e7-3714-46d2-b056-5feb3f503126/volumes"
Jul 01 21:16:56.840653 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:56.840619    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b8jb\" (UniqueName: \"kubernetes.io/projected/76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b-kube-api-access-8b8jb\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:16:56.840653 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:56.840647    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b-proxy-tls\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:16:57.381559 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:57.381528    2564 generic.go:356] "Generic (PLEG): container finished" podID="76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b" containerID="5be87c547641b8a60f151f315b06ec436ac9d711962b6125c71c6f9e8e99ff16" exitCode=0
Jul 01 21:16:57.381968 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:57.381580    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb"
Jul 01 21:16:57.381968 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:57.381613    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb" event={"ID":"76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b","Type":"ContainerDied","Data":"5be87c547641b8a60f151f315b06ec436ac9d711962b6125c71c6f9e8e99ff16"}
Jul 01 21:16:57.381968 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:57.381668    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb" event={"ID":"76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b","Type":"ContainerDied","Data":"01ea7d455fc91e4ac87e5e3d7332b7c8116a8f3d42de2c01b354b352ec762106"}
Jul 01 21:16:57.381968 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:57.381701    2564 scope.go:117] "RemoveContainer" containerID="3a87e888a4cb846a5200c502bb6ba9af86c70128ce455658db81e1d7e7974943"
Jul 01 21:16:57.389780 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:57.389765    2564 scope.go:117] "RemoveContainer" containerID="5be87c547641b8a60f151f315b06ec436ac9d711962b6125c71c6f9e8e99ff16"
Jul 01 21:16:57.397073 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:57.397054    2564 scope.go:117] "RemoveContainer" containerID="3a87e888a4cb846a5200c502bb6ba9af86c70128ce455658db81e1d7e7974943"
Jul 01 21:16:57.397378 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:16:57.397348    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a87e888a4cb846a5200c502bb6ba9af86c70128ce455658db81e1d7e7974943\": container with ID starting with 3a87e888a4cb846a5200c502bb6ba9af86c70128ce455658db81e1d7e7974943 not found: ID does not exist" containerID="3a87e888a4cb846a5200c502bb6ba9af86c70128ce455658db81e1d7e7974943"
Jul 01 21:16:57.397466 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:57.397377    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a87e888a4cb846a5200c502bb6ba9af86c70128ce455658db81e1d7e7974943"} err="failed to get container status \"3a87e888a4cb846a5200c502bb6ba9af86c70128ce455658db81e1d7e7974943\": rpc error: code = NotFound desc = could not find container \"3a87e888a4cb846a5200c502bb6ba9af86c70128ce455658db81e1d7e7974943\": container with ID starting with 3a87e888a4cb846a5200c502bb6ba9af86c70128ce455658db81e1d7e7974943 not found: ID does not exist"
Jul 01 21:16:57.397466 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:57.397397    2564 scope.go:117] "RemoveContainer" containerID="5be87c547641b8a60f151f315b06ec436ac9d711962b6125c71c6f9e8e99ff16"
Jul 01 21:16:57.397680 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:16:57.397664    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5be87c547641b8a60f151f315b06ec436ac9d711962b6125c71c6f9e8e99ff16\": container with ID starting with 5be87c547641b8a60f151f315b06ec436ac9d711962b6125c71c6f9e8e99ff16 not found: ID does not exist" containerID="5be87c547641b8a60f151f315b06ec436ac9d711962b6125c71c6f9e8e99ff16"
Jul 01 21:16:57.397738 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:57.397684    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5be87c547641b8a60f151f315b06ec436ac9d711962b6125c71c6f9e8e99ff16"} err="failed to get container status \"5be87c547641b8a60f151f315b06ec436ac9d711962b6125c71c6f9e8e99ff16\": rpc error: code = NotFound desc = could not find container \"5be87c547641b8a60f151f315b06ec436ac9d711962b6125c71c6f9e8e99ff16\": container with ID starting with 5be87c547641b8a60f151f315b06ec436ac9d711962b6125c71c6f9e8e99ff16 not found: ID does not exist"
Jul 01 21:16:57.404398 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:57.404364    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb"]
Jul 01 21:16:57.408540 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:57.408519    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c5814-predictor-5c7686f9f4-mclpb"]
Jul 01 21:16:58.757487 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:58.757455    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b" path="/var/lib/kubelet/pods/76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b/volumes"
Jul 01 21:16:59.366410 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:59.366386    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2"
Jul 01 21:16:59.366911 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:16:59.366883    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2" podUID="8475069f-ef34-4979-9c40-9dcb2723997b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 01 21:17:00.277032 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:00.276996    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg" podUID="08d15b65-39b4-4b32-8c82-a4535bba8ee1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 01 21:17:00.277427 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:00.277133    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw" podUID="facd928f-2243-4d2e-addf-d9c156d21db4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 01 21:17:00.375599 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:00.375571    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m"
Jul 01 21:17:00.376025 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:00.376001    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m" podUID="d60c38a1-a61b-47d6-8da7-8584afb9c4b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 01 21:17:09.367101 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:09.367063    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2" podUID="8475069f-ef34-4979-9c40-9dcb2723997b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 01 21:17:10.277305 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:10.277267    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw" podUID="facd928f-2243-4d2e-addf-d9c156d21db4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 01 21:17:10.277502 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:10.277267    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg" podUID="08d15b65-39b4-4b32-8c82-a4535bba8ee1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 01 21:17:10.376311 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:10.376272    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m" podUID="d60c38a1-a61b-47d6-8da7-8584afb9c4b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 01 21:17:19.367483 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:19.367444    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2" podUID="8475069f-ef34-4979-9c40-9dcb2723997b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 01 21:17:20.276904 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:20.276845    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg" podUID="08d15b65-39b4-4b32-8c82-a4535bba8ee1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 01 21:17:20.277673 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:20.277652    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw"
Jul 01 21:17:20.376596 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:20.376556    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m" podUID="d60c38a1-a61b-47d6-8da7-8584afb9c4b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 01 21:17:29.367416 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:29.367372    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2" podUID="8475069f-ef34-4979-9c40-9dcb2723997b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 01 21:17:30.277578 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:30.277551    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg"
Jul 01 21:17:30.376561 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:30.376529    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m" podUID="d60c38a1-a61b-47d6-8da7-8584afb9c4b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 01 21:17:39.367401 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:39.367353    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2" podUID="8475069f-ef34-4979-9c40-9dcb2723997b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 01 21:17:40.376885 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:40.376836    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m"
Jul 01 21:17:49.367411 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:49.367383    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2"
Jul 01 21:17:52.839982 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.839950    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg"]
Jul 01 21:17:52.840445 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.840283    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg" podUID="08d15b65-39b4-4b32-8c82-a4535bba8ee1" containerName="kserve-container" containerID="cri-o://aa32221886ce78d063ae04e03a7519d47e5b155a4374332d78a2cfa7f150363c" gracePeriod=30
Jul 01 21:17:52.840445 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.840347    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg" podUID="08d15b65-39b4-4b32-8c82-a4535bba8ee1" containerName="kube-rbac-proxy" containerID="cri-o://3a771138c3cc240d9644560ad4ea4f32e47688779d5e6996341628286aa5fb69" gracePeriod=30
Jul 01 21:17:52.857444 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.857423    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt"]
Jul 01 21:17:52.857758 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.857747    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdfc78e7-3714-46d2-b056-5feb3f503126" containerName="kserve-container"
Jul 01 21:17:52.857803 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.857759    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdfc78e7-3714-46d2-b056-5feb3f503126" containerName="kserve-container"
Jul 01 21:17:52.857803 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.857773    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b" containerName="kserve-container"
Jul 01 21:17:52.857803 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.857779    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b" containerName="kserve-container"
Jul 01 21:17:52.857803 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.857785    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b" containerName="kube-rbac-proxy"
Jul 01 21:17:52.857803 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.857791    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b" containerName="kube-rbac-proxy"
Jul 01 21:17:52.857803 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.857798    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdfc78e7-3714-46d2-b056-5feb3f503126" containerName="kube-rbac-proxy"
Jul 01 21:17:52.857803 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.857803    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdfc78e7-3714-46d2-b056-5feb3f503126" containerName="kube-rbac-proxy"
Jul 01 21:17:52.858040 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.857854    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b" containerName="kube-rbac-proxy"
Jul 01 21:17:52.858040 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.857890    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="76aa05e4-cee0-4c7b-bbde-5a1c3bfea65b" containerName="kserve-container"
Jul 01 21:17:52.858040 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.857897    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdfc78e7-3714-46d2-b056-5feb3f503126" containerName="kserve-container"
Jul 01 21:17:52.858040 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.857902    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdfc78e7-3714-46d2-b056-5feb3f503126" containerName="kube-rbac-proxy"
Jul 01 21:17:52.860921 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.860900    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt"
Jul 01 21:17:52.863718 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.863699    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6396f-kube-rbac-proxy-sar-config\""
Jul 01 21:17:52.863813 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.863699    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6396f-predictor-serving-cert\""
Jul 01 21:17:52.877182 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.877161    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt"]
Jul 01 21:17:52.912747 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.912723    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50527622-67cb-46d0-a3d1-95ab459aaae0-proxy-tls\") pod \"success-200-isvc-6396f-predictor-85b9f855cc-l99gt\" (UID: \"50527622-67cb-46d0-a3d1-95ab459aaae0\") " pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt"
Jul 01 21:17:52.912880 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.912756    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttm8z\" (UniqueName: \"kubernetes.io/projected/50527622-67cb-46d0-a3d1-95ab459aaae0-kube-api-access-ttm8z\") pod \"success-200-isvc-6396f-predictor-85b9f855cc-l99gt\" (UID: \"50527622-67cb-46d0-a3d1-95ab459aaae0\") " pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt"
Jul 01 21:17:52.912880 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.912776    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50527622-67cb-46d0-a3d1-95ab459aaae0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6396f-predictor-85b9f855cc-l99gt\" (UID: \"50527622-67cb-46d0-a3d1-95ab459aaae0\") " pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt"
Jul 01 21:17:52.978656 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.978634    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f"]
Jul 01 21:17:52.982214 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.982197    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f"
Jul 01 21:17:52.985558 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.985544    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6396f-predictor-serving-cert\""
Jul 01 21:17:52.985709 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.985694    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6396f-kube-rbac-proxy-sar-config\""
Jul 01 21:17:52.994660 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:52.994641    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f"]
Jul 01 21:17:53.013519 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:53.013493    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50527622-67cb-46d0-a3d1-95ab459aaae0-proxy-tls\") pod \"success-200-isvc-6396f-predictor-85b9f855cc-l99gt\" (UID: \"50527622-67cb-46d0-a3d1-95ab459aaae0\") " pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt"
Jul 01 21:17:53.013652 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:53.013533    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttm8z\" (UniqueName: \"kubernetes.io/projected/50527622-67cb-46d0-a3d1-95ab459aaae0-kube-api-access-ttm8z\") pod \"success-200-isvc-6396f-predictor-85b9f855cc-l99gt\" (UID: \"50527622-67cb-46d0-a3d1-95ab459aaae0\") " pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt"
Jul 01 21:17:53.013652 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:53.013562    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a87761b4-21ed-4470-ae67-77e1c811ee2f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f\" (UID: \"a87761b4-21ed-4470-ae67-77e1c811ee2f\") " pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f"
Jul 01 21:17:53.013652 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:53.013588    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50527622-67cb-46d0-a3d1-95ab459aaae0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6396f-predictor-85b9f855cc-l99gt\" (UID: \"50527622-67cb-46d0-a3d1-95ab459aaae0\") " pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt"
Jul 01 21:17:53.013781 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:17:53.013661    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-6396f-predictor-serving-cert: secret "success-200-isvc-6396f-predictor-serving-cert" not found
Jul 01 21:17:53.013781 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:17:53.013717    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50527622-67cb-46d0-a3d1-95ab459aaae0-proxy-tls podName:50527622-67cb-46d0-a3d1-95ab459aaae0 nodeName:}" failed. No retries permitted until 2026-07-01 21:17:53.513697984 +0000 UTC m=+1873.189713546 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/50527622-67cb-46d0-a3d1-95ab459aaae0-proxy-tls") pod "success-200-isvc-6396f-predictor-85b9f855cc-l99gt" (UID: "50527622-67cb-46d0-a3d1-95ab459aaae0") : secret "success-200-isvc-6396f-predictor-serving-cert" not found
Jul 01 21:17:53.013915 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:53.013832    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4m6r\" (UniqueName: \"kubernetes.io/projected/a87761b4-21ed-4470-ae67-77e1c811ee2f-kube-api-access-f4m6r\") pod \"error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f\" (UID: \"a87761b4-21ed-4470-ae67-77e1c811ee2f\") " pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f"
Jul 01 21:17:53.014004 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:53.013977    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a87761b4-21ed-4470-ae67-77e1c811ee2f-proxy-tls\") pod \"error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f\" (UID: \"a87761b4-21ed-4470-ae67-77e1c811ee2f\") " pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f"
Jul 01 21:17:53.014338 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:53.014302    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50527622-67cb-46d0-a3d1-95ab459aaae0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6396f-predictor-85b9f855cc-l99gt\" (UID: \"50527622-67cb-46d0-a3d1-95ab459aaae0\") " pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt"
Jul 01 21:17:53.027025 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:53.027001    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw"]
Jul 01 21:17:53.027264 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:53.027243    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw" podUID="facd928f-2243-4d2e-addf-d9c156d21db4" containerName="kserve-container" containerID="cri-o://5faa8d16cea3f0fbb1c0177964439239074654473bb7b43074df137256c6ea90" gracePeriod=30
Jul 01 21:17:53.027345 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:53.027255    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw" podUID="facd928f-2243-4d2e-addf-d9c156d21db4" containerName="kube-rbac-proxy" containerID="cri-o://63cebe879e3535d2687d5cb5ad022f033a96f5fad61d08a889251428a2413a79" gracePeriod=30
Jul 01 21:17:53.028515 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:53.028486    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttm8z\" (UniqueName: \"kubernetes.io/projected/50527622-67cb-46d0-a3d1-95ab459aaae0-kube-api-access-ttm8z\") pod \"success-200-isvc-6396f-predictor-85b9f855cc-l99gt\" (UID: \"50527622-67cb-46d0-a3d1-95ab459aaae0\") " pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt"
Jul 01 21:17:53.114585 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:53.114520    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a87761b4-21ed-4470-ae67-77e1c811ee2f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f\" (UID: \"a87761b4-21ed-4470-ae67-77e1c811ee2f\") " pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f"
Jul 01 21:17:53.114585 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:53.114549    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4m6r\" (UniqueName: \"kubernetes.io/projected/a87761b4-21ed-4470-ae67-77e1c811ee2f-kube-api-access-f4m6r\") pod \"error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f\" (UID: \"a87761b4-21ed-4470-ae67-77e1c811ee2f\") " pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f"
Jul 01 21:17:53.114753 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:53.114591    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a87761b4-21ed-4470-ae67-77e1c811ee2f-proxy-tls\") pod \"error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f\" (UID: \"a87761b4-21ed-4470-ae67-77e1c811ee2f\") " pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f"
Jul 01 21:17:53.114753 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:17:53.114705    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-6396f-predictor-serving-cert: secret "error-404-isvc-6396f-predictor-serving-cert" not found
Jul 01 21:17:53.114830 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:17:53.114764    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a87761b4-21ed-4470-ae67-77e1c811ee2f-proxy-tls podName:a87761b4-21ed-4470-ae67-77e1c811ee2f nodeName:}" failed. No retries permitted until 2026-07-01 21:17:53.614749971 +0000 UTC m=+1873.290765516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a87761b4-21ed-4470-ae67-77e1c811ee2f-proxy-tls") pod "error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f" (UID: "a87761b4-21ed-4470-ae67-77e1c811ee2f") : secret "error-404-isvc-6396f-predictor-serving-cert" not found
Jul 01 21:17:53.115188 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:53.115170    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a87761b4-21ed-4470-ae67-77e1c811ee2f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f\" (UID: \"a87761b4-21ed-4470-ae67-77e1c811ee2f\") " pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f"
Jul 01 21:17:53.126764 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:53.126746    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4m6r\" (UniqueName: \"kubernetes.io/projected/a87761b4-21ed-4470-ae67-77e1c811ee2f-kube-api-access-f4m6r\") pod \"error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f\" (UID: \"a87761b4-21ed-4470-ae67-77e1c811ee2f\") " pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f"
Jul 01 21:17:53.518030 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:53.517995    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50527622-67cb-46d0-a3d1-95ab459aaae0-proxy-tls\") pod \"success-200-isvc-6396f-predictor-85b9f855cc-l99gt\" (UID: \"50527622-67cb-46d0-a3d1-95ab459aaae0\") " pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt"
Jul 01 21:17:53.520366 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:53.520346    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50527622-67cb-46d0-a3d1-95ab459aaae0-proxy-tls\") pod \"success-200-isvc-6396f-predictor-85b9f855cc-l99gt\" (UID: \"50527622-67cb-46d0-a3d1-95ab459aaae0\") " pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt"
Jul 01 21:17:53.617709 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:53.617679    2564 generic.go:356] "Generic (PLEG): container finished" podID="facd928f-2243-4d2e-addf-d9c156d21db4" containerID="63cebe879e3535d2687d5cb5ad022f033a96f5fad61d08a889251428a2413a79" exitCode=2
Jul 01 21:17:53.617884 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:53.617751    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw" event={"ID":"facd928f-2243-4d2e-addf-d9c156d21db4","Type":"ContainerDied","Data":"63cebe879e3535d2687d5cb5ad022f033a96f5fad61d08a889251428a2413a79"}
Jul 01 21:17:53.618317 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:53.618297    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a87761b4-21ed-4470-ae67-77e1c811ee2f-proxy-tls\") pod \"error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f\" (UID: \"a87761b4-21ed-4470-ae67-77e1c811ee2f\") " pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f"
Jul 01 21:17:53.619469 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:53.619450    2564 generic.go:356] "Generic (PLEG): container finished" podID="08d15b65-39b4-4b32-8c82-a4535bba8ee1" containerID="3a771138c3cc240d9644560ad4ea4f32e47688779d5e6996341628286aa5fb69" exitCode=2
Jul 01 21:17:53.619568 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:53.619505    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg" event={"ID":"08d15b65-39b4-4b32-8c82-a4535bba8ee1","Type":"ContainerDied","Data":"3a771138c3cc240d9644560ad4ea4f32e47688779d5e6996341628286aa5fb69"}
Jul 01 21:17:53.620792 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:53.620773    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a87761b4-21ed-4470-ae67-77e1c811ee2f-proxy-tls\") pod \"error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f\" (UID: \"a87761b4-21ed-4470-ae67-77e1c811ee2f\") " pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f"
Jul 01 21:17:53.770731 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:53.770654    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt"
Jul 01 21:17:53.892441 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:53.892405    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f"
Jul 01 21:17:53.902246 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:53.902217    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt"]
Jul 01 21:17:53.905498 ip-10-0-131-198 kubenswrapper[2564]: W0701 21:17:53.905470    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50527622_67cb_46d0_a3d1_95ab459aaae0.slice/crio-d2b58089369ea99c3d5c783dac81d20e21e9b990c48ecc0d68b145ba60e6cb7b WatchSource:0}: Error finding container d2b58089369ea99c3d5c783dac81d20e21e9b990c48ecc0d68b145ba60e6cb7b: Status 404 returned error can't find the container with id d2b58089369ea99c3d5c783dac81d20e21e9b990c48ecc0d68b145ba60e6cb7b
Jul 01 21:17:54.034036 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:54.034015    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f"]
Jul 01 21:17:54.036668 ip-10-0-131-198 kubenswrapper[2564]: W0701 21:17:54.036643    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87761b4_21ed_4470_ae67_77e1c811ee2f.slice/crio-1b331dfb3711218bd2f35437f7cba7274d569e765c7f6f22b4ce6f91d1682262 WatchSource:0}: Error finding container 1b331dfb3711218bd2f35437f7cba7274d569e765c7f6f22b4ce6f91d1682262: Status 404 returned error can't find the container with id 1b331dfb3711218bd2f35437f7cba7274d569e765c7f6f22b4ce6f91d1682262
Jul 01 21:17:54.625846 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:54.625804    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt" event={"ID":"50527622-67cb-46d0-a3d1-95ab459aaae0","Type":"ContainerStarted","Data":"58762811c13d00580234058033795936c17f695fd0e92410ae39ea457f479ee3"}
Jul 01 21:17:54.626063 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:54.625899    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt" event={"ID":"50527622-67cb-46d0-a3d1-95ab459aaae0","Type":"ContainerStarted","Data":"633fee731006a739dcf31627dea58196c6676dc1a19f919d302c987664566ee5"}
Jul 01 21:17:54.626063 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:54.625966    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt" event={"ID":"50527622-67cb-46d0-a3d1-95ab459aaae0","Type":"ContainerStarted","Data":"d2b58089369ea99c3d5c783dac81d20e21e9b990c48ecc0d68b145ba60e6cb7b"}
Jul 01 21:17:54.627655 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:54.627628    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f" event={"ID":"a87761b4-21ed-4470-ae67-77e1c811ee2f","Type":"ContainerStarted","Data":"2e3465c624443af7d8daf157377974cdd98aad5dc09528e7972357ceacafebca"}
Jul 01 21:17:54.627794 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:54.627681    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f" event={"ID":"a87761b4-21ed-4470-ae67-77e1c811ee2f","Type":"ContainerStarted","Data":"1cb0ed9bf436fbd0ff7959f9f89bf69413ada246bc4f19fde17068c7c5860fb8"}
Jul 01 21:17:54.627794 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:54.627702    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f" event={"ID":"a87761b4-21ed-4470-ae67-77e1c811ee2f","Type":"ContainerStarted","Data":"1b331dfb3711218bd2f35437f7cba7274d569e765c7f6f22b4ce6f91d1682262"}
Jul 01 21:17:54.627923 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:54.627818    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f"
Jul 01 21:17:54.627923 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:54.627883    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f"
Jul 01 21:17:54.629096 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:54.629073    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f" podUID="a87761b4-21ed-4470-ae67-77e1c811ee2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 01 21:17:54.649058 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:54.649018    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt" podStartSLOduration=2.649005838 podStartE2EDuration="2.649005838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:17:54.64739673 +0000 UTC m=+1874.323412297" watchObservedRunningTime="2026-07-01 21:17:54.649005838 +0000 UTC m=+1874.325021415"
Jul 01 21:17:54.667135 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:54.667097    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f" podStartSLOduration=2.667085496 podStartE2EDuration="2.667085496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:17:54.666619003 +0000 UTC m=+1874.342634581" watchObservedRunningTime="2026-07-01 21:17:54.667085496 +0000 UTC m=+1874.343101064"
Jul 01 21:17:55.272190 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:55.272156    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg" podUID="08d15b65-39b4-4b32-8c82-a4535bba8ee1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 01 21:17:55.272583 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:55.272166    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw" podUID="facd928f-2243-4d2e-addf-d9c156d21db4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 01 21:17:55.631821 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:55.631724    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f" podUID="a87761b4-21ed-4470-ae67-77e1c811ee2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 01 21:17:55.631821 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:55.631770    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt"
Jul 01 21:17:55.632074 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:55.631826    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt"
Jul 01 21:17:55.633038 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:55.633014    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt" podUID="50527622-67cb-46d0-a3d1-95ab459aaae0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 01 21:17:56.290903 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.290876    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg"
Jul 01 21:17:56.335904 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.335851    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08d15b65-39b4-4b32-8c82-a4535bba8ee1-kube-rbac-proxy-sar-config\") pod \"08d15b65-39b4-4b32-8c82-a4535bba8ee1\" (UID: \"08d15b65-39b4-4b32-8c82-a4535bba8ee1\") "
Jul 01 21:17:56.335904 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.335903    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08d15b65-39b4-4b32-8c82-a4535bba8ee1-proxy-tls\") pod \"08d15b65-39b4-4b32-8c82-a4535bba8ee1\" (UID: \"08d15b65-39b4-4b32-8c82-a4535bba8ee1\") "
Jul 01 21:17:56.336127 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.335932    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwvrw\" (UniqueName: \"kubernetes.io/projected/08d15b65-39b4-4b32-8c82-a4535bba8ee1-kube-api-access-fwvrw\") pod \"08d15b65-39b4-4b32-8c82-a4535bba8ee1\" (UID: \"08d15b65-39b4-4b32-8c82-a4535bba8ee1\") "
Jul 01 21:17:56.336289 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.336264    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08d15b65-39b4-4b32-8c82-a4535bba8ee1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "08d15b65-39b4-4b32-8c82-a4535bba8ee1" (UID: "08d15b65-39b4-4b32-8c82-a4535bba8ee1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:17:56.339848 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.339808    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08d15b65-39b4-4b32-8c82-a4535bba8ee1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "08d15b65-39b4-4b32-8c82-a4535bba8ee1" (UID: "08d15b65-39b4-4b32-8c82-a4535bba8ee1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:17:56.339988 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.339969    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08d15b65-39b4-4b32-8c82-a4535bba8ee1-kube-api-access-fwvrw" (OuterVolumeSpecName: "kube-api-access-fwvrw") pod "08d15b65-39b4-4b32-8c82-a4535bba8ee1" (UID: "08d15b65-39b4-4b32-8c82-a4535bba8ee1"). InnerVolumeSpecName "kube-api-access-fwvrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:17:56.361096 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.361075    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw"
Jul 01 21:17:56.437125 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.437096    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/facd928f-2243-4d2e-addf-d9c156d21db4-kube-rbac-proxy-sar-config\") pod \"facd928f-2243-4d2e-addf-d9c156d21db4\" (UID: \"facd928f-2243-4d2e-addf-d9c156d21db4\") "
Jul 01 21:17:56.437125 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.437127    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/facd928f-2243-4d2e-addf-d9c156d21db4-proxy-tls\") pod \"facd928f-2243-4d2e-addf-d9c156d21db4\" (UID: \"facd928f-2243-4d2e-addf-d9c156d21db4\") "
Jul 01 21:17:56.437287 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.437157    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6x6c\" (UniqueName: \"kubernetes.io/projected/facd928f-2243-4d2e-addf-d9c156d21db4-kube-api-access-k6x6c\") pod \"facd928f-2243-4d2e-addf-d9c156d21db4\" (UID: \"facd928f-2243-4d2e-addf-d9c156d21db4\") "
Jul 01 21:17:56.437403 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.437385    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08d15b65-39b4-4b32-8c82-a4535bba8ee1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:17:56.437451 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.437410    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08d15b65-39b4-4b32-8c82-a4535bba8ee1-proxy-tls\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:17:56.437451 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.437425    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwvrw\" (UniqueName: \"kubernetes.io/projected/08d15b65-39b4-4b32-8c82-a4535bba8ee1-kube-api-access-fwvrw\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:17:56.437517 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.437471    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/facd928f-2243-4d2e-addf-d9c156d21db4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "facd928f-2243-4d2e-addf-d9c156d21db4" (UID: "facd928f-2243-4d2e-addf-d9c156d21db4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:17:56.439150 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.439129    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/facd928f-2243-4d2e-addf-d9c156d21db4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "facd928f-2243-4d2e-addf-d9c156d21db4" (UID: "facd928f-2243-4d2e-addf-d9c156d21db4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:17:56.439230 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.439179    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/facd928f-2243-4d2e-addf-d9c156d21db4-kube-api-access-k6x6c" (OuterVolumeSpecName: "kube-api-access-k6x6c") pod "facd928f-2243-4d2e-addf-d9c156d21db4" (UID: "facd928f-2243-4d2e-addf-d9c156d21db4"). InnerVolumeSpecName "kube-api-access-k6x6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:17:56.538259 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.538218    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/facd928f-2243-4d2e-addf-d9c156d21db4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:17:56.538259 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.538257    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/facd928f-2243-4d2e-addf-d9c156d21db4-proxy-tls\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:17:56.538259 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.538271    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6x6c\" (UniqueName: \"kubernetes.io/projected/facd928f-2243-4d2e-addf-d9c156d21db4-kube-api-access-k6x6c\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:17:56.637129 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.637099    2564 generic.go:356] "Generic (PLEG): container finished" podID="facd928f-2243-4d2e-addf-d9c156d21db4" containerID="5faa8d16cea3f0fbb1c0177964439239074654473bb7b43074df137256c6ea90" exitCode=0
Jul 01 21:17:56.637268 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.637156    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw"
Jul 01 21:17:56.637268 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.637171    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw" event={"ID":"facd928f-2243-4d2e-addf-d9c156d21db4","Type":"ContainerDied","Data":"5faa8d16cea3f0fbb1c0177964439239074654473bb7b43074df137256c6ea90"}
Jul 01 21:17:56.637268 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.637235    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw" event={"ID":"facd928f-2243-4d2e-addf-d9c156d21db4","Type":"ContainerDied","Data":"e5b26777c4412af0a1b2c889a07886c96da51cd4c03910886e7b45cf1e003b0e"}
Jul 01 21:17:56.637391 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.637276    2564 scope.go:117] "RemoveContainer" containerID="63cebe879e3535d2687d5cb5ad022f033a96f5fad61d08a889251428a2413a79"
Jul 01 21:17:56.638700 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.638673    2564 generic.go:356] "Generic (PLEG): container finished" podID="08d15b65-39b4-4b32-8c82-a4535bba8ee1" containerID="aa32221886ce78d063ae04e03a7519d47e5b155a4374332d78a2cfa7f150363c" exitCode=0
Jul 01 21:17:56.638801 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.638734    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg"
Jul 01 21:17:56.638801 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.638740    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg" event={"ID":"08d15b65-39b4-4b32-8c82-a4535bba8ee1","Type":"ContainerDied","Data":"aa32221886ce78d063ae04e03a7519d47e5b155a4374332d78a2cfa7f150363c"}
Jul 01 21:17:56.638801 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.638788    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg" event={"ID":"08d15b65-39b4-4b32-8c82-a4535bba8ee1","Type":"ContainerDied","Data":"73fc2d8f73e55f316acd270baa0c5dcf9d046765fdcefad9360093214537af92"}
Jul 01 21:17:56.639234 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.639204    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt" podUID="50527622-67cb-46d0-a3d1-95ab459aaae0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 01 21:17:56.645766 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.645750    2564 scope.go:117] "RemoveContainer" containerID="5faa8d16cea3f0fbb1c0177964439239074654473bb7b43074df137256c6ea90"
Jul 01 21:17:56.653209 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.653193    2564 scope.go:117] "RemoveContainer" containerID="63cebe879e3535d2687d5cb5ad022f033a96f5fad61d08a889251428a2413a79"
Jul 01 21:17:56.653452 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:17:56.653430    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63cebe879e3535d2687d5cb5ad022f033a96f5fad61d08a889251428a2413a79\": container with ID starting with 63cebe879e3535d2687d5cb5ad022f033a96f5fad61d08a889251428a2413a79 not found: ID does not exist" containerID="63cebe879e3535d2687d5cb5ad022f033a96f5fad61d08a889251428a2413a79"
Jul 01 21:17:56.653537 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.653458    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63cebe879e3535d2687d5cb5ad022f033a96f5fad61d08a889251428a2413a79"} err="failed to get container status \"63cebe879e3535d2687d5cb5ad022f033a96f5fad61d08a889251428a2413a79\": rpc error: code = NotFound desc = could not find container \"63cebe879e3535d2687d5cb5ad022f033a96f5fad61d08a889251428a2413a79\": container with ID starting with 63cebe879e3535d2687d5cb5ad022f033a96f5fad61d08a889251428a2413a79 not found: ID does not exist"
Jul 01 21:17:56.653537 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.653474    2564 scope.go:117] "RemoveContainer" containerID="5faa8d16cea3f0fbb1c0177964439239074654473bb7b43074df137256c6ea90"
Jul 01 21:17:56.653713 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:17:56.653694    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5faa8d16cea3f0fbb1c0177964439239074654473bb7b43074df137256c6ea90\": container with ID starting with 5faa8d16cea3f0fbb1c0177964439239074654473bb7b43074df137256c6ea90 not found: ID does not exist" containerID="5faa8d16cea3f0fbb1c0177964439239074654473bb7b43074df137256c6ea90"
Jul 01 21:17:56.653775 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.653718    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5faa8d16cea3f0fbb1c0177964439239074654473bb7b43074df137256c6ea90"} err="failed to get container status \"5faa8d16cea3f0fbb1c0177964439239074654473bb7b43074df137256c6ea90\": rpc error: code = NotFound desc = could not find container \"5faa8d16cea3f0fbb1c0177964439239074654473bb7b43074df137256c6ea90\": container with ID starting with 5faa8d16cea3f0fbb1c0177964439239074654473bb7b43074df137256c6ea90 not found: ID does not exist"
Jul 01 21:17:56.653775 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.653733    2564 scope.go:117] "RemoveContainer" containerID="3a771138c3cc240d9644560ad4ea4f32e47688779d5e6996341628286aa5fb69"
Jul 01 21:17:56.660599 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.660584    2564 scope.go:117] "RemoveContainer" containerID="aa32221886ce78d063ae04e03a7519d47e5b155a4374332d78a2cfa7f150363c"
Jul 01 21:17:56.664675 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.664655    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg"]
Jul 01 21:17:56.667605 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.667559    2564 scope.go:117] "RemoveContainer" containerID="3a771138c3cc240d9644560ad4ea4f32e47688779d5e6996341628286aa5fb69"
Jul 01 21:17:56.668000 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:17:56.667975    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a771138c3cc240d9644560ad4ea4f32e47688779d5e6996341628286aa5fb69\": container with ID starting with 3a771138c3cc240d9644560ad4ea4f32e47688779d5e6996341628286aa5fb69 not found: ID does not exist" containerID="3a771138c3cc240d9644560ad4ea4f32e47688779d5e6996341628286aa5fb69"
Jul 01 21:17:56.668169 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.668003    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a771138c3cc240d9644560ad4ea4f32e47688779d5e6996341628286aa5fb69"} err="failed to get container status \"3a771138c3cc240d9644560ad4ea4f32e47688779d5e6996341628286aa5fb69\": rpc error: code = NotFound desc = could not find container \"3a771138c3cc240d9644560ad4ea4f32e47688779d5e6996341628286aa5fb69\": container with ID starting with 3a771138c3cc240d9644560ad4ea4f32e47688779d5e6996341628286aa5fb69 not found: ID does not exist"
Jul 01 21:17:56.668169 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.668019    2564 scope.go:117] "RemoveContainer" containerID="aa32221886ce78d063ae04e03a7519d47e5b155a4374332d78a2cfa7f150363c"
Jul 01 21:17:56.668523 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:17:56.668276    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa32221886ce78d063ae04e03a7519d47e5b155a4374332d78a2cfa7f150363c\": container with ID starting with aa32221886ce78d063ae04e03a7519d47e5b155a4374332d78a2cfa7f150363c not found: ID does not exist" containerID="aa32221886ce78d063ae04e03a7519d47e5b155a4374332d78a2cfa7f150363c"
Jul 01 21:17:56.668523 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.668301    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa32221886ce78d063ae04e03a7519d47e5b155a4374332d78a2cfa7f150363c"} err="failed to get container status \"aa32221886ce78d063ae04e03a7519d47e5b155a4374332d78a2cfa7f150363c\": rpc error: code = NotFound desc = could not find container \"aa32221886ce78d063ae04e03a7519d47e5b155a4374332d78a2cfa7f150363c\": container with ID starting with aa32221886ce78d063ae04e03a7519d47e5b155a4374332d78a2cfa7f150363c not found: ID does not exist"
Jul 01 21:17:56.669991 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.669974    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3b58c-predictor-66bc46db55-g6zxg"]
Jul 01 21:17:56.682695 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.682678    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw"]
Jul 01 21:17:56.689723 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.689702    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3b58c-predictor-5cc76b4998-dnlrw"]
Jul 01 21:17:56.762473 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.759410    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08d15b65-39b4-4b32-8c82-a4535bba8ee1" path="/var/lib/kubelet/pods/08d15b65-39b4-4b32-8c82-a4535bba8ee1/volumes"
Jul 01 21:17:56.762473 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:17:56.760071    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="facd928f-2243-4d2e-addf-d9c156d21db4" path="/var/lib/kubelet/pods/facd928f-2243-4d2e-addf-d9c156d21db4/volumes"
Jul 01 21:18:00.636123 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:18:00.636093    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f"
Jul 01 21:18:00.636606 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:18:00.636583    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f" podUID="a87761b4-21ed-4470-ae67-77e1c811ee2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 01 21:18:01.643479 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:18:01.643450    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt"
Jul 01 21:18:01.643880 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:18:01.643825    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt" podUID="50527622-67cb-46d0-a3d1-95ab459aaae0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 01 21:18:10.637301 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:18:10.637262    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f" podUID="a87761b4-21ed-4470-ae67-77e1c811ee2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 01 21:18:11.644852 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:18:11.644814    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt" podUID="50527622-67cb-46d0-a3d1-95ab459aaae0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 01 21:18:20.637328 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:18:20.637286    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f" podUID="a87761b4-21ed-4470-ae67-77e1c811ee2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 01 21:18:21.644662 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:18:21.644621    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt" podUID="50527622-67cb-46d0-a3d1-95ab459aaae0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 01 21:18:30.637162 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:18:30.637123    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f" podUID="a87761b4-21ed-4470-ae67-77e1c811ee2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 01 21:18:31.644071 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:18:31.644033    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt" podUID="50527622-67cb-46d0-a3d1-95ab459aaae0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 01 21:18:40.638047 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:18:40.638015    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f"
Jul 01 21:18:41.643999 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:18:41.643975    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt"
Jul 01 21:21:40.872430 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:21:40.872406    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/ovn-acl-logging/0.log"
Jul 01 21:21:40.874930 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:21:40.874729    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/ovn-acl-logging/0.log"
Jul 01 21:26:40.897961 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:26:40.897937    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/ovn-acl-logging/0.log"
Jul 01 21:26:40.900495 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:26:40.900319    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/ovn-acl-logging/0.log"
Jul 01 21:27:07.839310 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:07.839227    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt"]
Jul 01 21:27:07.839828 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:07.839497    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt" podUID="50527622-67cb-46d0-a3d1-95ab459aaae0" containerName="kserve-container" containerID="cri-o://633fee731006a739dcf31627dea58196c6676dc1a19f919d302c987664566ee5" gracePeriod=30
Jul 01 21:27:07.839828 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:07.839523    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt" podUID="50527622-67cb-46d0-a3d1-95ab459aaae0" containerName="kube-rbac-proxy" containerID="cri-o://58762811c13d00580234058033795936c17f695fd0e92410ae39ea457f479ee3" gracePeriod=30
Jul 01 21:27:07.882042 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:07.882008    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f"]
Jul 01 21:27:07.882316 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:07.882275    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f" podUID="a87761b4-21ed-4470-ae67-77e1c811ee2f" containerName="kserve-container" containerID="cri-o://1cb0ed9bf436fbd0ff7959f9f89bf69413ada246bc4f19fde17068c7c5860fb8" gracePeriod=30
Jul 01 21:27:07.882405 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:07.882296    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f" podUID="a87761b4-21ed-4470-ae67-77e1c811ee2f" containerName="kube-rbac-proxy" containerID="cri-o://2e3465c624443af7d8daf157377974cdd98aad5dc09528e7972357ceacafebca" gracePeriod=30
Jul 01 21:27:08.823921 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:08.823889    2564 generic.go:356] "Generic (PLEG): container finished" podID="a87761b4-21ed-4470-ae67-77e1c811ee2f" containerID="2e3465c624443af7d8daf157377974cdd98aad5dc09528e7972357ceacafebca" exitCode=2
Jul 01 21:27:08.824098 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:08.823962    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f" event={"ID":"a87761b4-21ed-4470-ae67-77e1c811ee2f","Type":"ContainerDied","Data":"2e3465c624443af7d8daf157377974cdd98aad5dc09528e7972357ceacafebca"}
Jul 01 21:27:08.825565 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:08.825548    2564 generic.go:356] "Generic (PLEG): container finished" podID="50527622-67cb-46d0-a3d1-95ab459aaae0" containerID="58762811c13d00580234058033795936c17f695fd0e92410ae39ea457f479ee3" exitCode=2
Jul 01 21:27:08.825662 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:08.825575    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt" event={"ID":"50527622-67cb-46d0-a3d1-95ab459aaae0","Type":"ContainerDied","Data":"58762811c13d00580234058033795936c17f695fd0e92410ae39ea457f479ee3"}
Jul 01 21:27:10.632290 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:10.632244    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f" podUID="a87761b4-21ed-4470-ae67-77e1c811ee2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 01 21:27:10.636517 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:10.636494    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f" podUID="a87761b4-21ed-4470-ae67-77e1c811ee2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 01 21:27:10.836552 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:10.836520    2564 generic.go:356] "Generic (PLEG): container finished" podID="50527622-67cb-46d0-a3d1-95ab459aaae0" containerID="633fee731006a739dcf31627dea58196c6676dc1a19f919d302c987664566ee5" exitCode=0
Jul 01 21:27:10.836715 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:10.836560    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt" event={"ID":"50527622-67cb-46d0-a3d1-95ab459aaae0","Type":"ContainerDied","Data":"633fee731006a739dcf31627dea58196c6676dc1a19f919d302c987664566ee5"}
Jul 01 21:27:10.838569 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:10.838538    2564 generic.go:356] "Generic (PLEG): container finished" podID="a87761b4-21ed-4470-ae67-77e1c811ee2f" containerID="1cb0ed9bf436fbd0ff7959f9f89bf69413ada246bc4f19fde17068c7c5860fb8" exitCode=0
Jul 01 21:27:10.838693 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:10.838605    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f" event={"ID":"a87761b4-21ed-4470-ae67-77e1c811ee2f","Type":"ContainerDied","Data":"1cb0ed9bf436fbd0ff7959f9f89bf69413ada246bc4f19fde17068c7c5860fb8"}
Jul 01 21:27:10.945255 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:10.945235    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt"
Jul 01 21:27:10.947987 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:10.947970    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f"
Jul 01 21:27:11.089614 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.089585    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50527622-67cb-46d0-a3d1-95ab459aaae0-kube-rbac-proxy-sar-config\") pod \"50527622-67cb-46d0-a3d1-95ab459aaae0\" (UID: \"50527622-67cb-46d0-a3d1-95ab459aaae0\") "
Jul 01 21:27:11.089788 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.089628    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a87761b4-21ed-4470-ae67-77e1c811ee2f-kube-rbac-proxy-sar-config\") pod \"a87761b4-21ed-4470-ae67-77e1c811ee2f\" (UID: \"a87761b4-21ed-4470-ae67-77e1c811ee2f\") "
Jul 01 21:27:11.089788 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.089663    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttm8z\" (UniqueName: \"kubernetes.io/projected/50527622-67cb-46d0-a3d1-95ab459aaae0-kube-api-access-ttm8z\") pod \"50527622-67cb-46d0-a3d1-95ab459aaae0\" (UID: \"50527622-67cb-46d0-a3d1-95ab459aaae0\") "
Jul 01 21:27:11.089788 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.089690    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50527622-67cb-46d0-a3d1-95ab459aaae0-proxy-tls\") pod \"50527622-67cb-46d0-a3d1-95ab459aaae0\" (UID: \"50527622-67cb-46d0-a3d1-95ab459aaae0\") "
Jul 01 21:27:11.089788 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.089756    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4m6r\" (UniqueName: \"kubernetes.io/projected/a87761b4-21ed-4470-ae67-77e1c811ee2f-kube-api-access-f4m6r\") pod \"a87761b4-21ed-4470-ae67-77e1c811ee2f\" (UID: \"a87761b4-21ed-4470-ae67-77e1c811ee2f\") "
Jul 01 21:27:11.090059 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.089919    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a87761b4-21ed-4470-ae67-77e1c811ee2f-proxy-tls\") pod \"a87761b4-21ed-4470-ae67-77e1c811ee2f\" (UID: \"a87761b4-21ed-4470-ae67-77e1c811ee2f\") "
Jul 01 21:27:11.090112 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.090078    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50527622-67cb-46d0-a3d1-95ab459aaae0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "50527622-67cb-46d0-a3d1-95ab459aaae0" (UID: "50527622-67cb-46d0-a3d1-95ab459aaae0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:27:11.090112 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.090083    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a87761b4-21ed-4470-ae67-77e1c811ee2f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a87761b4-21ed-4470-ae67-77e1c811ee2f" (UID: "a87761b4-21ed-4470-ae67-77e1c811ee2f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:27:11.090267 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.090244    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50527622-67cb-46d0-a3d1-95ab459aaae0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:27:11.090267 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.090268    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a87761b4-21ed-4470-ae67-77e1c811ee2f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:27:11.091914 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.091888    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50527622-67cb-46d0-a3d1-95ab459aaae0-kube-api-access-ttm8z" (OuterVolumeSpecName: "kube-api-access-ttm8z") pod "50527622-67cb-46d0-a3d1-95ab459aaae0" (UID: "50527622-67cb-46d0-a3d1-95ab459aaae0"). InnerVolumeSpecName "kube-api-access-ttm8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:27:11.092011 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.091951    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a87761b4-21ed-4470-ae67-77e1c811ee2f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a87761b4-21ed-4470-ae67-77e1c811ee2f" (UID: "a87761b4-21ed-4470-ae67-77e1c811ee2f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:27:11.092160 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.092146    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50527622-67cb-46d0-a3d1-95ab459aaae0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "50527622-67cb-46d0-a3d1-95ab459aaae0" (UID: "50527622-67cb-46d0-a3d1-95ab459aaae0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:27:11.092241 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.092226    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a87761b4-21ed-4470-ae67-77e1c811ee2f-kube-api-access-f4m6r" (OuterVolumeSpecName: "kube-api-access-f4m6r") pod "a87761b4-21ed-4470-ae67-77e1c811ee2f" (UID: "a87761b4-21ed-4470-ae67-77e1c811ee2f"). InnerVolumeSpecName "kube-api-access-f4m6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:27:11.190965 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.190912    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttm8z\" (UniqueName: \"kubernetes.io/projected/50527622-67cb-46d0-a3d1-95ab459aaae0-kube-api-access-ttm8z\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:27:11.190965 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.190931    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50527622-67cb-46d0-a3d1-95ab459aaae0-proxy-tls\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:27:11.190965 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.190941    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4m6r\" (UniqueName: \"kubernetes.io/projected/a87761b4-21ed-4470-ae67-77e1c811ee2f-kube-api-access-f4m6r\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:27:11.190965 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.190950    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a87761b4-21ed-4470-ae67-77e1c811ee2f-proxy-tls\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:27:11.844283 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.844257    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f"
Jul 01 21:27:11.844716 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.844276    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f" event={"ID":"a87761b4-21ed-4470-ae67-77e1c811ee2f","Type":"ContainerDied","Data":"1b331dfb3711218bd2f35437f7cba7274d569e765c7f6f22b4ce6f91d1682262"}
Jul 01 21:27:11.844716 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.844342    2564 scope.go:117] "RemoveContainer" containerID="2e3465c624443af7d8daf157377974cdd98aad5dc09528e7972357ceacafebca"
Jul 01 21:27:11.845989 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.845964    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt" event={"ID":"50527622-67cb-46d0-a3d1-95ab459aaae0","Type":"ContainerDied","Data":"d2b58089369ea99c3d5c783dac81d20e21e9b990c48ecc0d68b145ba60e6cb7b"}
Jul 01 21:27:11.846111 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.846014    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt"
Jul 01 21:27:11.853074 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.853056    2564 scope.go:117] "RemoveContainer" containerID="1cb0ed9bf436fbd0ff7959f9f89bf69413ada246bc4f19fde17068c7c5860fb8"
Jul 01 21:27:11.860954 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.860935    2564 scope.go:117] "RemoveContainer" containerID="58762811c13d00580234058033795936c17f695fd0e92410ae39ea457f479ee3"
Jul 01 21:27:11.868723 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.868710    2564 scope.go:117] "RemoveContainer" containerID="633fee731006a739dcf31627dea58196c6676dc1a19f919d302c987664566ee5"
Jul 01 21:27:11.869744 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.869722    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt"]
Jul 01 21:27:11.873185 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.873166    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6396f-predictor-85b9f855cc-l99gt"]
Jul 01 21:27:11.884052 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.884033    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f"]
Jul 01 21:27:11.887900 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:11.887869    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6396f-predictor-77fbbbfd5b-g4m9f"]
Jul 01 21:27:12.758067 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:12.758041    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50527622-67cb-46d0-a3d1-95ab459aaae0" path="/var/lib/kubelet/pods/50527622-67cb-46d0-a3d1-95ab459aaae0/volumes"
Jul 01 21:27:12.758450 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:27:12.758437    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a87761b4-21ed-4470-ae67-77e1c811ee2f" path="/var/lib/kubelet/pods/a87761b4-21ed-4470-ae67-77e1c811ee2f/volumes"
Jul 01 21:31:40.927187 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:31:40.927088    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/ovn-acl-logging/0.log"
Jul 01 21:31:40.932195 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:31:40.932177    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/ovn-acl-logging/0.log"
Jul 01 21:34:21.656061 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:21.656028    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2"]
Jul 01 21:34:21.656652 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:21.656327    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2" podUID="8475069f-ef34-4979-9c40-9dcb2723997b" containerName="kserve-container" containerID="cri-o://93bd7d4e3ab30f1c627d39344feca687951e653618d60563a3003f1287a8e2a5" gracePeriod=30
Jul 01 21:34:21.656652 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:21.656349    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2" podUID="8475069f-ef34-4979-9c40-9dcb2723997b" containerName="kube-rbac-proxy" containerID="cri-o://2587094e3cbf813666dc9fe7188355fa989cdb2bdebdce0bbea06ef74bec148b" gracePeriod=30
Jul 01 21:34:21.710386 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:21.710356    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m"]
Jul 01 21:34:21.710634 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:21.710588    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m" podUID="d60c38a1-a61b-47d6-8da7-8584afb9c4b9" containerName="kserve-container" containerID="cri-o://d5a90d746799165656c18cd82f1884c687dd7b11e2d2b15365eabc624f7c7c1d" gracePeriod=30
Jul 01 21:34:21.710751 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:21.710609    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m" podUID="d60c38a1-a61b-47d6-8da7-8584afb9c4b9" containerName="kube-rbac-proxy" containerID="cri-o://59123239647b734b876976c22041f087190a5664c07198563242d9a284a8720d" gracePeriod=30
Jul 01 21:34:22.457154 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.457121    2564 generic.go:356] "Generic (PLEG): container finished" podID="8475069f-ef34-4979-9c40-9dcb2723997b" containerID="2587094e3cbf813666dc9fe7188355fa989cdb2bdebdce0bbea06ef74bec148b" exitCode=2
Jul 01 21:34:22.457340 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.457196    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2" event={"ID":"8475069f-ef34-4979-9c40-9dcb2723997b","Type":"ContainerDied","Data":"2587094e3cbf813666dc9fe7188355fa989cdb2bdebdce0bbea06ef74bec148b"}
Jul 01 21:34:22.458669 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.458650    2564 generic.go:356] "Generic (PLEG): container finished" podID="d60c38a1-a61b-47d6-8da7-8584afb9c4b9" containerID="59123239647b734b876976c22041f087190a5664c07198563242d9a284a8720d" exitCode=2
Jul 01 21:34:22.458770 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.458683    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m" event={"ID":"d60c38a1-a61b-47d6-8da7-8584afb9c4b9","Type":"ContainerDied","Data":"59123239647b734b876976c22041f087190a5664c07198563242d9a284a8720d"}
Jul 01 21:34:22.769149 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.769061    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jw56q/must-gather-ddjdb"]
Jul 01 21:34:22.769582 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.769533    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a87761b4-21ed-4470-ae67-77e1c811ee2f" containerName="kserve-container"
Jul 01 21:34:22.769582 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.769550    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a87761b4-21ed-4470-ae67-77e1c811ee2f" containerName="kserve-container"
Jul 01 21:34:22.769582 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.769567    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="facd928f-2243-4d2e-addf-d9c156d21db4" containerName="kube-rbac-proxy"
Jul 01 21:34:22.769582 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.769575    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="facd928f-2243-4d2e-addf-d9c156d21db4" containerName="kube-rbac-proxy"
Jul 01 21:34:22.769811 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.769584    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08d15b65-39b4-4b32-8c82-a4535bba8ee1" containerName="kserve-container"
Jul 01 21:34:22.769811 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.769594    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="08d15b65-39b4-4b32-8c82-a4535bba8ee1" containerName="kserve-container"
Jul 01 21:34:22.769811 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.769611    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50527622-67cb-46d0-a3d1-95ab459aaae0" containerName="kserve-container"
Jul 01 21:34:22.769811 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.769619    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="50527622-67cb-46d0-a3d1-95ab459aaae0" containerName="kserve-container"
Jul 01 21:34:22.769811 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.769631    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a87761b4-21ed-4470-ae67-77e1c811ee2f" containerName="kube-rbac-proxy"
Jul 01 21:34:22.769811 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.769639    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a87761b4-21ed-4470-ae67-77e1c811ee2f" containerName="kube-rbac-proxy"
Jul 01 21:34:22.769811 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.769651    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08d15b65-39b4-4b32-8c82-a4535bba8ee1" containerName="kube-rbac-proxy"
Jul 01 21:34:22.769811 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.769659    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="08d15b65-39b4-4b32-8c82-a4535bba8ee1" containerName="kube-rbac-proxy"
Jul 01 21:34:22.769811 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.769669    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50527622-67cb-46d0-a3d1-95ab459aaae0" containerName="kube-rbac-proxy"
Jul 01 21:34:22.769811 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.769677    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="50527622-67cb-46d0-a3d1-95ab459aaae0" containerName="kube-rbac-proxy"
Jul 01 21:34:22.769811 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.769693    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="facd928f-2243-4d2e-addf-d9c156d21db4" containerName="kserve-container"
Jul 01 21:34:22.769811 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.769701    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="facd928f-2243-4d2e-addf-d9c156d21db4" containerName="kserve-container"
Jul 01 21:34:22.769811 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.769772    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="50527622-67cb-46d0-a3d1-95ab459aaae0" containerName="kube-rbac-proxy"
Jul 01 21:34:22.769811 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.769783    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="08d15b65-39b4-4b32-8c82-a4535bba8ee1" containerName="kserve-container"
Jul 01 21:34:22.769811 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.769794    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="08d15b65-39b4-4b32-8c82-a4535bba8ee1" containerName="kube-rbac-proxy"
Jul 01 21:34:22.769811 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.769802    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="50527622-67cb-46d0-a3d1-95ab459aaae0" containerName="kserve-container"
Jul 01 21:34:22.769811 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.769811    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a87761b4-21ed-4470-ae67-77e1c811ee2f" containerName="kube-rbac-proxy"
Jul 01 21:34:22.769811 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.769819    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="facd928f-2243-4d2e-addf-d9c156d21db4" containerName="kserve-container"
Jul 01 21:34:22.770658 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.769826    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a87761b4-21ed-4470-ae67-77e1c811ee2f" containerName="kserve-container"
Jul 01 21:34:22.770658 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.769834    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="facd928f-2243-4d2e-addf-d9c156d21db4" containerName="kube-rbac-proxy"
Jul 01 21:34:22.774527 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.774019    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jw56q/must-gather-ddjdb"
Jul 01 21:34:22.778242 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.778215    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jw56q\"/\"kube-root-ca.crt\""
Jul 01 21:34:22.779742 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.779704    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jw56q\"/\"openshift-service-ca.crt\""
Jul 01 21:34:22.779742 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.779711    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jw56q\"/\"default-dockercfg-sn98p\""
Jul 01 21:34:22.781568 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.781547    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jw56q/must-gather-ddjdb"]
Jul 01 21:34:22.855904 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.855847    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6sbq\" (UniqueName: \"kubernetes.io/projected/759ede6b-4b0d-421a-8b8b-855994182882-kube-api-access-c6sbq\") pod \"must-gather-ddjdb\" (UID: \"759ede6b-4b0d-421a-8b8b-855994182882\") " pod="openshift-must-gather-jw56q/must-gather-ddjdb"
Jul 01 21:34:22.856066 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.855911    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/759ede6b-4b0d-421a-8b8b-855994182882-must-gather-output\") pod \"must-gather-ddjdb\" (UID: \"759ede6b-4b0d-421a-8b8b-855994182882\") " pod="openshift-must-gather-jw56q/must-gather-ddjdb"
Jul 01 21:34:22.956788 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.956747    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6sbq\" (UniqueName: \"kubernetes.io/projected/759ede6b-4b0d-421a-8b8b-855994182882-kube-api-access-c6sbq\") pod \"must-gather-ddjdb\" (UID: \"759ede6b-4b0d-421a-8b8b-855994182882\") " pod="openshift-must-gather-jw56q/must-gather-ddjdb"
Jul 01 21:34:22.956788 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.956793    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/759ede6b-4b0d-421a-8b8b-855994182882-must-gather-output\") pod \"must-gather-ddjdb\" (UID: \"759ede6b-4b0d-421a-8b8b-855994182882\") " pod="openshift-must-gather-jw56q/must-gather-ddjdb"
Jul 01 21:34:22.957119 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.957104    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/759ede6b-4b0d-421a-8b8b-855994182882-must-gather-output\") pod \"must-gather-ddjdb\" (UID: \"759ede6b-4b0d-421a-8b8b-855994182882\") " pod="openshift-must-gather-jw56q/must-gather-ddjdb"
Jul 01 21:34:22.965194 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:22.965170    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6sbq\" (UniqueName: \"kubernetes.io/projected/759ede6b-4b0d-421a-8b8b-855994182882-kube-api-access-c6sbq\") pod \"must-gather-ddjdb\" (UID: \"759ede6b-4b0d-421a-8b8b-855994182882\") " pod="openshift-must-gather-jw56q/must-gather-ddjdb"
Jul 01 21:34:23.094541 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:23.094451    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jw56q/must-gather-ddjdb"
Jul 01 21:34:23.221870 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:23.221821    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jw56q/must-gather-ddjdb"]
Jul 01 21:34:23.224768 ip-10-0-131-198 kubenswrapper[2564]: W0701 21:34:23.224745    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod759ede6b_4b0d_421a_8b8b_855994182882.slice/crio-7e4f31fa680e42bc24e5315aab8b59136ae7c82d4532495f7a12ef8ddf6d2de9 WatchSource:0}: Error finding container 7e4f31fa680e42bc24e5315aab8b59136ae7c82d4532495f7a12ef8ddf6d2de9: Status 404 returned error can't find the container with id 7e4f31fa680e42bc24e5315aab8b59136ae7c82d4532495f7a12ef8ddf6d2de9
Jul 01 21:34:23.226455 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:23.226442    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 21:34:23.463162 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:23.463126    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jw56q/must-gather-ddjdb" event={"ID":"759ede6b-4b0d-421a-8b8b-855994182882","Type":"ContainerStarted","Data":"7e4f31fa680e42bc24e5315aab8b59136ae7c82d4532495f7a12ef8ddf6d2de9"}
Jul 01 21:34:24.363008 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:24.362704    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2" podUID="8475069f-ef34-4979-9c40-9dcb2723997b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 01 21:34:25.171188 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.170918    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m"
Jul 01 21:34:25.174523 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.174501    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2"
Jul 01 21:34:25.278793 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.278758    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8475069f-ef34-4979-9c40-9dcb2723997b-kube-rbac-proxy-sar-config\") pod \"8475069f-ef34-4979-9c40-9dcb2723997b\" (UID: \"8475069f-ef34-4979-9c40-9dcb2723997b\") "
Jul 01 21:34:25.279001 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.278851    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tb8pc\" (UniqueName: \"kubernetes.io/projected/8475069f-ef34-4979-9c40-9dcb2723997b-kube-api-access-tb8pc\") pod \"8475069f-ef34-4979-9c40-9dcb2723997b\" (UID: \"8475069f-ef34-4979-9c40-9dcb2723997b\") "
Jul 01 21:34:25.279001 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.278985    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d60c38a1-a61b-47d6-8da7-8584afb9c4b9-proxy-tls\") pod \"d60c38a1-a61b-47d6-8da7-8584afb9c4b9\" (UID: \"d60c38a1-a61b-47d6-8da7-8584afb9c4b9\") "
Jul 01 21:34:25.279121 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.279021    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8475069f-ef34-4979-9c40-9dcb2723997b-proxy-tls\") pod \"8475069f-ef34-4979-9c40-9dcb2723997b\" (UID: \"8475069f-ef34-4979-9c40-9dcb2723997b\") "
Jul 01 21:34:25.279121 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.279079    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb5qd\" (UniqueName: \"kubernetes.io/projected/d60c38a1-a61b-47d6-8da7-8584afb9c4b9-kube-api-access-cb5qd\") pod \"d60c38a1-a61b-47d6-8da7-8584afb9c4b9\" (UID: \"d60c38a1-a61b-47d6-8da7-8584afb9c4b9\") "
Jul 01 21:34:25.279121 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.279113    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d60c38a1-a61b-47d6-8da7-8584afb9c4b9-kube-rbac-proxy-sar-config\") pod \"d60c38a1-a61b-47d6-8da7-8584afb9c4b9\" (UID: \"d60c38a1-a61b-47d6-8da7-8584afb9c4b9\") "
Jul 01 21:34:25.279407 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.279363    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8475069f-ef34-4979-9c40-9dcb2723997b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8475069f-ef34-4979-9c40-9dcb2723997b" (UID: "8475069f-ef34-4979-9c40-9dcb2723997b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:34:25.279693 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.279669    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d60c38a1-a61b-47d6-8da7-8584afb9c4b9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d60c38a1-a61b-47d6-8da7-8584afb9c4b9" (UID: "d60c38a1-a61b-47d6-8da7-8584afb9c4b9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 21:34:25.281874 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.281834    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8475069f-ef34-4979-9c40-9dcb2723997b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8475069f-ef34-4979-9c40-9dcb2723997b" (UID: "8475069f-ef34-4979-9c40-9dcb2723997b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:34:25.282180 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.282143    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8475069f-ef34-4979-9c40-9dcb2723997b-kube-api-access-tb8pc" (OuterVolumeSpecName: "kube-api-access-tb8pc") pod "8475069f-ef34-4979-9c40-9dcb2723997b" (UID: "8475069f-ef34-4979-9c40-9dcb2723997b"). InnerVolumeSpecName "kube-api-access-tb8pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:34:25.282431 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.282405    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d60c38a1-a61b-47d6-8da7-8584afb9c4b9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d60c38a1-a61b-47d6-8da7-8584afb9c4b9" (UID: "d60c38a1-a61b-47d6-8da7-8584afb9c4b9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 21:34:25.282674 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.282646    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d60c38a1-a61b-47d6-8da7-8584afb9c4b9-kube-api-access-cb5qd" (OuterVolumeSpecName: "kube-api-access-cb5qd") pod "d60c38a1-a61b-47d6-8da7-8584afb9c4b9" (UID: "d60c38a1-a61b-47d6-8da7-8584afb9c4b9"). InnerVolumeSpecName "kube-api-access-cb5qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:34:25.380466 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.380426    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8475069f-ef34-4979-9c40-9dcb2723997b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:34:25.380466 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.380470    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb8pc\" (UniqueName: \"kubernetes.io/projected/8475069f-ef34-4979-9c40-9dcb2723997b-kube-api-access-tb8pc\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:34:25.381012 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.380487    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d60c38a1-a61b-47d6-8da7-8584afb9c4b9-proxy-tls\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:34:25.381012 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.380501    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8475069f-ef34-4979-9c40-9dcb2723997b-proxy-tls\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:34:25.381012 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.380515    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb5qd\" (UniqueName: \"kubernetes.io/projected/d60c38a1-a61b-47d6-8da7-8584afb9c4b9-kube-api-access-cb5qd\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:34:25.381012 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.380528    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d60c38a1-a61b-47d6-8da7-8584afb9c4b9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:34:25.476696 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.476604    2564 generic.go:356] "Generic (PLEG): container finished" podID="d60c38a1-a61b-47d6-8da7-8584afb9c4b9" containerID="d5a90d746799165656c18cd82f1884c687dd7b11e2d2b15365eabc624f7c7c1d" exitCode=0
Jul 01 21:34:25.476848 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.476747    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m" event={"ID":"d60c38a1-a61b-47d6-8da7-8584afb9c4b9","Type":"ContainerDied","Data":"d5a90d746799165656c18cd82f1884c687dd7b11e2d2b15365eabc624f7c7c1d"}
Jul 01 21:34:25.476848 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.476806    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m" event={"ID":"d60c38a1-a61b-47d6-8da7-8584afb9c4b9","Type":"ContainerDied","Data":"2db226583b8561f222784ed39dfc260813b04ca8fcd8a255039cd253b86ad651"}
Jul 01 21:34:25.476848 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.476840    2564 scope.go:117] "RemoveContainer" containerID="59123239647b734b876976c22041f087190a5664c07198563242d9a284a8720d"
Jul 01 21:34:25.477040 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.477024    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m"
Jul 01 21:34:25.480423 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.480390    2564 generic.go:356] "Generic (PLEG): container finished" podID="8475069f-ef34-4979-9c40-9dcb2723997b" containerID="93bd7d4e3ab30f1c627d39344feca687951e653618d60563a3003f1287a8e2a5" exitCode=0
Jul 01 21:34:25.480564 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.480449    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2" event={"ID":"8475069f-ef34-4979-9c40-9dcb2723997b","Type":"ContainerDied","Data":"93bd7d4e3ab30f1c627d39344feca687951e653618d60563a3003f1287a8e2a5"}
Jul 01 21:34:25.480564 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.480503    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2" event={"ID":"8475069f-ef34-4979-9c40-9dcb2723997b","Type":"ContainerDied","Data":"6e73e922a64a14abe22ef4791543b4748772a03526677dd2768a6bbbac7b310f"}
Jul 01 21:34:25.480564 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.480453    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2"
Jul 01 21:34:25.488237 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.488213    2564 scope.go:117] "RemoveContainer" containerID="d5a90d746799165656c18cd82f1884c687dd7b11e2d2b15365eabc624f7c7c1d"
Jul 01 21:34:25.497677 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.497658    2564 scope.go:117] "RemoveContainer" containerID="59123239647b734b876976c22041f087190a5664c07198563242d9a284a8720d"
Jul 01 21:34:25.497977 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:34:25.497952    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59123239647b734b876976c22041f087190a5664c07198563242d9a284a8720d\": container with ID starting with 59123239647b734b876976c22041f087190a5664c07198563242d9a284a8720d not found: ID does not exist" containerID="59123239647b734b876976c22041f087190a5664c07198563242d9a284a8720d"
Jul 01 21:34:25.498064 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.497987    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59123239647b734b876976c22041f087190a5664c07198563242d9a284a8720d"} err="failed to get container status \"59123239647b734b876976c22041f087190a5664c07198563242d9a284a8720d\": rpc error: code = NotFound desc = could not find container \"59123239647b734b876976c22041f087190a5664c07198563242d9a284a8720d\": container with ID starting with 59123239647b734b876976c22041f087190a5664c07198563242d9a284a8720d not found: ID does not exist"
Jul 01 21:34:25.498064 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.498011    2564 scope.go:117] "RemoveContainer" containerID="d5a90d746799165656c18cd82f1884c687dd7b11e2d2b15365eabc624f7c7c1d"
Jul 01 21:34:25.498276 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:34:25.498252    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5a90d746799165656c18cd82f1884c687dd7b11e2d2b15365eabc624f7c7c1d\": container with ID starting with d5a90d746799165656c18cd82f1884c687dd7b11e2d2b15365eabc624f7c7c1d not found: ID does not exist" containerID="d5a90d746799165656c18cd82f1884c687dd7b11e2d2b15365eabc624f7c7c1d"
Jul 01 21:34:25.498349 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.498285    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5a90d746799165656c18cd82f1884c687dd7b11e2d2b15365eabc624f7c7c1d"} err="failed to get container status \"d5a90d746799165656c18cd82f1884c687dd7b11e2d2b15365eabc624f7c7c1d\": rpc error: code = NotFound desc = could not find container \"d5a90d746799165656c18cd82f1884c687dd7b11e2d2b15365eabc624f7c7c1d\": container with ID starting with d5a90d746799165656c18cd82f1884c687dd7b11e2d2b15365eabc624f7c7c1d not found: ID does not exist"
Jul 01 21:34:25.498349 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.498303    2564 scope.go:117] "RemoveContainer" containerID="2587094e3cbf813666dc9fe7188355fa989cdb2bdebdce0bbea06ef74bec148b"
Jul 01 21:34:25.503700 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.503675    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m"]
Jul 01 21:34:25.507208 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.507176    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ce993-predictor-59dfbf46cd-d8h5m"]
Jul 01 21:34:25.507518 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.507501    2564 scope.go:117] "RemoveContainer" containerID="93bd7d4e3ab30f1c627d39344feca687951e653618d60563a3003f1287a8e2a5"
Jul 01 21:34:25.518638 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.518383    2564 scope.go:117] "RemoveContainer" containerID="2587094e3cbf813666dc9fe7188355fa989cdb2bdebdce0bbea06ef74bec148b"
Jul 01 21:34:25.519117 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.519088    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2"]
Jul 01 21:34:25.519457 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:34:25.519433    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2587094e3cbf813666dc9fe7188355fa989cdb2bdebdce0bbea06ef74bec148b\": container with ID starting with 2587094e3cbf813666dc9fe7188355fa989cdb2bdebdce0bbea06ef74bec148b not found: ID does not exist" containerID="2587094e3cbf813666dc9fe7188355fa989cdb2bdebdce0bbea06ef74bec148b"
Jul 01 21:34:25.519558 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.519468    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2587094e3cbf813666dc9fe7188355fa989cdb2bdebdce0bbea06ef74bec148b"} err="failed to get container status \"2587094e3cbf813666dc9fe7188355fa989cdb2bdebdce0bbea06ef74bec148b\": rpc error: code = NotFound desc = could not find container \"2587094e3cbf813666dc9fe7188355fa989cdb2bdebdce0bbea06ef74bec148b\": container with ID starting with 2587094e3cbf813666dc9fe7188355fa989cdb2bdebdce0bbea06ef74bec148b not found: ID does not exist"
Jul 01 21:34:25.519558 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.519492    2564 scope.go:117] "RemoveContainer" containerID="93bd7d4e3ab30f1c627d39344feca687951e653618d60563a3003f1287a8e2a5"
Jul 01 21:34:25.519802 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:34:25.519777    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93bd7d4e3ab30f1c627d39344feca687951e653618d60563a3003f1287a8e2a5\": container with ID starting with 93bd7d4e3ab30f1c627d39344feca687951e653618d60563a3003f1287a8e2a5 not found: ID does not exist" containerID="93bd7d4e3ab30f1c627d39344feca687951e653618d60563a3003f1287a8e2a5"
Jul 01 21:34:25.519888 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.519810    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93bd7d4e3ab30f1c627d39344feca687951e653618d60563a3003f1287a8e2a5"} err="failed to get container status \"93bd7d4e3ab30f1c627d39344feca687951e653618d60563a3003f1287a8e2a5\": rpc error: code = NotFound desc = could not find container \"93bd7d4e3ab30f1c627d39344feca687951e653618d60563a3003f1287a8e2a5\": container with ID starting with 93bd7d4e3ab30f1c627d39344feca687951e653618d60563a3003f1287a8e2a5 not found: ID does not exist"
Jul 01 21:34:25.520974 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:25.520954    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ce993-predictor-6d75865fd7-pt9r2"]
Jul 01 21:34:26.758108 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:26.758073    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8475069f-ef34-4979-9c40-9dcb2723997b" path="/var/lib/kubelet/pods/8475069f-ef34-4979-9c40-9dcb2723997b/volumes"
Jul 01 21:34:26.758676 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:26.758656    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d60c38a1-a61b-47d6-8da7-8584afb9c4b9" path="/var/lib/kubelet/pods/d60c38a1-a61b-47d6-8da7-8584afb9c4b9/volumes"
Jul 01 21:34:30.505604 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:30.505563    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jw56q/must-gather-ddjdb" event={"ID":"759ede6b-4b0d-421a-8b8b-855994182882","Type":"ContainerStarted","Data":"e768b90ddb551d7c153662bbb66ab772ee523007323ec81117f725e21939e47d"}
Jul 01 21:34:30.506041 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:30.505615    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jw56q/must-gather-ddjdb" event={"ID":"759ede6b-4b0d-421a-8b8b-855994182882","Type":"ContainerStarted","Data":"6bedb0e3cf64c33940591c558886e4d3e002dbf301f364eb888f18b750a55778"}
Jul 01 21:34:30.523789 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:30.523728    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jw56q/must-gather-ddjdb" podStartSLOduration=2.173194168 podStartE2EDuration="8.523708235s" totalImagesPullingTime="6.350514067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:34:30.523525189 +0000 UTC m=+2870.199540760" watchObservedRunningTime="2026-07-01 21:34:30.523708235 +0000 UTC m=+2870.199723803"
Jul 01 21:34:51.597274 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:51.597243    2564 generic.go:356] "Generic (PLEG): container finished" podID="759ede6b-4b0d-421a-8b8b-855994182882" containerID="6bedb0e3cf64c33940591c558886e4d3e002dbf301f364eb888f18b750a55778" exitCode=0
Jul 01 21:34:51.597769 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:51.597318    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jw56q/must-gather-ddjdb" event={"ID":"759ede6b-4b0d-421a-8b8b-855994182882","Type":"ContainerDied","Data":"6bedb0e3cf64c33940591c558886e4d3e002dbf301f364eb888f18b750a55778"}
Jul 01 21:34:51.597769 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:51.597640    2564 scope.go:117] "RemoveContainer" containerID="6bedb0e3cf64c33940591c558886e4d3e002dbf301f364eb888f18b750a55778"
Jul 01 21:34:52.396564 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:52.396535    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jw56q_must-gather-ddjdb_759ede6b-4b0d-421a-8b8b-855994182882/gather/0.log"
Jul 01 21:34:52.973217 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:52.973183    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4rws2/must-gather-t8njq"]
Jul 01 21:34:52.973591 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:52.973474    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8475069f-ef34-4979-9c40-9dcb2723997b" containerName="kube-rbac-proxy"
Jul 01 21:34:52.973591 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:52.973485    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8475069f-ef34-4979-9c40-9dcb2723997b" containerName="kube-rbac-proxy"
Jul 01 21:34:52.973591 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:52.973492    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d60c38a1-a61b-47d6-8da7-8584afb9c4b9" containerName="kserve-container"
Jul 01 21:34:52.973591 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:52.973497    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d60c38a1-a61b-47d6-8da7-8584afb9c4b9" containerName="kserve-container"
Jul 01 21:34:52.973591 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:52.973509    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8475069f-ef34-4979-9c40-9dcb2723997b" containerName="kserve-container"
Jul 01 21:34:52.973591 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:52.973514    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8475069f-ef34-4979-9c40-9dcb2723997b" containerName="kserve-container"
Jul 01 21:34:52.973591 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:52.973520    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d60c38a1-a61b-47d6-8da7-8584afb9c4b9" containerName="kube-rbac-proxy"
Jul 01 21:34:52.973591 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:52.973525    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d60c38a1-a61b-47d6-8da7-8584afb9c4b9" containerName="kube-rbac-proxy"
Jul 01 21:34:52.973591 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:52.973572    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d60c38a1-a61b-47d6-8da7-8584afb9c4b9" containerName="kube-rbac-proxy"
Jul 01 21:34:52.973591 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:52.973581    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8475069f-ef34-4979-9c40-9dcb2723997b" containerName="kube-rbac-proxy"
Jul 01 21:34:52.973591 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:52.973587    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8475069f-ef34-4979-9c40-9dcb2723997b" containerName="kserve-container"
Jul 01 21:34:52.973591 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:52.973593    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d60c38a1-a61b-47d6-8da7-8584afb9c4b9" containerName="kserve-container"
Jul 01 21:34:52.976280 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:52.976264    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4rws2/must-gather-t8njq"
Jul 01 21:34:52.979140 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:52.979117    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4rws2\"/\"openshift-service-ca.crt\""
Jul 01 21:34:52.980441 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:52.980423    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4rws2\"/\"default-dockercfg-dcx4p\""
Jul 01 21:34:52.980500 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:52.980441    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4rws2\"/\"kube-root-ca.crt\""
Jul 01 21:34:52.985029 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:52.985008    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4rws2/must-gather-t8njq"]
Jul 01 21:34:53.020727 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:53.020705    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/719ac929-6cc7-4251-a5f6-cf37e4452b58-must-gather-output\") pod \"must-gather-t8njq\" (UID: \"719ac929-6cc7-4251-a5f6-cf37e4452b58\") " pod="openshift-must-gather-4rws2/must-gather-t8njq"
Jul 01 21:34:53.020850 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:53.020734    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx5gk\" (UniqueName: \"kubernetes.io/projected/719ac929-6cc7-4251-a5f6-cf37e4452b58-kube-api-access-hx5gk\") pod \"must-gather-t8njq\" (UID: \"719ac929-6cc7-4251-a5f6-cf37e4452b58\") " pod="openshift-must-gather-4rws2/must-gather-t8njq"
Jul 01 21:34:53.121706 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:53.121679    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/719ac929-6cc7-4251-a5f6-cf37e4452b58-must-gather-output\") pod \"must-gather-t8njq\" (UID: \"719ac929-6cc7-4251-a5f6-cf37e4452b58\") " pod="openshift-must-gather-4rws2/must-gather-t8njq"
Jul 01 21:34:53.121841 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:53.121714    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx5gk\" (UniqueName: \"kubernetes.io/projected/719ac929-6cc7-4251-a5f6-cf37e4452b58-kube-api-access-hx5gk\") pod \"must-gather-t8njq\" (UID: \"719ac929-6cc7-4251-a5f6-cf37e4452b58\") " pod="openshift-must-gather-4rws2/must-gather-t8njq"
Jul 01 21:34:53.122032 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:53.122016    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/719ac929-6cc7-4251-a5f6-cf37e4452b58-must-gather-output\") pod \"must-gather-t8njq\" (UID: \"719ac929-6cc7-4251-a5f6-cf37e4452b58\") " pod="openshift-must-gather-4rws2/must-gather-t8njq"
Jul 01 21:34:53.130548 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:53.130528    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx5gk\" (UniqueName: \"kubernetes.io/projected/719ac929-6cc7-4251-a5f6-cf37e4452b58-kube-api-access-hx5gk\") pod \"must-gather-t8njq\" (UID: \"719ac929-6cc7-4251-a5f6-cf37e4452b58\") " pod="openshift-must-gather-4rws2/must-gather-t8njq"
Jul 01 21:34:53.286136 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:53.286043    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4rws2/must-gather-t8njq"
Jul 01 21:34:53.411646 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:53.411617    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4rws2/must-gather-t8njq"]
Jul 01 21:34:53.414749 ip-10-0-131-198 kubenswrapper[2564]: W0701 21:34:53.414722    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod719ac929_6cc7_4251_a5f6_cf37e4452b58.slice/crio-cbd953c3cb219f66e443fff01182adbc307ed173ed472ffe66249a272742fb7d WatchSource:0}: Error finding container cbd953c3cb219f66e443fff01182adbc307ed173ed472ffe66249a272742fb7d: Status 404 returned error can't find the container with id cbd953c3cb219f66e443fff01182adbc307ed173ed472ffe66249a272742fb7d
Jul 01 21:34:53.604852 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:53.604778    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4rws2/must-gather-t8njq" event={"ID":"719ac929-6cc7-4251-a5f6-cf37e4452b58","Type":"ContainerStarted","Data":"cbd953c3cb219f66e443fff01182adbc307ed173ed472ffe66249a272742fb7d"}
Jul 01 21:34:54.613242 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:54.613152    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4rws2/must-gather-t8njq" event={"ID":"719ac929-6cc7-4251-a5f6-cf37e4452b58","Type":"ContainerStarted","Data":"34f28c1e3db0de6dd19076351a817a3d1d100653edca4778a5b567a09afce563"}
Jul 01 21:34:54.613242 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:54.613199    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4rws2/must-gather-t8njq" event={"ID":"719ac929-6cc7-4251-a5f6-cf37e4452b58","Type":"ContainerStarted","Data":"b1b46fd945af47fab8f59ed726f5da13323254a0afea549a1cf17a41f2c5c134"}
Jul 01 21:34:54.633015 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:54.632970    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4rws2/must-gather-t8njq" podStartSLOduration=1.7593822970000002 podStartE2EDuration="2.632956693s" totalImagesPullingTime="873.574396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 21:34:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:34:54.631544343 +0000 UTC m=+2894.307559911" watchObservedRunningTime="2026-07-01 21:34:54.632956693 +0000 UTC m=+2894.308972259"
Jul 01 21:34:55.659740 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:55.659714    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-64pmt_ed4e44b6-7fbe-4e18-974e-edbc64ce0398/global-pull-secret-syncer/0.log"
Jul 01 21:34:55.789485 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:55.789454    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mkqr5_8634b68f-65d5-4747-b19c-2b2e1d44ba76/konnectivity-agent/0.log"
Jul 01 21:34:55.865567 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:55.865542    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-198.ec2.internal_7bae69993a9f5e773a5205c5e78d2b81/haproxy/0.log"
Jul 01 21:34:57.820882 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:57.820829    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-jw56q/must-gather-ddjdb"]
Jul 01 21:34:57.821472 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:57.821084    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-jw56q/must-gather-ddjdb" podUID="759ede6b-4b0d-421a-8b8b-855994182882" containerName="copy" containerID="cri-o://e768b90ddb551d7c153662bbb66ab772ee523007323ec81117f725e21939e47d" gracePeriod=2
Jul 01 21:34:57.827483 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:57.827453    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-jw56q/must-gather-ddjdb"]
Jul 01 21:34:57.828597 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:34:57.828560    2564 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"759ede6b-4b0d-421a-8b8b-855994182882\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"openshift-must-gather-jw56q\"/\"must-gather-ddjdb\": pods \"must-gather-ddjdb\" not found" pod="openshift-must-gather-jw56q/must-gather-ddjdb"
Jul 01 21:34:58.182374 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:58.182087    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jw56q_must-gather-ddjdb_759ede6b-4b0d-421a-8b8b-855994182882/copy/0.log"
Jul 01 21:34:58.187005 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:58.186733    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jw56q/must-gather-ddjdb"
Jul 01 21:34:58.190115 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:34:58.190061    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-ddjdb\" is forbidden: User \"system:node:ip-10-0-131-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-jw56q\": no relationship found between node 'ip-10-0-131-198.ec2.internal' and this object" podUID="759ede6b-4b0d-421a-8b8b-855994182882" pod="openshift-must-gather-jw56q/must-gather-ddjdb"
Jul 01 21:34:58.276191 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:58.274088    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6sbq\" (UniqueName: \"kubernetes.io/projected/759ede6b-4b0d-421a-8b8b-855994182882-kube-api-access-c6sbq\") pod \"759ede6b-4b0d-421a-8b8b-855994182882\" (UID: \"759ede6b-4b0d-421a-8b8b-855994182882\") "
Jul 01 21:34:58.276191 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:58.274179    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/759ede6b-4b0d-421a-8b8b-855994182882-must-gather-output\") pod \"759ede6b-4b0d-421a-8b8b-855994182882\" (UID: \"759ede6b-4b0d-421a-8b8b-855994182882\") "
Jul 01 21:34:58.276191 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:58.276125    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759ede6b-4b0d-421a-8b8b-855994182882-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "759ede6b-4b0d-421a-8b8b-855994182882" (UID: "759ede6b-4b0d-421a-8b8b-855994182882"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 21:34:58.278143 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:58.278082    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/759ede6b-4b0d-421a-8b8b-855994182882-kube-api-access-c6sbq" (OuterVolumeSpecName: "kube-api-access-c6sbq") pod "759ede6b-4b0d-421a-8b8b-855994182882" (UID: "759ede6b-4b0d-421a-8b8b-855994182882"). InnerVolumeSpecName "kube-api-access-c6sbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 21:34:58.374896 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:58.374822    2564 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/759ede6b-4b0d-421a-8b8b-855994182882-must-gather-output\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:34:58.374896 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:58.374853    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6sbq\" (UniqueName: \"kubernetes.io/projected/759ede6b-4b0d-421a-8b8b-855994182882-kube-api-access-c6sbq\") on node \"ip-10-0-131-198.ec2.internal\" DevicePath \"\""
Jul 01 21:34:58.637061 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:58.636992    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jw56q_must-gather-ddjdb_759ede6b-4b0d-421a-8b8b-855994182882/copy/0.log"
Jul 01 21:34:58.640885 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:58.637736    2564 generic.go:356] "Generic (PLEG): container finished" podID="759ede6b-4b0d-421a-8b8b-855994182882" containerID="e768b90ddb551d7c153662bbb66ab772ee523007323ec81117f725e21939e47d" exitCode=143
Jul 01 21:34:58.640885 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:58.637884    2564 scope.go:117] "RemoveContainer" containerID="e768b90ddb551d7c153662bbb66ab772ee523007323ec81117f725e21939e47d"
Jul 01 21:34:58.640885 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:58.638035    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jw56q/must-gather-ddjdb"
Jul 01 21:34:58.646593 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:34:58.646562    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-ddjdb\" is forbidden: User \"system:node:ip-10-0-131-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-jw56q\": no relationship found between node 'ip-10-0-131-198.ec2.internal' and this object" podUID="759ede6b-4b0d-421a-8b8b-855994182882" pod="openshift-must-gather-jw56q/must-gather-ddjdb"
Jul 01 21:34:58.653044 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:34:58.653005    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-ddjdb\" is forbidden: User \"system:node:ip-10-0-131-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-jw56q\": no relationship found between node 'ip-10-0-131-198.ec2.internal' and this object" podUID="759ede6b-4b0d-421a-8b8b-855994182882" pod="openshift-must-gather-jw56q/must-gather-ddjdb"
Jul 01 21:34:58.659853 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:58.659827    2564 scope.go:117] "RemoveContainer" containerID="6bedb0e3cf64c33940591c558886e4d3e002dbf301f364eb888f18b750a55778"
Jul 01 21:34:58.687795 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:58.687768    2564 scope.go:117] "RemoveContainer" containerID="e768b90ddb551d7c153662bbb66ab772ee523007323ec81117f725e21939e47d"
Jul 01 21:34:58.688342 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:34:58.688311    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e768b90ddb551d7c153662bbb66ab772ee523007323ec81117f725e21939e47d\": container with ID starting with e768b90ddb551d7c153662bbb66ab772ee523007323ec81117f725e21939e47d not found: ID does not exist" containerID="e768b90ddb551d7c153662bbb66ab772ee523007323ec81117f725e21939e47d"
Jul 01 21:34:58.688439 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:58.688355    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e768b90ddb551d7c153662bbb66ab772ee523007323ec81117f725e21939e47d"} err="failed to get container status \"e768b90ddb551d7c153662bbb66ab772ee523007323ec81117f725e21939e47d\": rpc error: code = NotFound desc = could not find container \"e768b90ddb551d7c153662bbb66ab772ee523007323ec81117f725e21939e47d\": container with ID starting with e768b90ddb551d7c153662bbb66ab772ee523007323ec81117f725e21939e47d not found: ID does not exist"
Jul 01 21:34:58.688439 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:58.688384    2564 scope.go:117] "RemoveContainer" containerID="6bedb0e3cf64c33940591c558886e4d3e002dbf301f364eb888f18b750a55778"
Jul 01 21:34:58.688819 ip-10-0-131-198 kubenswrapper[2564]: E0701 21:34:58.688788    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bedb0e3cf64c33940591c558886e4d3e002dbf301f364eb888f18b750a55778\": container with ID starting with 6bedb0e3cf64c33940591c558886e4d3e002dbf301f364eb888f18b750a55778 not found: ID does not exist" containerID="6bedb0e3cf64c33940591c558886e4d3e002dbf301f364eb888f18b750a55778"
Jul 01 21:34:58.688928 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:58.688848    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bedb0e3cf64c33940591c558886e4d3e002dbf301f364eb888f18b750a55778"} err="failed to get container status \"6bedb0e3cf64c33940591c558886e4d3e002dbf301f364eb888f18b750a55778\": rpc error: code = NotFound desc = could not find container \"6bedb0e3cf64c33940591c558886e4d3e002dbf301f364eb888f18b750a55778\": container with ID starting with 6bedb0e3cf64c33940591c558886e4d3e002dbf301f364eb888f18b750a55778 not found: ID does not exist"
Jul 01 21:34:58.762282 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:58.762243    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="759ede6b-4b0d-421a-8b8b-855994182882" path="/var/lib/kubelet/pods/759ede6b-4b0d-421a-8b8b-855994182882/volumes"
Jul 01 21:34:59.856178 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:59.856153    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8fc5460b-1728-45b9-b9a4-f26031e5e37e/alertmanager/0.log"
Jul 01 21:34:59.884040 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:59.884007    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8fc5460b-1728-45b9-b9a4-f26031e5e37e/config-reloader/0.log"
Jul 01 21:34:59.909884 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:59.909817    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8fc5460b-1728-45b9-b9a4-f26031e5e37e/kube-rbac-proxy-web/0.log"
Jul 01 21:34:59.934801 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:59.934764    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8fc5460b-1728-45b9-b9a4-f26031e5e37e/kube-rbac-proxy/0.log"
Jul 01 21:34:59.957791 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:59.957752    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8fc5460b-1728-45b9-b9a4-f26031e5e37e/kube-rbac-proxy-metric/0.log"
Jul 01 21:34:59.979547 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:34:59.979513    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8fc5460b-1728-45b9-b9a4-f26031e5e37e/prom-label-proxy/0.log"
Jul 01 21:35:00.000213 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:00.000173    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8fc5460b-1728-45b9-b9a4-f26031e5e37e/init-config-reloader/0.log"
Jul 01 21:35:00.076584 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:00.076548    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-22nc7_83c9de00-f066-4aa4-900c-4e0d6c5042ca/kube-state-metrics/0.log"
Jul 01 21:35:00.097651 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:00.097620    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-22nc7_83c9de00-f066-4aa4-900c-4e0d6c5042ca/kube-rbac-proxy-main/0.log"
Jul 01 21:35:00.119470 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:00.119391    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-22nc7_83c9de00-f066-4aa4-900c-4e0d6c5042ca/kube-rbac-proxy-self/0.log"
Jul 01 21:35:00.402888 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:00.402844    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x5xnh_866f6082-54b0-4b03-8d36-69dc77a6ca61/node-exporter/0.log"
Jul 01 21:35:00.426176 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:00.426136    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x5xnh_866f6082-54b0-4b03-8d36-69dc77a6ca61/kube-rbac-proxy/0.log"
Jul 01 21:35:00.449737 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:00.449707    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x5xnh_866f6082-54b0-4b03-8d36-69dc77a6ca61/init-textfile/0.log"
Jul 01 21:35:00.481499 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:00.481467    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-md7qm_edb0e1f0-4d38-4af1-80a9-7a2e362797c6/kube-rbac-proxy-main/0.log"
Jul 01 21:35:00.504899 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:00.504849    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-md7qm_edb0e1f0-4d38-4af1-80a9-7a2e362797c6/kube-rbac-proxy-self/0.log"
Jul 01 21:35:00.528981 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:00.528954    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-md7qm_edb0e1f0-4d38-4af1-80a9-7a2e362797c6/openshift-state-metrics/0.log"
Jul 01 21:35:00.573950 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:00.573908    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_84d38e1e-02b7-4104-8f60-ca9b7fa89cfd/prometheus/0.log"
Jul 01 21:35:00.595676 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:00.595647    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_84d38e1e-02b7-4104-8f60-ca9b7fa89cfd/config-reloader/0.log"
Jul 01 21:35:00.618300 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:00.618274    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_84d38e1e-02b7-4104-8f60-ca9b7fa89cfd/thanos-sidecar/0.log"
Jul 01 21:35:00.641137 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:00.641105    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_84d38e1e-02b7-4104-8f60-ca9b7fa89cfd/kube-rbac-proxy-web/0.log"
Jul 01 21:35:00.663304 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:00.663232    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_84d38e1e-02b7-4104-8f60-ca9b7fa89cfd/kube-rbac-proxy/0.log"
Jul 01 21:35:00.687538 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:00.687510    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_84d38e1e-02b7-4104-8f60-ca9b7fa89cfd/kube-rbac-proxy-thanos/0.log"
Jul 01 21:35:00.715394 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:00.715352    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_84d38e1e-02b7-4104-8f60-ca9b7fa89cfd/init-config-reloader/0.log"
Jul 01 21:35:00.845419 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:00.845390    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5fbb849ddd-7w6dx_d584412e-45b7-45b6-8b32-158eb4a93739/telemeter-client/0.log"
Jul 01 21:35:00.869534 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:00.869507    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5fbb849ddd-7w6dx_d584412e-45b7-45b6-8b32-158eb4a93739/reload/0.log"
Jul 01 21:35:00.892126 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:00.892083    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5fbb849ddd-7w6dx_d584412e-45b7-45b6-8b32-158eb4a93739/kube-rbac-proxy/0.log"
Jul 01 21:35:04.463352 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.463315    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4rws2/perf-node-gather-daemonset-fz9cj"]
Jul 01 21:35:04.463941 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.463822    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="759ede6b-4b0d-421a-8b8b-855994182882" containerName="copy"
Jul 01 21:35:04.463941 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.463843    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="759ede6b-4b0d-421a-8b8b-855994182882" containerName="copy"
Jul 01 21:35:04.463941 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.463884    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="759ede6b-4b0d-421a-8b8b-855994182882" containerName="gather"
Jul 01 21:35:04.463941 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.463894    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="759ede6b-4b0d-421a-8b8b-855994182882" containerName="gather"
Jul 01 21:35:04.464177 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.463980    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="759ede6b-4b0d-421a-8b8b-855994182882" containerName="gather"
Jul 01 21:35:04.464177 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.463997    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="759ede6b-4b0d-421a-8b8b-855994182882" containerName="copy"
Jul 01 21:35:04.469175 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.469147    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4rws2/perf-node-gather-daemonset-fz9cj"
Jul 01 21:35:04.477147 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.477117    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4rws2/perf-node-gather-daemonset-fz9cj"]
Jul 01 21:35:04.536487 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.536450    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e565617-8a99-4874-9cc3-27aee3b91710-lib-modules\") pod \"perf-node-gather-daemonset-fz9cj\" (UID: \"0e565617-8a99-4874-9cc3-27aee3b91710\") " pod="openshift-must-gather-4rws2/perf-node-gather-daemonset-fz9cj"
Jul 01 21:35:04.536487 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.536491    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-854vw\" (UniqueName: \"kubernetes.io/projected/0e565617-8a99-4874-9cc3-27aee3b91710-kube-api-access-854vw\") pod \"perf-node-gather-daemonset-fz9cj\" (UID: \"0e565617-8a99-4874-9cc3-27aee3b91710\") " pod="openshift-must-gather-4rws2/perf-node-gather-daemonset-fz9cj"
Jul 01 21:35:04.536695 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.536515    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0e565617-8a99-4874-9cc3-27aee3b91710-podres\") pod \"perf-node-gather-daemonset-fz9cj\" (UID: \"0e565617-8a99-4874-9cc3-27aee3b91710\") " pod="openshift-must-gather-4rws2/perf-node-gather-daemonset-fz9cj"
Jul 01 21:35:04.536695 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.536584    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0e565617-8a99-4874-9cc3-27aee3b91710-proc\") pod \"perf-node-gather-daemonset-fz9cj\" (UID: \"0e565617-8a99-4874-9cc3-27aee3b91710\") " pod="openshift-must-gather-4rws2/perf-node-gather-daemonset-fz9cj"
Jul 01 21:35:04.536695 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.536621    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e565617-8a99-4874-9cc3-27aee3b91710-sys\") pod \"perf-node-gather-daemonset-fz9cj\" (UID: \"0e565617-8a99-4874-9cc3-27aee3b91710\") " pod="openshift-must-gather-4rws2/perf-node-gather-daemonset-fz9cj"
Jul 01 21:35:04.637327 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.637293    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e565617-8a99-4874-9cc3-27aee3b91710-lib-modules\") pod \"perf-node-gather-daemonset-fz9cj\" (UID: \"0e565617-8a99-4874-9cc3-27aee3b91710\") " pod="openshift-must-gather-4rws2/perf-node-gather-daemonset-fz9cj"
Jul 01 21:35:04.637327 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.637331    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-854vw\" (UniqueName: \"kubernetes.io/projected/0e565617-8a99-4874-9cc3-27aee3b91710-kube-api-access-854vw\") pod \"perf-node-gather-daemonset-fz9cj\" (UID: \"0e565617-8a99-4874-9cc3-27aee3b91710\") " pod="openshift-must-gather-4rws2/perf-node-gather-daemonset-fz9cj"
Jul 01 21:35:04.637571 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.637350    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0e565617-8a99-4874-9cc3-27aee3b91710-podres\") pod \"perf-node-gather-daemonset-fz9cj\" (UID: \"0e565617-8a99-4874-9cc3-27aee3b91710\") " pod="openshift-must-gather-4rws2/perf-node-gather-daemonset-fz9cj"
Jul 01 21:35:04.637571 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.637369    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0e565617-8a99-4874-9cc3-27aee3b91710-proc\") pod \"perf-node-gather-daemonset-fz9cj\" (UID: \"0e565617-8a99-4874-9cc3-27aee3b91710\") " pod="openshift-must-gather-4rws2/perf-node-gather-daemonset-fz9cj"
Jul 01 21:35:04.637571 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.637390    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e565617-8a99-4874-9cc3-27aee3b91710-sys\") pod \"perf-node-gather-daemonset-fz9cj\" (UID: \"0e565617-8a99-4874-9cc3-27aee3b91710\") " pod="openshift-must-gather-4rws2/perf-node-gather-daemonset-fz9cj"
Jul 01 21:35:04.637571 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.637468    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e565617-8a99-4874-9cc3-27aee3b91710-sys\") pod \"perf-node-gather-daemonset-fz9cj\" (UID: \"0e565617-8a99-4874-9cc3-27aee3b91710\") " pod="openshift-must-gather-4rws2/perf-node-gather-daemonset-fz9cj"
Jul 01 21:35:04.637571 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.637472    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e565617-8a99-4874-9cc3-27aee3b91710-lib-modules\") pod \"perf-node-gather-daemonset-fz9cj\" (UID: \"0e565617-8a99-4874-9cc3-27aee3b91710\") " pod="openshift-must-gather-4rws2/perf-node-gather-daemonset-fz9cj"
Jul 01 21:35:04.637571 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.637506    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0e565617-8a99-4874-9cc3-27aee3b91710-proc\") pod \"perf-node-gather-daemonset-fz9cj\" (UID: \"0e565617-8a99-4874-9cc3-27aee3b91710\") " pod="openshift-must-gather-4rws2/perf-node-gather-daemonset-fz9cj"
Jul 01 21:35:04.637571 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.637526    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0e565617-8a99-4874-9cc3-27aee3b91710-podres\") pod \"perf-node-gather-daemonset-fz9cj\" (UID: \"0e565617-8a99-4874-9cc3-27aee3b91710\") " pod="openshift-must-gather-4rws2/perf-node-gather-daemonset-fz9cj"
Jul 01 21:35:04.646595 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.646555    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-854vw\" (UniqueName: \"kubernetes.io/projected/0e565617-8a99-4874-9cc3-27aee3b91710-kube-api-access-854vw\") pod \"perf-node-gather-daemonset-fz9cj\" (UID: \"0e565617-8a99-4874-9cc3-27aee3b91710\") " pod="openshift-must-gather-4rws2/perf-node-gather-daemonset-fz9cj"
Jul 01 21:35:04.782850 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.782782    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4rws2/perf-node-gather-daemonset-fz9cj"
Jul 01 21:35:04.824203 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.824171    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zt5qb_f9ed73c7-bfbc-427c-b455-736a3c19ec2e/dns/0.log"
Jul 01 21:35:04.846353 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.846321    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zt5qb_f9ed73c7-bfbc-427c-b455-736a3c19ec2e/kube-rbac-proxy/0.log"
Jul 01 21:35:04.927160 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.927133    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wmgbq_54d8b430-fceb-4c6b-9f65-233650a04f7f/dns-node-resolver/0.log"
Jul 01 21:35:04.931606 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:04.931587    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4rws2/perf-node-gather-daemonset-fz9cj"]
Jul 01 21:35:04.934367 ip-10-0-131-198 kubenswrapper[2564]: W0701 21:35:04.934319    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0e565617_8a99_4874_9cc3_27aee3b91710.slice/crio-b66ef08ba53e6833d7888ecdda7137a8b6841c26ccbdd347395e3bab61ba7f51 WatchSource:0}: Error finding container b66ef08ba53e6833d7888ecdda7137a8b6841c26ccbdd347395e3bab61ba7f51: Status 404 returned error can't find the container with id b66ef08ba53e6833d7888ecdda7137a8b6841c26ccbdd347395e3bab61ba7f51
Jul 01 21:35:05.398584 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:05.398551    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fr7j6_d0e7e2a9-4cad-47c6-8a6b-46c461461be1/node-ca/0.log"
Jul 01 21:35:05.693969 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:05.693880    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4rws2/perf-node-gather-daemonset-fz9cj" event={"ID":"0e565617-8a99-4874-9cc3-27aee3b91710","Type":"ContainerStarted","Data":"dece0694b89fb1173d0b7346f228f1a1aa666bdd05aa25bc8c79146872165120"}
Jul 01 21:35:05.693969 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:05.693954    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4rws2/perf-node-gather-daemonset-fz9cj" event={"ID":"0e565617-8a99-4874-9cc3-27aee3b91710","Type":"ContainerStarted","Data":"b66ef08ba53e6833d7888ecdda7137a8b6841c26ccbdd347395e3bab61ba7f51"}
Jul 01 21:35:05.694681 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:05.694655    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4rws2/perf-node-gather-daemonset-fz9cj"
Jul 01 21:35:05.712690 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:05.712630    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4rws2/perf-node-gather-daemonset-fz9cj" podStartSLOduration=1.712611232 podStartE2EDuration="1.712611232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 21:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 21:35:05.712223751 +0000 UTC m=+2905.388239318" watchObservedRunningTime="2026-07-01 21:35:05.712611232 +0000 UTC m=+2905.388626803"
Jul 01 21:35:06.594432 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:06.594392    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ls26v_912ca207-8e8d-42bf-b1b3-9d8c062a40a9/serve-healthcheck-canary/0.log"
Jul 01 21:35:06.994890 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:06.994849    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9chmn_f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2/kube-rbac-proxy/0.log"
Jul 01 21:35:07.013847 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:07.013824    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9chmn_f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2/exporter/0.log"
Jul 01 21:35:07.033601 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:07.033572    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9chmn_f2c1cd54-42a7-40c2-bb7f-1ac7ed095eb2/extractor/0.log"
Jul 01 21:35:09.371270 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:09.371242    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-xssr5_6a9b8c43-1627-4b2c-a4df-aab673fe4577/s3-init/0.log"
Jul 01 21:35:09.398220 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:09.398200    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-xssr5_6a9b8c43-1627-4b2c-a4df-aab673fe4577/download-hf-model/0.log"
Jul 01 21:35:09.425301 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:09.425276    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-qrxw8_5e570536-4933-4cf0-bbbb-9783c637914b/seaweedfs/0.log"
Jul 01 21:35:12.713766 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:12.712944    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4rws2/perf-node-gather-daemonset-fz9cj"
Jul 01 21:35:14.438948 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:14.438920    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mfxlf_fb41c3f0-f0dc-4340-b524-bb23e5426370/kube-multus-additional-cni-plugins/0.log"
Jul 01 21:35:14.460055 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:14.460012    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mfxlf_fb41c3f0-f0dc-4340-b524-bb23e5426370/egress-router-binary-copy/0.log"
Jul 01 21:35:14.482299 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:14.482273    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mfxlf_fb41c3f0-f0dc-4340-b524-bb23e5426370/cni-plugins/0.log"
Jul 01 21:35:14.503526 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:14.503499    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mfxlf_fb41c3f0-f0dc-4340-b524-bb23e5426370/bond-cni-plugin/0.log"
Jul 01 21:35:14.525528 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:14.525500    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mfxlf_fb41c3f0-f0dc-4340-b524-bb23e5426370/routeoverride-cni/0.log"
Jul 01 21:35:14.548150 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:14.548127    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mfxlf_fb41c3f0-f0dc-4340-b524-bb23e5426370/whereabouts-cni-bincopy/0.log"
Jul 01 21:35:14.571547 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:14.571525    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mfxlf_fb41c3f0-f0dc-4340-b524-bb23e5426370/whereabouts-cni/0.log"
Jul 01 21:35:14.774263 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:14.774195    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ftdgt_c2ef23e3-d38f-47a1-94ea-aa4f8fc40725/kube-multus/0.log"
Jul 01 21:35:14.824283 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:14.824252    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9xhz4_c606e570-2fd6-43cb-9a84-9cdeab442815/network-metrics-daemon/0.log"
Jul 01 21:35:14.841646 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:14.841615    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9xhz4_c606e570-2fd6-43cb-9a84-9cdeab442815/kube-rbac-proxy/0.log"
Jul 01 21:35:16.107689 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:16.107664    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/ovn-controller/0.log"
Jul 01 21:35:16.124675 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:16.124653    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/ovn-acl-logging/1.log"
Jul 01 21:35:16.124960 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:16.124942    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/ovn-acl-logging/0.log"
Jul 01 21:35:16.143887 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:16.143850    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/kube-rbac-proxy-node/0.log"
Jul 01 21:35:16.167705 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:16.167680    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/kube-rbac-proxy-ovn-metrics/0.log"
Jul 01 21:35:16.187675 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:16.187656    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/northd/0.log"
Jul 01 21:35:16.207728 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:16.207705    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/nbdb/0.log"
Jul 01 21:35:16.227594 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:16.227567    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/sbdb/0.log"
Jul 01 21:35:16.277579 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:16.277526    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vfs6k_ffa69aa3-5a0f-4fb7-b2aa-a44b5754e78e/ovnkube-controller/0.log"
Jul 01 21:35:17.334585 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:17.334509    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-9bl8d_f4699b63-6ba0-4401-a43e-57c15fea25b3/network-check-target-container/0.log"
Jul 01 21:35:18.290969 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:18.290946    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-lmpqc_48225b8f-cd34-4fc1-b18e-7b219a899f86/iptables-alerter/0.log"
Jul 01 21:35:18.899261 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:18.899235    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-ljqg5_eb8effda-0dd6-437e-86f5-ac9c97c08193/tuned/0.log"
Jul 01 21:35:22.321543 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:22.321489    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-xsscq_47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2/csi-driver/0.log"
Jul 01 21:35:22.340742 ip-10-0-131-198 kubenswrapper[2564]: I0701 21:35:22.340717    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-xsscq_47a26d5f-fa90-4d3e-95c3-d8ff0afbd4c2/csi-node-driver-registrar/0.log"