Jul 02 21:18:59.034384 ip-10-0-130-230 systemd[1]: Starting Kubernetes Kubelet...
Jul 02 21:18:59.613770 ip-10-0-130-230 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 21:18:59.613770 ip-10-0-130-230 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 02 21:18:59.613770 ip-10-0-130-230 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 21:18:59.613770 ip-10-0-130-230 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 02 21:18:59.613770 ip-10-0-130-230 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 21:18:59.613770 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.565332    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 02 21:18:59.613770 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570121    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 21:18:59.613770 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570134    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 21:18:59.613770 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570138    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 21:18:59.613770 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570141    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 21:18:59.613770 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570144    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 21:18:59.613770 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570146    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 21:18:59.613770 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570150    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 21:18:59.613770 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570152    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 21:18:59.615104 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570155    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 21:18:59.615104 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570158    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 21:18:59.615104 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570160    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 21:18:59.615104 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570163    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 21:18:59.615104 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570165    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 21:18:59.615104 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570173    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 21:18:59.615104 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570176    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 21:18:59.615104 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570178    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 21:18:59.615104 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570181    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 21:18:59.615104 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570183    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 21:18:59.615104 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570186    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 21:18:59.615104 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570191    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 21:18:59.615104 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570195    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 21:18:59.615104 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570198    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 21:18:59.615104 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570201    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 21:18:59.615104 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570204    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 21:18:59.615104 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570206    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 21:18:59.615104 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570209    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 21:18:59.615104 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570212    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 21:18:59.616025 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570214    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 21:18:59.616025 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570217    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 21:18:59.616025 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570219    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 21:18:59.616025 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570222    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 21:18:59.616025 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570224    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 21:18:59.616025 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570227    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 21:18:59.616025 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570230    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 21:18:59.616025 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570232    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 21:18:59.616025 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570235    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 21:18:59.616025 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570239    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 21:18:59.616025 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570242    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 21:18:59.616025 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570246    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 21:18:59.616025 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570248    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 21:18:59.616025 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570251    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 21:18:59.616025 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570253    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 21:18:59.616025 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570257    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 21:18:59.616025 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570260    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 21:18:59.616025 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570262    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 21:18:59.616025 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570265    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 21:18:59.616933 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570267    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 21:18:59.616933 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570270    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 21:18:59.616933 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570272    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 21:18:59.616933 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570275    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 21:18:59.616933 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570278    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 21:18:59.616933 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570281    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 21:18:59.616933 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570284    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 21:18:59.616933 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570286    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 21:18:59.616933 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570289    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 21:18:59.616933 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570293    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 21:18:59.616933 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570297    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 21:18:59.616933 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570299    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 21:18:59.616933 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570302    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 21:18:59.616933 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570305    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 21:18:59.616933 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570307    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 21:18:59.616933 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570310    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 21:18:59.616933 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570312    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 21:18:59.616933 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570315    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 21:18:59.616933 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570318    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 21:18:59.616933 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570321    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 21:18:59.617843 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570323    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 21:18:59.617843 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570327    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 21:18:59.617843 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570332    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 21:18:59.617843 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570335    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 21:18:59.617843 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570338    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 21:18:59.617843 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570340    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 21:18:59.617843 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570343    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 21:18:59.617843 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570345    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 21:18:59.617843 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570348    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 21:18:59.617843 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570351    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 21:18:59.617843 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570354    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 21:18:59.617843 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570357    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 21:18:59.617843 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570359    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 21:18:59.617843 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570361    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 21:18:59.617843 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570364    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 21:18:59.617843 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570368    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 21:18:59.617843 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570370    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 21:18:59.617843 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570373    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 21:18:59.617843 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570376    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 21:18:59.617843 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570379    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 21:18:59.618740 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570381    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 21:18:59.618740 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570384    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 21:18:59.618740 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570387    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 21:18:59.618740 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570390    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 21:18:59.618740 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570392    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 21:18:59.618740 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570395    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 21:18:59.618740 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570398    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 21:18:59.618740 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570400    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 21:18:59.618740 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570403    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 21:18:59.618740 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570406    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 21:18:59.618740 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570409    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 21:18:59.618740 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570412    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 21:18:59.618740 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.570414    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 21:18:59.618740 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571147    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 21:18:59.618740 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571155    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 21:18:59.618740 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571157    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 21:18:59.618740 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571160    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 21:18:59.618740 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571164    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 21:18:59.618740 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571167    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 21:18:59.619649 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571169    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 21:18:59.619649 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571172    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 21:18:59.619649 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571175    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 21:18:59.619649 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571184    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 21:18:59.619649 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571187    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 21:18:59.619649 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571190    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 21:18:59.619649 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571193    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 21:18:59.619649 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571195    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 21:18:59.619649 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571198    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 21:18:59.619649 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571201    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 21:18:59.619649 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571203    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 21:18:59.619649 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571206    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 21:18:59.619649 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571208    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 21:18:59.619649 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571211    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 21:18:59.619649 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571213    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 21:18:59.619649 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571216    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 21:18:59.619649 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571218    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 21:18:59.619649 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571223    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 21:18:59.619649 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571226    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 21:18:59.620546 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571229    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 21:18:59.620546 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571233    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 21:18:59.620546 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571236    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 21:18:59.620546 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571239    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 21:18:59.620546 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571243    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 21:18:59.620546 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571246    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 21:18:59.620546 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571249    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 21:18:59.620546 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571252    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 21:18:59.620546 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571254    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 21:18:59.620546 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571257    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 21:18:59.620546 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571259    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 21:18:59.620546 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571262    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 21:18:59.620546 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571265    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 21:18:59.620546 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571267    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 21:18:59.620546 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571270    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 21:18:59.620546 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571273    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 21:18:59.620546 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571275    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 21:18:59.620546 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571284    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 21:18:59.620546 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571287    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 21:18:59.620546 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571289    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 21:18:59.621450 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571292    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 21:18:59.621450 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571294    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 21:18:59.621450 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571297    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 21:18:59.621450 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571299    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 21:18:59.621450 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571302    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 21:18:59.621450 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571305    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 21:18:59.621450 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571307    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 21:18:59.621450 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571309    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 21:18:59.621450 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571329    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 21:18:59.621450 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571333    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 21:18:59.621450 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571336    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 21:18:59.621450 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571338    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 21:18:59.621450 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571341    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 21:18:59.621450 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571344    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 21:18:59.621450 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571347    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 21:18:59.621450 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571350    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 21:18:59.621450 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571353    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 21:18:59.621450 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571356    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 21:18:59.621450 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571359    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 21:18:59.621450 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571362    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 21:18:59.622339 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571365    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 21:18:59.622339 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571368    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 21:18:59.622339 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571370    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 21:18:59.622339 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571373    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 21:18:59.622339 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571376    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 21:18:59.622339 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571378    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 21:18:59.622339 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571381    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 21:18:59.622339 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571384    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 21:18:59.622339 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571389    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 21:18:59.622339 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571392    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 21:18:59.622339 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571401    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 21:18:59.622339 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571404    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 21:18:59.622339 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571407    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 21:18:59.622339 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571410    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 21:18:59.622339 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571412    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 21:18:59.622339 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571415    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 21:18:59.622339 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571418    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 21:18:59.622339 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571420    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 21:18:59.622339 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571423    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 21:18:59.623470 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571426    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 21:18:59.623470 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571429    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 21:18:59.623470 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571432    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 21:18:59.623470 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571434    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 21:18:59.623470 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571437    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 21:18:59.623470 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571440    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 21:18:59.623470 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571442    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 21:18:59.623470 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571445    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 21:18:59.623470 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571447    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 21:18:59.623470 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571450    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 21:18:59.623470 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571452    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 21:18:59.623470 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571455    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 21:18:59.623470 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571458    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 21:18:59.623470 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571460    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 21:18:59.623470 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571463    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 21:18:59.623470 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571893    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 21:18:59.623470 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571899    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 21:18:59.623470 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571902    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 21:18:59.623470 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571905    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 21:18:59.623470 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571907    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 21:18:59.624381 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571910    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 21:18:59.624381 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571912    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 21:18:59.624381 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571915    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 21:18:59.624381 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571924    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 21:18:59.624381 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571927    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 21:18:59.624381 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571929    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 21:18:59.624381 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571932    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 21:18:59.624381 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571934    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 21:18:59.624381 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571937    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 21:18:59.624381 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571940    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 21:18:59.624381 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571942    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 21:18:59.624381 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571945    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 21:18:59.624381 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571948    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 21:18:59.624381 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571952    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 21:18:59.624381 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571955    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 21:18:59.624381 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571957    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 21:18:59.624381 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571975    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 21:18:59.624381 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571978    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 21:18:59.624381 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571981    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 21:18:59.624381 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571984    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 21:18:59.625501 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571987    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 21:18:59.625501 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571991    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 21:18:59.625501 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571994    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 21:18:59.625501 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.571998    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 21:18:59.625501 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572000    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 21:18:59.625501 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572003    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 21:18:59.625501 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572006    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 21:18:59.625501 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572008    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 21:18:59.625501 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572011    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 21:18:59.625501 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572013    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 21:18:59.625501 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572016    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 21:18:59.625501 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572018    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 21:18:59.625501 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572021    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 21:18:59.625501 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572026    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 21:18:59.625501 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572029    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 21:18:59.625501 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572032    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 21:18:59.625501 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572042    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 21:18:59.625501 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572044    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 21:18:59.625501 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572047    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 21:18:59.626403 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572051    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 21:18:59.626403 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572054    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 21:18:59.626403 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572056    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 21:18:59.626403 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572058    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 21:18:59.626403 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572061    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 21:18:59.626403 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572064    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 21:18:59.626403 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572067    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 21:18:59.626403 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572070    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 21:18:59.626403 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572072    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 21:18:59.626403 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572075    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 21:18:59.626403 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572077    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 21:18:59.626403 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572080    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 21:18:59.626403 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572082    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 21:18:59.626403 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572085    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 21:18:59.626403 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572087    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 21:18:59.626403 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572090    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 21:18:59.626403 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572092    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 21:18:59.626403 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572095    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 21:18:59.626403 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572097    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 21:18:59.626403 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572100    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 21:18:59.627302 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572102    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 21:18:59.627302 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572104    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 21:18:59.627302 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572107    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 21:18:59.627302 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572124    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 21:18:59.627302 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572128    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 21:18:59.627302 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572131    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 21:18:59.627302 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572134    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 21:18:59.627302 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572137    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 21:18:59.627302 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572139    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 21:18:59.627302 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572141    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 21:18:59.627302 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572150    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 21:18:59.627302 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572153    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 21:18:59.627302 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572157    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 21:18:59.627302 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572160    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 21:18:59.627302 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572162    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 21:18:59.627302 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572165    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 21:18:59.627302 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572167    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 21:18:59.627302 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572170    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 21:18:59.627302 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572174    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 21:18:59.628408 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572177    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 21:18:59.628408 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572179    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 21:18:59.628408 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572182    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 21:18:59.628408 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572184    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 21:18:59.628408 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572187    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 21:18:59.628408 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572190    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 21:18:59.628408 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572192    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 21:18:59.628408 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572194    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 21:18:59.628408 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572197    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 21:18:59.628408 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572199    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 21:18:59.628408 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572202    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 21:18:59.628408 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572204    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 21:18:59.628408 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572207    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 21:18:59.628408 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572209    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 21:18:59.628408 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572212    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 21:18:59.628408 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.572214    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 21:18:59.628408 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572284    2564 flags.go:64] FLAG: --address="0.0.0.0"
Jul 02 21:18:59.628408 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572295    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 02 21:18:59.628408 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572312    2564 flags.go:64] FLAG: --anonymous-auth="true"
Jul 02 21:18:59.628408 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572319    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 02 21:18:59.628408 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572323    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 02 21:18:59.629322 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572326    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 02 21:18:59.629322 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572330    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 02 21:18:59.629322 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572335    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 02 21:18:59.629322 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572344    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 02 21:18:59.629322 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572349    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 02 21:18:59.629322 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572352    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 02 21:18:59.629322 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572356    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 02 21:18:59.629322 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572359    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 02 21:18:59.629322 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572362    2564 flags.go:64] FLAG: --cgroup-root=""
Jul 02 21:18:59.629322 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572365    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 02 21:18:59.629322 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572369    2564 flags.go:64] FLAG: --client-ca-file=""
Jul 02 21:18:59.629322 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572371    2564 flags.go:64] FLAG: --cloud-provider="external"
Jul 02 21:18:59.629322 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572375    2564 flags.go:64] FLAG: --cluster-dns="[]"
Jul 02 21:18:59.629322 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572381    2564 flags.go:64] FLAG: --cluster-domain=""
Jul 02 21:18:59.629322 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572384    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 02 21:18:59.629322 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572387    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 02 21:18:59.629322 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572390    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 02 21:18:59.629322 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572394    2564 flags.go:64] FLAG: --container-log-max-files="5"
Jul 02 21:18:59.629322 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572398    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 02 21:18:59.629322 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572400    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 02 21:18:59.629322 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572404    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 02 21:18:59.629322 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572407    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 02 21:18:59.629322 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572410    2564 flags.go:64] FLAG: --contention-profiling="false"
Jul 02 21:18:59.629322 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572413    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 02 21:18:59.630454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572416    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 02 21:18:59.630454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572419    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 02 21:18:59.630454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572422    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 02 21:18:59.630454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572426    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 02 21:18:59.630454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572429    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 02 21:18:59.630454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572432    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 02 21:18:59.630454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572435    2564 flags.go:64] FLAG: --enable-load-reader="false"
Jul 02 21:18:59.630454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572438    2564 flags.go:64] FLAG: --enable-server="true"
Jul 02 21:18:59.630454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572440    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 02 21:18:59.630454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572448    2564 flags.go:64] FLAG: --event-burst="100"
Jul 02 21:18:59.630454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572451    2564 flags.go:64] FLAG: --event-qps="50"
Jul 02 21:18:59.630454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572454    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 02 21:18:59.630454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572463    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 02 21:18:59.630454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572467    2564 flags.go:64] FLAG: --eviction-hard=""
Jul 02 21:18:59.630454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572470    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 02 21:18:59.630454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572473    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 02 21:18:59.630454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572476    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 02 21:18:59.630454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572479    2564 flags.go:64] FLAG: --eviction-soft=""
Jul 02 21:18:59.630454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572483    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 02 21:18:59.630454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572486    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 02 21:18:59.630454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572489    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 02 21:18:59.630454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572492    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 02 21:18:59.630454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572495    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 02 21:18:59.630454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572497    2564 flags.go:64] FLAG: --fail-swap-on="true"
Jul 02 21:18:59.630454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572500    2564 flags.go:64] FLAG: --feature-gates=""
Jul 02 21:18:59.631605 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572504    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 02 21:18:59.631605 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572506    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 02 21:18:59.631605 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572513    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 02 21:18:59.631605 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572516    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 02 21:18:59.631605 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572519    2564 flags.go:64] FLAG: --healthz-port="10248"
Jul 02 21:18:59.631605 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572522    2564 flags.go:64] FLAG: --help="false"
Jul 02 21:18:59.631605 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572525    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-130-230.ec2.internal"
Jul 02 21:18:59.631605 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572528    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 02 21:18:59.631605 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572530    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 02 21:18:59.631605 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572533    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 02 21:18:59.631605 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572537    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 02 21:18:59.631605 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572540    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 02 21:18:59.631605 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572543    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 02 21:18:59.631605 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572546    2564 flags.go:64] FLAG: --image-service-endpoint=""
Jul 02 21:18:59.631605 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572549    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 02 21:18:59.631605 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572552    2564 flags.go:64] FLAG: --kube-api-burst="100"
Jul 02 21:18:59.631605 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572554    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 02 21:18:59.631605 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572558    2564 flags.go:64] FLAG: --kube-api-qps="50"
Jul 02 21:18:59.631605 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572560    2564 flags.go:64] FLAG: --kube-reserved=""
Jul 02 21:18:59.631605 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572565    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 02 21:18:59.631605 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572573    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 02 21:18:59.631605 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572577    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 02 21:18:59.631605 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572579    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 02 21:18:59.631605 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572582    2564 flags.go:64] FLAG: --lock-file=""
Jul 02 21:18:59.632737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572585    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 02 21:18:59.632737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572589    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 02 21:18:59.632737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572592    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 02 21:18:59.632737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572597    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 02 21:18:59.632737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572600    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 02 21:18:59.632737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572603    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 02 21:18:59.632737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572605    2564 flags.go:64] FLAG: --logging-format="text"
Jul 02 21:18:59.632737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572608    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 02 21:18:59.632737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572612    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 02 21:18:59.632737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572614    2564 flags.go:64] FLAG: --manifest-url=""
Jul 02 21:18:59.632737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572617    2564 flags.go:64] FLAG: --manifest-url-header=""
Jul 02 21:18:59.632737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572621    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 02 21:18:59.632737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572624    2564 flags.go:64] FLAG: --max-open-files="1000000"
Jul 02 21:18:59.632737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572629    2564 flags.go:64] FLAG: --max-pods="110"
Jul 02 21:18:59.632737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572632    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 02 21:18:59.632737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572634    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 02 21:18:59.632737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572637    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 02 21:18:59.632737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572640    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 02 21:18:59.632737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572643    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 02 21:18:59.632737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572646    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 02 21:18:59.632737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572657    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 02 21:18:59.632737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572664    2564 flags.go:64] FLAG: --node-status-max-images="50"
Jul 02 21:18:59.632737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572667    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 02 21:18:59.632737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572670    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 02 21:18:59.632737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572673    2564 flags.go:64] FLAG: --pod-cidr=""
Jul 02 21:18:59.636060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572675    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 02 21:18:59.636060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572681    2564 flags.go:64] FLAG: --pod-manifest-path=""
Jul 02 21:18:59.636060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572685    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 02 21:18:59.636060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572688    2564 flags.go:64] FLAG: --pods-per-core="0"
Jul 02 21:18:59.636060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572696    2564 flags.go:64] FLAG: --port="10250"
Jul 02 21:18:59.636060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572700    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 02 21:18:59.636060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572703    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e40303ffc4e1d9b7"
Jul 02 21:18:59.636060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572706    2564 flags.go:64] FLAG: --qos-reserved=""
Jul 02 21:18:59.636060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572709    2564 flags.go:64] FLAG: --read-only-port="10255"
Jul 02 21:18:59.636060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572713    2564 flags.go:64] FLAG: --register-node="true"
Jul 02 21:18:59.636060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572716    2564 flags.go:64] FLAG: --register-with-taints=""
Jul 02 21:18:59.636060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572720    2564 flags.go:64] FLAG: --registry-burst="10"
Jul 02 21:18:59.636060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572723    2564 flags.go:64] FLAG: --registry-qps="5"
Jul 02 21:18:59.636060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572725    2564 flags.go:64] FLAG: --reserved-cpus=""
Jul 02 21:18:59.636060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572728    2564 flags.go:64] FLAG: --reserved-memory=""
Jul 02 21:18:59.636060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572732    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 02 21:18:59.636060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572734    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 02 21:18:59.636060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572737    2564 flags.go:64] FLAG: --rotate-certificates="false"
Jul 02 21:18:59.636060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572740    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 02 21:18:59.636060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572743    2564 flags.go:64] FLAG: --runonce="false"
Jul 02 21:18:59.636060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572746    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 02 21:18:59.636060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572749    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 02 21:18:59.636060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572751    2564 flags.go:64] FLAG: --seccomp-default="false"
Jul 02 21:18:59.636060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572754    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 02 21:18:59.636060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572757    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 02 21:18:59.637378 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572760    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 02 21:18:59.637378 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572763    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 02 21:18:59.637378 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572766    2564 flags.go:64] FLAG: --storage-driver-password="root"
Jul 02 21:18:59.637378 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572769    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 02 21:18:59.637378 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572772    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 02 21:18:59.637378 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572774    2564 flags.go:64] FLAG: --storage-driver-user="root"
Jul 02 21:18:59.637378 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572777    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 02 21:18:59.637378 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572780    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 02 21:18:59.637378 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572783    2564 flags.go:64] FLAG: --system-cgroups=""
Jul 02 21:18:59.637378 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572787    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 02 21:18:59.637378 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572792    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 02 21:18:59.637378 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572795    2564 flags.go:64] FLAG: --tls-cert-file=""
Jul 02 21:18:59.637378 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572805    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 02 21:18:59.637378 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572811    2564 flags.go:64] FLAG: --tls-min-version=""
Jul 02 21:18:59.637378 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572820    2564 flags.go:64] FLAG: --tls-private-key-file=""
Jul 02 21:18:59.637378 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572822    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 02 21:18:59.637378 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572826    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 02 21:18:59.637378 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572829    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 02 21:18:59.637378 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572832    2564 flags.go:64] FLAG: --v="2"
Jul 02 21:18:59.637378 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572836    2564 flags.go:64] FLAG: --version="false"
Jul 02 21:18:59.637378 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572840    2564 flags.go:64] FLAG: --vmodule=""
Jul 02 21:18:59.637378 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572843    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 02 21:18:59.637378 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.572847    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 02 21:18:59.637378 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573010    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 21:18:59.637378 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573014    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 21:18:59.638908 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573017    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 21:18:59.638908 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573020    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 21:18:59.638908 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573022    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 21:18:59.638908 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573025    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 21:18:59.638908 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573028    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 21:18:59.638908 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573030    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 21:18:59.638908 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573033    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 21:18:59.638908 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573035    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 21:18:59.638908 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573039    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 21:18:59.638908 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573041    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 21:18:59.638908 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573044    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 21:18:59.638908 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573046    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 21:18:59.638908 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573049    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 21:18:59.638908 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573051    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 21:18:59.638908 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573053    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 21:18:59.638908 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573056    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 21:18:59.638908 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573060    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 21:18:59.638908 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573062    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 21:18:59.638908 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573065    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 21:18:59.638908 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573067    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 21:18:59.639795 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573076    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 21:18:59.639795 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573079    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 21:18:59.639795 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573081    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 21:18:59.639795 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573086    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 21:18:59.639795 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573089    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 21:18:59.639795 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573091    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 21:18:59.639795 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573094    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 21:18:59.639795 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573097    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 21:18:59.639795 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573099    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 21:18:59.639795 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573102    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 21:18:59.639795 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573104    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 21:18:59.639795 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573107    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 21:18:59.639795 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573123    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 21:18:59.639795 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573126    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 21:18:59.639795 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573129    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 21:18:59.639795 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573132    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 21:18:59.639795 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573134    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 21:18:59.639795 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573137    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 21:18:59.639795 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573139    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 21:18:59.639795 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573142    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 21:18:59.703590 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573144    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 21:18:59.703590 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573147    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 21:18:59.703590 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573149    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 21:18:59.703590 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573152    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 21:18:59.703590 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573154    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 21:18:59.703590 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573157    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 21:18:59.703590 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573159    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 21:18:59.703590 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573162    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 21:18:59.703590 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573166    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 21:18:59.703590 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573170    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 21:18:59.703590 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573173    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 21:18:59.703590 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573176    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 21:18:59.703590 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573179    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 21:18:59.703590 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573188    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 21:18:59.703590 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573191    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 21:18:59.703590 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573196    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 21:18:59.703590 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573199    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 21:18:59.703590 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573201    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 21:18:59.703590 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573204    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 21:18:59.705033 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573208    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 21:18:59.705033 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573211    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 21:18:59.705033 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573214    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 21:18:59.705033 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573217    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 21:18:59.705033 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573220    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 21:18:59.705033 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573223    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 21:18:59.705033 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573225    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 21:18:59.705033 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573228    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 21:18:59.705033 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573230    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 21:18:59.705033 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573233    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 21:18:59.705033 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573235    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 21:18:59.705033 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573238    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 21:18:59.705033 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573241    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 21:18:59.705033 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573244    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 21:18:59.705033 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573247    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 21:18:59.705033 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573250    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 21:18:59.705033 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573252    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 21:18:59.705033 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573255    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 21:18:59.705033 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573257    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 21:18:59.710085 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573260    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 21:18:59.710085 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573262    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 21:18:59.710085 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573267    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 21:18:59.710085 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573269    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 21:18:59.710085 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573272    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 21:18:59.710085 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573274    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 21:18:59.710085 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573277    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 21:18:59.710085 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573279    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 21:18:59.710085 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573287    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 21:18:59.710085 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573291    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 21:18:59.710085 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573294    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 21:18:59.710085 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573297    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 21:18:59.710085 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573299    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 21:18:59.710085 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573301    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 21:18:59.710085 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573304    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 21:18:59.710085 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573307    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 21:18:59.710085 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573309    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 21:18:59.710085 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573312    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 21:18:59.710085 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.573314    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 21:18:59.711068 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.573850    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 21:18:59.711068 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.582865    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 02 21:18:59.711068 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.582881    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 02 21:18:59.711068 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.582902    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 02 21:18:59.711068 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.582908    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 02 21:18:59.711068 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583020    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 21:18:59.711068 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583030    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 21:18:59.711068 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583034    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 21:18:59.711068 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583039    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 21:18:59.711068 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583042    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 21:18:59.711068 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583045    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 21:18:59.711068 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583048    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 21:18:59.711068 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583050    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 21:18:59.711068 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583053    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 21:18:59.711068 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583056    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 21:18:59.712728 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583059    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 21:18:59.712728 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583062    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 21:18:59.712728 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583065    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 21:18:59.712728 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583068    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 21:18:59.712728 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583074    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 21:18:59.712728 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583077    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 21:18:59.712728 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583080    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 21:18:59.712728 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583082    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 21:18:59.712728 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583085    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 21:18:59.712728 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583092    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 21:18:59.712728 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583095    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 21:18:59.712728 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583097    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 21:18:59.712728 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583100    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 21:18:59.712728 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583103    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 21:18:59.712728 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583106    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 21:18:59.712728 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583122    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 21:18:59.712728 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583134    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 21:18:59.712728 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583137    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 21:18:59.712728 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583140    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 21:18:59.713714 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583146    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 21:18:59.713714 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583150    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 21:18:59.713714 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583153    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 21:18:59.713714 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583156    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 21:18:59.713714 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583159    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 21:18:59.713714 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583162    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 21:18:59.713714 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583165    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 21:18:59.713714 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583168    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 21:18:59.713714 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583170    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 21:18:59.713714 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583175    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 21:18:59.713714 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583180    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 21:18:59.713714 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583183    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 21:18:59.713714 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583185    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 21:18:59.713714 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583188    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 21:18:59.713714 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583191    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 21:18:59.713714 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583193    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 21:18:59.713714 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583196    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 21:18:59.713714 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583199    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 21:18:59.713714 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583202    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 21:18:59.713714 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583205    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 21:18:59.714925 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583207    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 21:18:59.714925 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583210    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 21:18:59.714925 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583214    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 21:18:59.714925 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583219    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 21:18:59.714925 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583222    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 21:18:59.714925 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583225    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 21:18:59.714925 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583228    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 21:18:59.714925 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583231    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 21:18:59.714925 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583234    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 21:18:59.714925 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583237    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 21:18:59.714925 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583239    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 21:18:59.714925 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583242    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 21:18:59.714925 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583245    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 21:18:59.714925 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583248    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 21:18:59.714925 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583251    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 21:18:59.714925 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583256    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 21:18:59.714925 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583259    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 21:18:59.714925 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583262    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 21:18:59.714925 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583265    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 21:18:59.714925 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583267    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 21:18:59.720042 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583270    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 21:18:59.720042 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583274    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 21:18:59.720042 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583279    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 21:18:59.720042 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583284    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 21:18:59.720042 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583292    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 21:18:59.720042 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583295    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 21:18:59.720042 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583298    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 21:18:59.720042 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583303    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 21:18:59.720042 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583306    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 21:18:59.720042 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583308    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 21:18:59.720042 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583311    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 21:18:59.720042 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583314    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 21:18:59.720042 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583317    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 21:18:59.720042 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583336    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 21:18:59.720042 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583342    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 21:18:59.720042 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583346    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 21:18:59.720042 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583349    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 21:18:59.720042 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583352    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 21:18:59.720042 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583355    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 21:18:59.721010 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583358    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 21:18:59.721010 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583364    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 21:18:59.721010 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583367    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 21:18:59.721010 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583370    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 21:18:59.721010 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583373    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 21:18:59.721010 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583375    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 21:18:59.721010 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583378    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 21:18:59.721010 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583382    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 21:18:59.721010 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583384    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 21:18:59.721010 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583388    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 21:18:59.721010 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583391    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 21:18:59.721010 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.583397    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 21:18:59.721010 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583660    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 21:18:59.721010 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583667    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 21:18:59.721010 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583670    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 21:18:59.721993 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583676    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 21:18:59.721993 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583679    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 21:18:59.721993 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583682    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 21:18:59.721993 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583685    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 21:18:59.721993 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583692    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 21:18:59.721993 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583696    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 21:18:59.721993 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583699    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 21:18:59.721993 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583702    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 21:18:59.721993 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583705    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 21:18:59.721993 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583708    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 21:18:59.721993 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583711    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 21:18:59.721993 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583714    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 21:18:59.721993 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583718    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 21:18:59.721993 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583721    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 21:18:59.721993 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583723    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 21:18:59.721993 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583726    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 21:18:59.721993 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583729    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 21:18:59.721993 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583734    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 21:18:59.721993 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583737    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 21:18:59.723686 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583739    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 21:18:59.723686 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583742    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 21:18:59.723686 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583744    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 21:18:59.723686 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583747    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 21:18:59.723686 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583750    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 21:18:59.723686 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583753    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 21:18:59.723686 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583756    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 21:18:59.723686 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583759    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 21:18:59.723686 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583764    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 21:18:59.723686 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583769    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 21:18:59.723686 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583782    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 21:18:59.723686 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583784    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 21:18:59.723686 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583787    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 21:18:59.723686 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583791    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 21:18:59.723686 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583794    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 21:18:59.723686 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583797    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 21:18:59.723686 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583800    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 21:18:59.723686 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583802    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 21:18:59.723686 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583805    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 21:18:59.723686 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583807    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 21:18:59.724652 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583810    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 21:18:59.724652 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583812    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 21:18:59.724652 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583815    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 21:18:59.724652 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583820    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 21:18:59.724652 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583823    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 21:18:59.724652 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583827    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 21:18:59.724652 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583829    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 21:18:59.724652 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583833    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 21:18:59.724652 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583836    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 21:18:59.724652 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583840    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 21:18:59.724652 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583843    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 21:18:59.724652 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583846    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 21:18:59.724652 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583849    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 21:18:59.724652 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583852    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 21:18:59.724652 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583858    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 21:18:59.724652 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583861    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 21:18:59.724652 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583864    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 21:18:59.724652 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583867    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 21:18:59.724652 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583870    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 21:18:59.725610 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583874    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 21:18:59.725610 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583877    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 21:18:59.725610 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583881    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 21:18:59.725610 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583883    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 21:18:59.725610 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583886    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 21:18:59.725610 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583889    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 21:18:59.725610 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583892    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 21:18:59.725610 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583895    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 21:18:59.725610 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583900    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 21:18:59.725610 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583903    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 21:18:59.725610 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583906    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 21:18:59.725610 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583908    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 21:18:59.725610 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583911    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 21:18:59.725610 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583914    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 21:18:59.725610 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583917    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 21:18:59.725610 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583919    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 21:18:59.725610 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583923    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 21:18:59.725610 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583927    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 21:18:59.725610 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583933    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 21:18:59.725610 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583937    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 21:18:59.728037 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583942    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 21:18:59.728037 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583945    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 21:18:59.728037 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583948    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 21:18:59.728037 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583950    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 21:18:59.728037 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583953    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 21:18:59.728037 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583956    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 21:18:59.728037 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583958    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 21:18:59.728037 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583961    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 21:18:59.728037 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583963    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 21:18:59.728037 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583966    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 21:18:59.728037 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583969    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 21:18:59.728037 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583971    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 21:18:59.728037 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583974    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 21:18:59.728037 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583979    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 21:18:59.728037 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583982    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 21:18:59.728037 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583985    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 21:18:59.728037 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583988    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 21:18:59.728037 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:18:59.583990    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 21:18:59.796603 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.583997    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 21:18:59.796603 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.584181    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 02 21:18:59.796603 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.650883    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 02 21:18:59.796603 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.652268    2564 server.go:1019] "Starting client certificate rotation"
Jul 02 21:18:59.796603 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.652360    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 21:18:59.796603 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.652439    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 21:18:59.796603 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.654816    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 21:18:59.796603 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.678080    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 21:18:59.796603 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.704139    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 21:18:59.796603 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.730229    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j9zz2"
Jul 02 21:18:59.796603 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.738007    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j9zz2"
Jul 02 21:18:59.797910 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.761929    2564 log.go:25] "Validated CRI v1 runtime API"
Jul 02 21:18:59.797910 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.764350    2564 log.go:25] "Validated CRI v1 image API"
Jul 02 21:18:59.797910 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.765299    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 02 21:18:59.797910 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.769868    2564 fs.go:135] Filesystem UUIDs: map[044e53ec-3b59-4583-b0a4-3cc983d8a96a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ce4c15b5-0831-434f-bbf6-020c36c5c3f1:/dev/nvme0n1p3]
Jul 02 21:18:59.797910 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.769884    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 02 21:18:59.799612 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.775533    2564 manager.go:216] Machine: {Timestamp:2026-07-02 21:18:59.773778242 +0000 UTC m=+0.611018283 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100851 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e8f8bdc4f57deeae002ca64b60ba0 SystemUUID:ec2e8f8b-dc4f-57de-eae0-02ca64b60ba0 BootID:ab785e15-c357-4936-a857-e6edb6eb78d4 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8d:f1:49:bb:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8d:f1:49:bb:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:a1:49:ee:0c:da Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 02 21:18:59.799612 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.775650    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 02 21:18:59.799612 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.775740    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 02 21:18:59.799612 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.779514    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 02 21:18:59.799612 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.779538    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-230.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 02 21:18:59.799612 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.780872    2564 topology_manager.go:138] "Creating topology manager with none policy"
Jul 02 21:18:59.799612 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.780883    2564 container_manager_linux.go:309] "Creating device plugin manager"
Jul 02 21:18:59.799612 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.780892    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 21:18:59.799612 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.781567    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 21:18:59.799612 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.782350    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 02 21:18:59.799612 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.785665    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 21:18:59.799612 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.785801    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 02 21:18:59.799612 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.785833    2564 kubelet.go:486] "Attempting to sync node with API server"
Jul 02 21:18:59.799612 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.785846    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 02 21:18:59.799612 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.785862    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 02 21:18:59.799612 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.785872    2564 kubelet.go:398] "Adding apiserver pod source"
Jul 02 21:18:59.799612 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.785911    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 02 21:18:59.799612 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.786313    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 21:18:59.800596 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.786334    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.825885    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.828528    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.828564    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.829697    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.829712    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.829719    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.829725    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.829734    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.829740    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.829748    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.829754    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.829762    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.829767    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.829779    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.829784    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.831387    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.831397    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.832001    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.834040    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.834839    2564 server.go:1268] "Started kubelet"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.834956    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.834971    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.835027    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.836757    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.837251    2564 server.go:310] "Adding debug handlers to kubelet server"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.843477    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.843485    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:18:59.843603    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-230.ec2.internal\" not found"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.844195    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.844197    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.844220    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.844331    2564 reconstruct.go:97] "Volume reconstruction finished"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.844339    2564 reconciler.go:29] "Reconciler: start to sync state"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.853479    2564 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-130-230.ec2.internal" not found
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.853646    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.854089    2564 factory.go:153] Registering CRI-O factory
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.854129    2564 factory.go:223] Registration of the crio container factory successfully
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.854202    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.854211    2564 factory.go:55] Registering systemd factory
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.854218    2564 factory.go:223] Registration of the systemd container factory successfully
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.854240    2564 factory.go:103] Registering Raw factory
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.854250    2564 manager.go:1196] Started watching for new ooms in manager
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:18:59.854443    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.854679    2564 manager.go:349] Starting recovery of all containers
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:18:59.855805    2564 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-230.ec2.internal\" not found" node="ip-10-0-130-230.ec2.internal"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.865699    2564 manager.go:354] Recovery completed
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.869602    2564 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-130-230.ec2.internal" not found
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.872667    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.874795    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-230.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.874824    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-230.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 21:18:59.887832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.874835    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-230.ec2.internal" event="NodeHasSufficientPID"
Jul 02 21:18:59.835616 ip-10-0-130-230 systemd[1]: Started Kubernetes Kubelet.
Jul 02 21:18:59.889981 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.875530    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 02 21:18:59.889981 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.875540    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 21:18:59.889981 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.875539    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 02 21:18:59.889981 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.875580    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 21:18:59.889981 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.876540    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-230.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 21:18:59.889981 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.876572    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-230.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 21:18:59.889981 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.876582    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-230.ec2.internal" event="NodeHasSufficientPID"
Jul 02 21:18:59.889981 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.877238    2564 policy_none.go:49] "None policy: Start"
Jul 02 21:18:59.889981 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.877251    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 02 21:18:59.889981 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.877261    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 02 21:18:59.889981 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.878457    2564 policy_none.go:47] "Start"
Jul 02 21:19:00.021046 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.926052    2564 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-130-230.ec2.internal" not found
Jul 02 21:19:00.021046 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.927733    2564 manager.go:337] "Starting Device Plugin manager"
Jul 02 21:19:00.021046 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:18:59.927768    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 02 21:19:00.021046 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.927779    2564 server.go:87] "Starting device plugin registration server"
Jul 02 21:19:00.021046 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.928003    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 02 21:19:00.021046 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.928015    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 02 21:19:00.021046 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.928137    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 02 21:19:00.021046 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.928239    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 02 21:19:00.021046 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:18:59.928253    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 02 21:19:00.021046 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:18:59.928687    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 02 21:19:00.021046 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:18:59.928732    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-230.ec2.internal\" not found"
Jul 02 21:19:00.030684 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.028676    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 21:19:00.030684 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.029720    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-230.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 21:19:00.030684 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.029751    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-230.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 21:19:00.030684 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.029766    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-230.ec2.internal" event="NodeHasSufficientPID"
Jul 02 21:19:00.030684 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.029793    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-230.ec2.internal"
Jul 02 21:19:00.171153 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.039566    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-230.ec2.internal"
Jul 02 21:19:00.171153 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:00.039587    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-230.ec2.internal\": node \"ip-10-0-130-230.ec2.internal\" not found"
Jul 02 21:19:00.171153 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.059874    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 02 21:19:00.171153 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.065553    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 02 21:19:00.171153 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.065599    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 02 21:19:00.171153 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.065625    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 02 21:19:00.171153 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:00.065658    2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 02 21:19:00.171153 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.067746    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 21:19:00.171153 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.166057    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-230.ec2.internal"]
Jul 02 21:19:00.171153 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.167728    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal"
Jul 02 21:19:00.171153 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.169028    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-230.ec2.internal"
Jul 02 21:19:00.330645 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.221184    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal"
Jul 02 21:19:00.330645 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.232402    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-230.ec2.internal"
Jul 02 21:19:00.330645 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.244795    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 21:19:00.330645 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.247171    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9a5d467eae83b5ac2b132e3f387ffbc3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal\" (UID: \"9a5d467eae83b5ac2b132e3f387ffbc3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal"
Jul 02 21:19:00.330645 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.247198    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a5d467eae83b5ac2b132e3f387ffbc3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal\" (UID: \"9a5d467eae83b5ac2b132e3f387ffbc3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal"
Jul 02 21:19:00.330645 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.247219    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/005b67a3b8a4751eb2ba150e034470d6-config\") pod \"kube-apiserver-proxy-ip-10-0-130-230.ec2.internal\" (UID: \"005b67a3b8a4751eb2ba150e034470d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-230.ec2.internal"
Jul 02 21:19:00.330645 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.289844    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 21:19:00.349101 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.348432    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9a5d467eae83b5ac2b132e3f387ffbc3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal\" (UID: \"9a5d467eae83b5ac2b132e3f387ffbc3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal"
Jul 02 21:19:00.349101 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.348476    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9a5d467eae83b5ac2b132e3f387ffbc3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal\" (UID: \"9a5d467eae83b5ac2b132e3f387ffbc3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal"
Jul 02 21:19:00.349101 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.348500    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a5d467eae83b5ac2b132e3f387ffbc3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal\" (UID: \"9a5d467eae83b5ac2b132e3f387ffbc3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal"
Jul 02 21:19:00.349101 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.348476    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a5d467eae83b5ac2b132e3f387ffbc3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal\" (UID: \"9a5d467eae83b5ac2b132e3f387ffbc3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal"
Jul 02 21:19:00.349101 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.348530    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/005b67a3b8a4751eb2ba150e034470d6-config\") pod \"kube-apiserver-proxy-ip-10-0-130-230.ec2.internal\" (UID: \"005b67a3b8a4751eb2ba150e034470d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-230.ec2.internal"
Jul 02 21:19:00.349101 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.348557    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/005b67a3b8a4751eb2ba150e034470d6-config\") pod \"kube-apiserver-proxy-ip-10-0-130-230.ec2.internal\" (UID: \"005b67a3b8a4751eb2ba150e034470d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-230.ec2.internal"
Jul 02 21:19:00.560400 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.548798    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-230.ec2.internal"
Jul 02 21:19:00.679699 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.592900    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal"
Jul 02 21:19:00.679699 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.652279    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 02 21:19:00.679699 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.652436    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 02 21:19:00.679699 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.652473    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 02 21:19:00.679699 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.652501    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 02 21:19:00.763930 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.763828    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 21:14:00 +0000 UTC" deadline="2027-12-21 01:54:05.169905957 +0000 UTC"
Jul 02 21:19:00.763930 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.763890    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12868h35m4.406018718s"
Jul 02 21:19:00.786516 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.786486    2564 apiserver.go:52] "Watching apiserver"
Jul 02 21:19:00.807837 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.807817    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 02 21:19:00.808224 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.808203    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-tm8gq","openshift-network-operator/iptables-alerter-dhs4b","openshift-ovn-kubernetes/ovnkube-node-hlmv7","kube-system/konnectivity-agent-zd5dg","kube-system/kube-apiserver-proxy-ip-10-0-130-230.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc","openshift-dns/node-resolver-6k66x","openshift-image-registry/node-ca-ctb57","openshift-multus/multus-cg58w","openshift-multus/network-metrics-daemon-7rzx4","openshift-network-diagnostics/network-check-target-65mqk","kube-system/global-pull-secret-syncer-f8nf7","openshift-cluster-node-tuning-operator/tuned-5rn2k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal"]
Jul 02 21:19:00.809712 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.809698    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6k66x"
Jul 02 21:19:00.810780 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.810759    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:00.810892 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:00.810860    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65mqk" podUID="238eb474-294d-49e5-8086-6b9cf9711ac6"
Jul 02 21:19:00.812664 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.812650    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.815531 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.815492    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zd5dg"
Jul 02 21:19:00.816636 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.816611    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ctb57"
Jul 02 21:19:00.817160 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.817139    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 02 21:19:00.817328 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.817310    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 02 21:19:00.817415 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.817383    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 02 21:19:00.818051 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.818032    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc"
Jul 02 21:19:00.818793 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.818771    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 02 21:19:00.818891 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.818823    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 02 21:19:00.818891 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.818852    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jgkd9\""
Jul 02 21:19:00.819425 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.819409    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.820732 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.820708    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.822181 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.822158    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dhs4b"
Jul 02 21:19:00.823649 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.823629    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:00.823884 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:00.823771    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:00.824751 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.824730    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 02 21:19:00.824855 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.824813    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 02 21:19:00.824855 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.824836    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jsx94\""
Jul 02 21:19:00.824967 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.824838    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 02 21:19:00.824967 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.824936    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 02 21:19:00.825574 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.825556    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tm8gq"
Jul 02 21:19:00.825825 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.825796    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 02 21:19:00.825825 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.825810    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 02 21:19:00.826010 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.825851    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 02 21:19:00.826010 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.825806    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 02 21:19:00.826407 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.826393    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 02 21:19:00.826711 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.826694    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hfw7r\""
Jul 02 21:19:00.826795 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.826731    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 02 21:19:00.826795 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.826750    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-szc9x\""
Jul 02 21:19:00.826795 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.826733    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tczz5\""
Jul 02 21:19:00.826795 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.826700    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 02 21:19:00.827059 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.826821    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 02 21:19:00.827059 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.826762    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:00.827059 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:00.826901    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8nf7" podUID="453c93ba-280d-45f1-8ade-994e6242ccc7"
Jul 02 21:19:00.827337 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.827323    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 02 21:19:00.827403 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.827327    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 02 21:19:00.828147 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.828130    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 02 21:19:00.829109 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.829092    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nmg2z\""
Jul 02 21:19:00.829215 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.829091    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 02 21:19:00.829310 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.829294    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 02 21:19:00.829515 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.829502    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vb7wc\""
Jul 02 21:19:00.829571 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.829557    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 02 21:19:00.830274 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.830258    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 02 21:19:00.834346 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.834324    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 02 21:19:00.834441 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.834407    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d4pz6\""
Jul 02 21:19:00.834600 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.834580    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j9jmq\""
Jul 02 21:19:00.834682 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.834604    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 02 21:19:00.834682 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.834616    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 02 21:19:00.844594 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.844569    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 02 21:19:00.845298 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.845285    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 02 21:19:00.849266 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849250    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-var-lib-openvswitch\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.849318 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849274    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-run-openvswitch\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.849318 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849289    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-run-ovn\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.849318 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849304    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-host-cni-netd\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.849440 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849321    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rf9z\" (UniqueName: \"kubernetes.io/projected/53db7e45-3dab-4292-930c-8628a2f51800-kube-api-access-2rf9z\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.849440 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849337    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-etc-kubernetes\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.849440 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849387    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w42mm\" (UniqueName: \"kubernetes.io/projected/188c6c1c-be85-48e2-9cbb-9ab9722bb2b4-kube-api-access-w42mm\") pod \"node-resolver-6k66x\" (UID: \"188c6c1c-be85-48e2-9cbb-9ab9722bb2b4\") " pod="openshift-dns/node-resolver-6k66x"
Jul 02 21:19:00.849440 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849416    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-lib-modules\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.849597 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849477    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-host\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.849597 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849502    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zmzq\" (UniqueName: \"kubernetes.io/projected/de79c9aa-e167-492c-af48-e818dafe9d6a-kube-api-access-9zmzq\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.849597 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849529    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b0f5fcb9-715f-4dc1-915b-ae229225e5c2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tm8gq\" (UID: \"b0f5fcb9-715f-4dc1-915b-ae229225e5c2\") " pod="openshift-multus/multus-additional-cni-plugins-tm8gq"
Jul 02 21:19:00.849597 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849546    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-system-cni-dir\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.849762 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849590    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-multus-conf-dir\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.849762 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849624    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-host-run-multus-certs\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.849762 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849653    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b5sw\" (UniqueName: \"kubernetes.io/projected/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-kube-api-access-6b5sw\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.849762 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849675    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-run-systemd\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.849762 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849695    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-etc-sysconfig\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.849762 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849713    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-host-var-lib-cni-multus\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.849762 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849737    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de79c9aa-e167-492c-af48-e818dafe9d6a-etc-tuned\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.849762 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849757    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de79c9aa-e167-492c-af48-e818dafe9d6a-tmp\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.850092 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849773    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slw7v\" (UniqueName: \"kubernetes.io/projected/238eb474-294d-49e5-8086-6b9cf9711ac6-kube-api-access-slw7v\") pod \"network-check-target-65mqk\" (UID: \"238eb474-294d-49e5-8086-6b9cf9711ac6\") " pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:00.850092 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849789    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-host-run-k8s-cni-cncf-io\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.850092 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849814    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-host-kubelet\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.850092 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849830    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-host-slash\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.850092 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849844    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-node-log\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.850092 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849859    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/53db7e45-3dab-4292-930c-8628a2f51800-ovn-node-metrics-cert\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.850092 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849875    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-multus-cni-dir\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.850092 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849905    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-host-var-lib-kubelet\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.850092 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849936    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-sys\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.850092 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.849978    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-host-run-netns\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.850092 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850025    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.850092 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850052    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-etc-sysctl-conf\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.850092 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850084    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b0f5fcb9-715f-4dc1-915b-ae229225e5c2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tm8gq\" (UID: \"b0f5fcb9-715f-4dc1-915b-ae229225e5c2\") " pod="openshift-multus/multus-additional-cni-plugins-tm8gq"
Jul 02 21:19:00.850563 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850110    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/39ea2cc9-ac41-4474-99dd-1ce8c95d8b87-iptables-alerter-script\") pod \"iptables-alerter-dhs4b\" (UID: \"39ea2cc9-ac41-4474-99dd-1ce8c95d8b87\") " pod="openshift-network-operator/iptables-alerter-dhs4b"
Jul 02 21:19:00.850563 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850146    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/53db7e45-3dab-4292-930c-8628a2f51800-ovnkube-config\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.850563 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850166    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/53db7e45-3dab-4292-930c-8628a2f51800-env-overrides\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.850563 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850181    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b0f5fcb9-715f-4dc1-915b-ae229225e5c2-cnibin\") pod \"multus-additional-cni-plugins-tm8gq\" (UID: \"b0f5fcb9-715f-4dc1-915b-ae229225e5c2\") " pod="openshift-multus/multus-additional-cni-plugins-tm8gq"
Jul 02 21:19:00.850563 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850200    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188c6c1c-be85-48e2-9cbb-9ab9722bb2b4-tmp-dir\") pod \"node-resolver-6k66x\" (UID: \"188c6c1c-be85-48e2-9cbb-9ab9722bb2b4\") " pod="openshift-dns/node-resolver-6k66x"
Jul 02 21:19:00.850563 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850231    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-host-run-netns\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.850563 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850266    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-etc-openvswitch\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.850563 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850290    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-strxr\" (UniqueName: \"kubernetes.io/projected/0a417763-5387-48be-9a81-3055e4a86b7e-kube-api-access-strxr\") pod \"aws-ebs-csi-driver-node-vjbzc\" (UID: \"0a417763-5387-48be-9a81-3055e4a86b7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc"
Jul 02 21:19:00.850563 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850332    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-etc-modprobe-d\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.850563 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850351    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-etc-systemd\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.850563 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850385    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/453c93ba-280d-45f1-8ade-994e6242ccc7-kubelet-config\") pod \"global-pull-secret-syncer-f8nf7\" (UID: \"453c93ba-280d-45f1-8ade-994e6242ccc7\") " pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:00.850563 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850406    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-cnibin\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.850563 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850420    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-host-var-lib-cni-bin\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.850563 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850435    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-log-socket\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.850563 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850455    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0a417763-5387-48be-9a81-3055e4a86b7e-etc-selinux\") pod \"aws-ebs-csi-driver-node-vjbzc\" (UID: \"0a417763-5387-48be-9a81-3055e4a86b7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc"
Jul 02 21:19:00.850563 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850490    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-hostroot\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.851064 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850516    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b0f5fcb9-715f-4dc1-915b-ae229225e5c2-system-cni-dir\") pod \"multus-additional-cni-plugins-tm8gq\" (UID: \"b0f5fcb9-715f-4dc1-915b-ae229225e5c2\") " pod="openshift-multus/multus-additional-cni-plugins-tm8gq"
Jul 02 21:19:00.851064 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850542    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b0f5fcb9-715f-4dc1-915b-ae229225e5c2-cni-binary-copy\") pod \"multus-additional-cni-plugins-tm8gq\" (UID: \"b0f5fcb9-715f-4dc1-915b-ae229225e5c2\") " pod="openshift-multus/multus-additional-cni-plugins-tm8gq"
Jul 02 21:19:00.851064 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850569    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-os-release\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.851064 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850603    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-cni-binary-copy\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.851064 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850642    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b988618a-9a30-40af-964a-6c68d01dacfb-agent-certs\") pod \"konnectivity-agent-zd5dg\" (UID: \"b988618a-9a30-40af-964a-6c68d01dacfb\") " pod="kube-system/konnectivity-agent-zd5dg"
Jul 02 21:19:00.851064 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850661    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-etc-kubernetes\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.851064 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850679    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0a417763-5387-48be-9a81-3055e4a86b7e-registration-dir\") pod \"aws-ebs-csi-driver-node-vjbzc\" (UID: \"0a417763-5387-48be-9a81-3055e4a86b7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc"
Jul 02 21:19:00.851064 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850699    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0a417763-5387-48be-9a81-3055e4a86b7e-device-dir\") pod \"aws-ebs-csi-driver-node-vjbzc\" (UID: \"0a417763-5387-48be-9a81-3055e4a86b7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc"
Jul 02 21:19:00.851064 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850726    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0a417763-5387-48be-9a81-3055e4a86b7e-sys-fs\") pod \"aws-ebs-csi-driver-node-vjbzc\" (UID: \"0a417763-5387-48be-9a81-3055e4a86b7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc"
Jul 02 21:19:00.851064 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850744    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgrl2\" (UniqueName: \"kubernetes.io/projected/426d3178-3121-44cc-b885-d8b7a3b902a9-kube-api-access-hgrl2\") pod \"node-ca-ctb57\" (UID: \"426d3178-3121-44cc-b885-d8b7a3b902a9\") " pod="openshift-image-registry/node-ca-ctb57"
Jul 02 21:19:00.851064 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850772    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9gpb\" (UniqueName: \"kubernetes.io/projected/39ea2cc9-ac41-4474-99dd-1ce8c95d8b87-kube-api-access-s9gpb\") pod \"iptables-alerter-dhs4b\" (UID: \"39ea2cc9-ac41-4474-99dd-1ce8c95d8b87\") " pod="openshift-network-operator/iptables-alerter-dhs4b"
Jul 02 21:19:00.851064 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850804    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-host-run-ovn-kubernetes\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.851064 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850823    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-etc-sysctl-d\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.851064 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850838    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-run\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.851064 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850853    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fc43c39-ea1c-47da-b658-7834776917a8-metrics-certs\") pod \"network-metrics-daemon-7rzx4\" (UID: \"7fc43c39-ea1c-47da-b658-7834776917a8\") " pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:00.851064 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850868    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b0f5fcb9-715f-4dc1-915b-ae229225e5c2-os-release\") pod \"multus-additional-cni-plugins-tm8gq\" (UID: \"b0f5fcb9-715f-4dc1-915b-ae229225e5c2\") " pod="openshift-multus/multus-additional-cni-plugins-tm8gq"
Jul 02 21:19:00.851544 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850884    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcsjr\" (UniqueName: \"kubernetes.io/projected/b0f5fcb9-715f-4dc1-915b-ae229225e5c2-kube-api-access-kcsjr\") pod \"multus-additional-cni-plugins-tm8gq\" (UID: \"b0f5fcb9-715f-4dc1-915b-ae229225e5c2\") " pod="openshift-multus/multus-additional-cni-plugins-tm8gq"
Jul 02 21:19:00.851544 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850906    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b988618a-9a30-40af-964a-6c68d01dacfb-konnectivity-ca\") pod \"konnectivity-agent-zd5dg\" (UID: \"b988618a-9a30-40af-964a-6c68d01dacfb\") " pod="kube-system/konnectivity-agent-zd5dg"
Jul 02 21:19:00.851544 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850925    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/426d3178-3121-44cc-b885-d8b7a3b902a9-host\") pod \"node-ca-ctb57\" (UID: \"426d3178-3121-44cc-b885-d8b7a3b902a9\") " pod="openshift-image-registry/node-ca-ctb57"
Jul 02 21:19:00.851544 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850948    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-host-cni-bin\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.851544 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850967    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/53db7e45-3dab-4292-930c-8628a2f51800-ovnkube-script-lib\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.851544 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850982    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-var-lib-kubelet\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.851544 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.850997    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwfhq\" (UniqueName: \"kubernetes.io/projected/7fc43c39-ea1c-47da-b658-7834776917a8-kube-api-access-hwfhq\") pod \"network-metrics-daemon-7rzx4\" (UID: \"7fc43c39-ea1c-47da-b658-7834776917a8\") " pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:00.851544 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.851021    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-multus-socket-dir-parent\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.851544 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.851040    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39ea2cc9-ac41-4474-99dd-1ce8c95d8b87-host-slash\") pod \"iptables-alerter-dhs4b\" (UID: \"39ea2cc9-ac41-4474-99dd-1ce8c95d8b87\") " pod="openshift-network-operator/iptables-alerter-dhs4b"
Jul 02 21:19:00.851544 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.851055    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0a417763-5387-48be-9a81-3055e4a86b7e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vjbzc\" (UID: \"0a417763-5387-48be-9a81-3055e4a86b7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc"
Jul 02 21:19:00.851544 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.851070    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/453c93ba-280d-45f1-8ade-994e6242ccc7-original-pull-secret\") pod \"global-pull-secret-syncer-f8nf7\" (UID: \"453c93ba-280d-45f1-8ade-994e6242ccc7\") " pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:00.851544 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.851085    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/188c6c1c-be85-48e2-9cbb-9ab9722bb2b4-hosts-file\") pod \"node-resolver-6k66x\" (UID: \"188c6c1c-be85-48e2-9cbb-9ab9722bb2b4\") " pod="openshift-dns/node-resolver-6k66x"
Jul 02 21:19:00.851544 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.851110    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-multus-daemon-config\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.851544 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.851162    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-systemd-units\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.851544 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.851196    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0a417763-5387-48be-9a81-3055e4a86b7e-socket-dir\") pod \"aws-ebs-csi-driver-node-vjbzc\" (UID: \"0a417763-5387-48be-9a81-3055e4a86b7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc"
Jul 02 21:19:00.851544 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.851223    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b0f5fcb9-715f-4dc1-915b-ae229225e5c2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tm8gq\" (UID: \"b0f5fcb9-715f-4dc1-915b-ae229225e5c2\") " pod="openshift-multus/multus-additional-cni-plugins-tm8gq"
Jul 02 21:19:00.852121 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.851247    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/453c93ba-280d-45f1-8ade-994e6242ccc7-dbus\") pod \"global-pull-secret-syncer-f8nf7\" (UID: \"453c93ba-280d-45f1-8ade-994e6242ccc7\") " pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:00.852121 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.851269    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/426d3178-3121-44cc-b885-d8b7a3b902a9-serviceca\") pod \"node-ca-ctb57\" (UID: \"426d3178-3121-44cc-b885-d8b7a3b902a9\") " pod="openshift-image-registry/node-ca-ctb57"
Jul 02 21:19:00.858096 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.858068    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 21:19:00.886018 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.886001    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jfgh5"
Jul 02 21:19:00.894788 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.894773    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jfgh5"
Jul 02 21:19:00.951700 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.951676    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-var-lib-kubelet\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.951807 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.951706    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwfhq\" (UniqueName: \"kubernetes.io/projected/7fc43c39-ea1c-47da-b658-7834776917a8-kube-api-access-hwfhq\") pod \"network-metrics-daemon-7rzx4\" (UID: \"7fc43c39-ea1c-47da-b658-7834776917a8\") " pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:00.951807 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.951725    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-multus-socket-dir-parent\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.951807 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.951749    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39ea2cc9-ac41-4474-99dd-1ce8c95d8b87-host-slash\") pod \"iptables-alerter-dhs4b\" (UID: \"39ea2cc9-ac41-4474-99dd-1ce8c95d8b87\") " pod="openshift-network-operator/iptables-alerter-dhs4b"
Jul 02 21:19:00.951807 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.951773    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0a417763-5387-48be-9a81-3055e4a86b7e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vjbzc\" (UID: \"0a417763-5387-48be-9a81-3055e4a86b7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc"
Jul 02 21:19:00.951807 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.951798    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/453c93ba-280d-45f1-8ade-994e6242ccc7-original-pull-secret\") pod \"global-pull-secret-syncer-f8nf7\" (UID: \"453c93ba-280d-45f1-8ade-994e6242ccc7\") " pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:00.952057 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.951819    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/188c6c1c-be85-48e2-9cbb-9ab9722bb2b4-hosts-file\") pod \"node-resolver-6k66x\" (UID: \"188c6c1c-be85-48e2-9cbb-9ab9722bb2b4\") " pod="openshift-dns/node-resolver-6k66x"
Jul 02 21:19:00.952057 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.951842    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-multus-daemon-config\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.952057 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.951840    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-var-lib-kubelet\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.952057 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.951865    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-systemd-units\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.952057 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.951847    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39ea2cc9-ac41-4474-99dd-1ce8c95d8b87-host-slash\") pod \"iptables-alerter-dhs4b\" (UID: \"39ea2cc9-ac41-4474-99dd-1ce8c95d8b87\") " pod="openshift-network-operator/iptables-alerter-dhs4b"
Jul 02 21:19:00.952057 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.951888    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0a417763-5387-48be-9a81-3055e4a86b7e-socket-dir\") pod \"aws-ebs-csi-driver-node-vjbzc\" (UID: \"0a417763-5387-48be-9a81-3055e4a86b7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc"
Jul 02 21:19:00.952057 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.951915    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b0f5fcb9-715f-4dc1-915b-ae229225e5c2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tm8gq\" (UID: \"b0f5fcb9-715f-4dc1-915b-ae229225e5c2\") " pod="openshift-multus/multus-additional-cni-plugins-tm8gq"
Jul 02 21:19:00.952057 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.951918    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-multus-socket-dir-parent\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.952057 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.951955    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/453c93ba-280d-45f1-8ade-994e6242ccc7-dbus\") pod \"global-pull-secret-syncer-f8nf7\" (UID: \"453c93ba-280d-45f1-8ade-994e6242ccc7\") " pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:00.952057 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:00.951973    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:00.952057 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.951982    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/426d3178-3121-44cc-b885-d8b7a3b902a9-serviceca\") pod \"node-ca-ctb57\" (UID: \"426d3178-3121-44cc-b885-d8b7a3b902a9\") " pod="openshift-image-registry/node-ca-ctb57"
Jul 02 21:19:00.952057 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952007    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-var-lib-openvswitch\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.952057 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952012    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/188c6c1c-be85-48e2-9cbb-9ab9722bb2b4-hosts-file\") pod \"node-resolver-6k66x\" (UID: \"188c6c1c-be85-48e2-9cbb-9ab9722bb2b4\") " pod="openshift-dns/node-resolver-6k66x"
Jul 02 21:19:00.952057 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952033    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-run-openvswitch\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.952794 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:00.952070    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/453c93ba-280d-45f1-8ade-994e6242ccc7-original-pull-secret podName:453c93ba-280d-45f1-8ade-994e6242ccc7 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:01.452044158 +0000 UTC m=+2.289284191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/453c93ba-280d-45f1-8ade-994e6242ccc7-original-pull-secret") pod "global-pull-secret-syncer-f8nf7" (UID: "453c93ba-280d-45f1-8ade-994e6242ccc7") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:00.952794 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952095    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-run-ovn\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.952794 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952102    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0a417763-5387-48be-9a81-3055e4a86b7e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vjbzc\" (UID: \"0a417763-5387-48be-9a81-3055e4a86b7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc"
Jul 02 21:19:00.952794 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952107    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-run-openvswitch\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.952794 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952129    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-host-cni-netd\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.952794 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952161    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/453c93ba-280d-45f1-8ade-994e6242ccc7-dbus\") pod \"global-pull-secret-syncer-f8nf7\" (UID: \"453c93ba-280d-45f1-8ade-994e6242ccc7\") " pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:00.952794 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952176    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-var-lib-openvswitch\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.952794 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952219    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-host-cni-netd\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.952794 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952221    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-systemd-units\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.952794 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952250    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rf9z\" (UniqueName: \"kubernetes.io/projected/53db7e45-3dab-4292-930c-8628a2f51800-kube-api-access-2rf9z\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.952794 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952255    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-run-ovn\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.952794 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952287    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-etc-kubernetes\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.952794 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952410    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w42mm\" (UniqueName: \"kubernetes.io/projected/188c6c1c-be85-48e2-9cbb-9ab9722bb2b4-kube-api-access-w42mm\") pod \"node-resolver-6k66x\" (UID: \"188c6c1c-be85-48e2-9cbb-9ab9722bb2b4\") " pod="openshift-dns/node-resolver-6k66x"
Jul 02 21:19:00.952794 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952447    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-etc-kubernetes\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.952794 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952481    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b0f5fcb9-715f-4dc1-915b-ae229225e5c2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tm8gq\" (UID: \"b0f5fcb9-715f-4dc1-915b-ae229225e5c2\") " pod="openshift-multus/multus-additional-cni-plugins-tm8gq"
Jul 02 21:19:00.952794 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952496    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-lib-modules\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.953586 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952482    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0a417763-5387-48be-9a81-3055e4a86b7e-socket-dir\") pod \"aws-ebs-csi-driver-node-vjbzc\" (UID: \"0a417763-5387-48be-9a81-3055e4a86b7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc"
Jul 02 21:19:00.953586 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952557    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-multus-daemon-config\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.953586 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952639    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-host\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.953586 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952669    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/426d3178-3121-44cc-b885-d8b7a3b902a9-serviceca\") pod \"node-ca-ctb57\" (UID: \"426d3178-3121-44cc-b885-d8b7a3b902a9\") " pod="openshift-image-registry/node-ca-ctb57"
Jul 02 21:19:00.953586 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952669    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zmzq\" (UniqueName: \"kubernetes.io/projected/de79c9aa-e167-492c-af48-e818dafe9d6a-kube-api-access-9zmzq\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.953586 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952702    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-host\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.953586 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952700    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-lib-modules\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.953586 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952721    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b0f5fcb9-715f-4dc1-915b-ae229225e5c2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tm8gq\" (UID: \"b0f5fcb9-715f-4dc1-915b-ae229225e5c2\") " pod="openshift-multus/multus-additional-cni-plugins-tm8gq"
Jul 02 21:19:00.953586 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952747    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-system-cni-dir\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.953586 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952772    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-multus-conf-dir\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.953586 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952798    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-host-run-multus-certs\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.953586 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952831    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-multus-conf-dir\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.953586 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952830    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-system-cni-dir\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.953586 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952831    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b5sw\" (UniqueName: \"kubernetes.io/projected/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-kube-api-access-6b5sw\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.953586 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952876    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-run-systemd\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.953586 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952904    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-etc-sysconfig\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.953586 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952908    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-host-run-multus-certs\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.953586 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952929    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-host-var-lib-cni-multus\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.954575 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952975    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-run-systemd\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.954575 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952978    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-host-var-lib-cni-multus\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.954575 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.952998    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-etc-sysconfig\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.954575 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953012    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de79c9aa-e167-492c-af48-e818dafe9d6a-etc-tuned\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.954575 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953031    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de79c9aa-e167-492c-af48-e818dafe9d6a-tmp\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.954575 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953085    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slw7v\" (UniqueName: \"kubernetes.io/projected/238eb474-294d-49e5-8086-6b9cf9711ac6-kube-api-access-slw7v\") pod \"network-check-target-65mqk\" (UID: \"238eb474-294d-49e5-8086-6b9cf9711ac6\") " pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:00.954575 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953146    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-host-run-k8s-cni-cncf-io\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.954575 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953171    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-host-kubelet\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.954575 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953192    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-host-slash\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.954575 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953214    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-node-log\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.954575 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953238    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/53db7e45-3dab-4292-930c-8628a2f51800-ovn-node-metrics-cert\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.954575 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953257    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b0f5fcb9-715f-4dc1-915b-ae229225e5c2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tm8gq\" (UID: \"b0f5fcb9-715f-4dc1-915b-ae229225e5c2\") " pod="openshift-multus/multus-additional-cni-plugins-tm8gq"
Jul 02 21:19:00.954575 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953266    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-host-run-k8s-cni-cncf-io\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.954575 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953264    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-multus-cni-dir\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.954575 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953319    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-node-log\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.954575 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953265    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-host-kubelet\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.954575 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953358    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-host-var-lib-kubelet\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.954575 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953320    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-host-var-lib-kubelet\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.955516 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953384    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-multus-cni-dir\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.955516 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953394    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-host-slash\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.955516 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953401    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-sys\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.955516 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953422    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-host-run-netns\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.955516 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953439    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.955516 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953438    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 02 21:19:00.955516 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953464    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-etc-sysctl-conf\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.955516 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953485    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-host-run-netns\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.955516 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953484    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-sys\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.955516 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953508    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.955516 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953539    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b0f5fcb9-715f-4dc1-915b-ae229225e5c2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tm8gq\" (UID: \"b0f5fcb9-715f-4dc1-915b-ae229225e5c2\") " pod="openshift-multus/multus-additional-cni-plugins-tm8gq"
Jul 02 21:19:00.955516 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953570    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/39ea2cc9-ac41-4474-99dd-1ce8c95d8b87-iptables-alerter-script\") pod \"iptables-alerter-dhs4b\" (UID: \"39ea2cc9-ac41-4474-99dd-1ce8c95d8b87\") " pod="openshift-network-operator/iptables-alerter-dhs4b"
Jul 02 21:19:00.955516 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953582    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-etc-sysctl-conf\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.955516 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953600    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/53db7e45-3dab-4292-930c-8628a2f51800-ovnkube-config\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.955516 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953627    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/53db7e45-3dab-4292-930c-8628a2f51800-env-overrides\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.955516 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953653    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b0f5fcb9-715f-4dc1-915b-ae229225e5c2-cnibin\") pod \"multus-additional-cni-plugins-tm8gq\" (UID: \"b0f5fcb9-715f-4dc1-915b-ae229225e5c2\") " pod="openshift-multus/multus-additional-cni-plugins-tm8gq"
Jul 02 21:19:00.955516 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953703    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b0f5fcb9-715f-4dc1-915b-ae229225e5c2-cnibin\") pod \"multus-additional-cni-plugins-tm8gq\" (UID: \"b0f5fcb9-715f-4dc1-915b-ae229225e5c2\") " pod="openshift-multus/multus-additional-cni-plugins-tm8gq"
Jul 02 21:19:00.955516 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953724    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b0f5fcb9-715f-4dc1-915b-ae229225e5c2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tm8gq\" (UID: \"b0f5fcb9-715f-4dc1-915b-ae229225e5c2\") " pod="openshift-multus/multus-additional-cni-plugins-tm8gq"
Jul 02 21:19:00.956463 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953756    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188c6c1c-be85-48e2-9cbb-9ab9722bb2b4-tmp-dir\") pod \"node-resolver-6k66x\" (UID: \"188c6c1c-be85-48e2-9cbb-9ab9722bb2b4\") " pod="openshift-dns/node-resolver-6k66x"
Jul 02 21:19:00.956463 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953782    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-host-run-netns\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.956463 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953799    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-etc-openvswitch\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.956463 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953823    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-strxr\" (UniqueName: \"kubernetes.io/projected/0a417763-5387-48be-9a81-3055e4a86b7e-kube-api-access-strxr\") pod \"aws-ebs-csi-driver-node-vjbzc\" (UID: \"0a417763-5387-48be-9a81-3055e4a86b7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc"
Jul 02 21:19:00.956463 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953852    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-etc-modprobe-d\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.956463 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953876    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-etc-systemd\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.956463 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953896    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/453c93ba-280d-45f1-8ade-994e6242ccc7-kubelet-config\") pod \"global-pull-secret-syncer-f8nf7\" (UID: \"453c93ba-280d-45f1-8ade-994e6242ccc7\") " pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:00.956463 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953923    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-cnibin\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.956463 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.953959    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-host-var-lib-cni-bin\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.956463 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954009    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-log-socket\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.956463 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954040    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0a417763-5387-48be-9a81-3055e4a86b7e-etc-selinux\") pod \"aws-ebs-csi-driver-node-vjbzc\" (UID: \"0a417763-5387-48be-9a81-3055e4a86b7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc"
Jul 02 21:19:00.956463 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954067    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-hostroot\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.956463 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954093    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b0f5fcb9-715f-4dc1-915b-ae229225e5c2-system-cni-dir\") pod \"multus-additional-cni-plugins-tm8gq\" (UID: \"b0f5fcb9-715f-4dc1-915b-ae229225e5c2\") " pod="openshift-multus/multus-additional-cni-plugins-tm8gq"
Jul 02 21:19:00.956463 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954123    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b0f5fcb9-715f-4dc1-915b-ae229225e5c2-cni-binary-copy\") pod \"multus-additional-cni-plugins-tm8gq\" (UID: \"b0f5fcb9-715f-4dc1-915b-ae229225e5c2\") " pod="openshift-multus/multus-additional-cni-plugins-tm8gq"
Jul 02 21:19:00.956463 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954151    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-os-release\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.956463 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954177    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-cni-binary-copy\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.956463 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954212    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/53db7e45-3dab-4292-930c-8628a2f51800-env-overrides\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.957118 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954218    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b988618a-9a30-40af-964a-6c68d01dacfb-agent-certs\") pod \"konnectivity-agent-zd5dg\" (UID: \"b988618a-9a30-40af-964a-6c68d01dacfb\") " pod="kube-system/konnectivity-agent-zd5dg"
Jul 02 21:19:00.957118 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954247    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-etc-kubernetes\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.957118 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954265    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-host-var-lib-cni-bin\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.957118 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954272    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0a417763-5387-48be-9a81-3055e4a86b7e-registration-dir\") pod \"aws-ebs-csi-driver-node-vjbzc\" (UID: \"0a417763-5387-48be-9a81-3055e4a86b7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc"
Jul 02 21:19:00.957118 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954281    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b0f5fcb9-715f-4dc1-915b-ae229225e5c2-system-cni-dir\") pod \"multus-additional-cni-plugins-tm8gq\" (UID: \"b0f5fcb9-715f-4dc1-915b-ae229225e5c2\") " pod="openshift-multus/multus-additional-cni-plugins-tm8gq"
Jul 02 21:19:00.957118 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954299    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0a417763-5387-48be-9a81-3055e4a86b7e-device-dir\") pod \"aws-ebs-csi-driver-node-vjbzc\" (UID: \"0a417763-5387-48be-9a81-3055e4a86b7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc"
Jul 02 21:19:00.957118 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954314    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-host-run-netns\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.957118 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954326    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0a417763-5387-48be-9a81-3055e4a86b7e-sys-fs\") pod \"aws-ebs-csi-driver-node-vjbzc\" (UID: \"0a417763-5387-48be-9a81-3055e4a86b7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc"
Jul 02 21:19:00.957118 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954349    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-etc-openvswitch\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.957118 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954357    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgrl2\" (UniqueName: \"kubernetes.io/projected/426d3178-3121-44cc-b885-d8b7a3b902a9-kube-api-access-hgrl2\") pod \"node-ca-ctb57\" (UID: \"426d3178-3121-44cc-b885-d8b7a3b902a9\") " pod="openshift-image-registry/node-ca-ctb57"
Jul 02 21:19:00.957118 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954365    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/53db7e45-3dab-4292-930c-8628a2f51800-ovnkube-config\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.957118 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954384    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9gpb\" (UniqueName: \"kubernetes.io/projected/39ea2cc9-ac41-4474-99dd-1ce8c95d8b87-kube-api-access-s9gpb\") pod \"iptables-alerter-dhs4b\" (UID: \"39ea2cc9-ac41-4474-99dd-1ce8c95d8b87\") " pod="openshift-network-operator/iptables-alerter-dhs4b"
Jul 02 21:19:00.957118 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954414    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-host-run-ovn-kubernetes\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.957118 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954440    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-etc-sysctl-d\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.957118 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954489    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-run\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.957118 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954491    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0a417763-5387-48be-9a81-3055e4a86b7e-etc-selinux\") pod \"aws-ebs-csi-driver-node-vjbzc\" (UID: \"0a417763-5387-48be-9a81-3055e4a86b7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc"
Jul 02 21:19:00.957118 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954517    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fc43c39-ea1c-47da-b658-7834776917a8-metrics-certs\") pod \"network-metrics-daemon-7rzx4\" (UID: \"7fc43c39-ea1c-47da-b658-7834776917a8\") " pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:00.958098 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954533    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188c6c1c-be85-48e2-9cbb-9ab9722bb2b4-tmp-dir\") pod \"node-resolver-6k66x\" (UID: \"188c6c1c-be85-48e2-9cbb-9ab9722bb2b4\") " pod="openshift-dns/node-resolver-6k66x"
Jul 02 21:19:00.958098 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954544    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b0f5fcb9-715f-4dc1-915b-ae229225e5c2-os-release\") pod \"multus-additional-cni-plugins-tm8gq\" (UID: \"b0f5fcb9-715f-4dc1-915b-ae229225e5c2\") " pod="openshift-multus/multus-additional-cni-plugins-tm8gq"
Jul 02 21:19:00.958098 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954545    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-log-socket\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.958098 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954219    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/39ea2cc9-ac41-4474-99dd-1ce8c95d8b87-iptables-alerter-script\") pod \"iptables-alerter-dhs4b\" (UID: \"39ea2cc9-ac41-4474-99dd-1ce8c95d8b87\") " pod="openshift-network-operator/iptables-alerter-dhs4b"
Jul 02 21:19:00.958098 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954737    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-etc-modprobe-d\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.958098 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954777    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-etc-systemd\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.958098 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954831    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/453c93ba-280d-45f1-8ade-994e6242ccc7-kubelet-config\") pod \"global-pull-secret-syncer-f8nf7\" (UID: \"453c93ba-280d-45f1-8ade-994e6242ccc7\") " pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:00.958098 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954864    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcsjr\" (UniqueName: \"kubernetes.io/projected/b0f5fcb9-715f-4dc1-915b-ae229225e5c2-kube-api-access-kcsjr\") pod \"multus-additional-cni-plugins-tm8gq\" (UID: \"b0f5fcb9-715f-4dc1-915b-ae229225e5c2\") " pod="openshift-multus/multus-additional-cni-plugins-tm8gq"
Jul 02 21:19:00.958098 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954896    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b988618a-9a30-40af-964a-6c68d01dacfb-konnectivity-ca\") pod \"konnectivity-agent-zd5dg\" (UID: \"b988618a-9a30-40af-964a-6c68d01dacfb\") " pod="kube-system/konnectivity-agent-zd5dg"
Jul 02 21:19:00.958098 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954922    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/426d3178-3121-44cc-b885-d8b7a3b902a9-host\") pod \"node-ca-ctb57\" (UID: \"426d3178-3121-44cc-b885-d8b7a3b902a9\") " pod="openshift-image-registry/node-ca-ctb57"
Jul 02 21:19:00.958098 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954949    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-host-cni-bin\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.958098 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954978    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/53db7e45-3dab-4292-930c-8628a2f51800-ovnkube-script-lib\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.958098 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.954953    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-hostroot\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.958098 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.955034    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-cnibin\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.958098 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.955065    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/426d3178-3121-44cc-b885-d8b7a3b902a9-host\") pod \"node-ca-ctb57\" (UID: \"426d3178-3121-44cc-b885-d8b7a3b902a9\") " pod="openshift-image-registry/node-ca-ctb57"
Jul 02 21:19:00.958098 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.955105    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-host-cni-bin\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.958098 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.955112    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0a417763-5387-48be-9a81-3055e4a86b7e-device-dir\") pod \"aws-ebs-csi-driver-node-vjbzc\" (UID: \"0a417763-5387-48be-9a81-3055e4a86b7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc"
Jul 02 21:19:00.958098 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.955266    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-etc-sysctl-d\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.959129 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.955449    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0a417763-5387-48be-9a81-3055e4a86b7e-sys-fs\") pod \"aws-ebs-csi-driver-node-vjbzc\" (UID: \"0a417763-5387-48be-9a81-3055e4a86b7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc"
Jul 02 21:19:00.959129 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.955644    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-os-release\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.959129 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.955700    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53db7e45-3dab-4292-930c-8628a2f51800-host-run-ovn-kubernetes\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.959129 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.955739    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/53db7e45-3dab-4292-930c-8628a2f51800-ovnkube-script-lib\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.959129 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:00.955834    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:00.959129 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.955848    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-etc-kubernetes\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.959129 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:00.955888    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fc43c39-ea1c-47da-b658-7834776917a8-metrics-certs podName:7fc43c39-ea1c-47da-b658-7834776917a8 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:01.455870681 +0000 UTC m=+2.293110710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7fc43c39-ea1c-47da-b658-7834776917a8-metrics-certs") pod "network-metrics-daemon-7rzx4" (UID: "7fc43c39-ea1c-47da-b658-7834776917a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:00.959129 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.955907    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b0f5fcb9-715f-4dc1-915b-ae229225e5c2-os-release\") pod \"multus-additional-cni-plugins-tm8gq\" (UID: \"b0f5fcb9-715f-4dc1-915b-ae229225e5c2\") " pod="openshift-multus/multus-additional-cni-plugins-tm8gq"
Jul 02 21:19:00.959129 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.955930    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b0f5fcb9-715f-4dc1-915b-ae229225e5c2-cni-binary-copy\") pod \"multus-additional-cni-plugins-tm8gq\" (UID: \"b0f5fcb9-715f-4dc1-915b-ae229225e5c2\") " pod="openshift-multus/multus-additional-cni-plugins-tm8gq"
Jul 02 21:19:00.959129 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.955960    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0a417763-5387-48be-9a81-3055e4a86b7e-registration-dir\") pod \"aws-ebs-csi-driver-node-vjbzc\" (UID: \"0a417763-5387-48be-9a81-3055e4a86b7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc"
Jul 02 21:19:00.959129 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.956040    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de79c9aa-e167-492c-af48-e818dafe9d6a-run\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.959129 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.956256    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b988618a-9a30-40af-964a-6c68d01dacfb-konnectivity-ca\") pod \"konnectivity-agent-zd5dg\" (UID: \"b988618a-9a30-40af-964a-6c68d01dacfb\") " pod="kube-system/konnectivity-agent-zd5dg"
Jul 02 21:19:00.959129 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.956374    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-cni-binary-copy\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.959129 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.957393    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de79c9aa-e167-492c-af48-e818dafe9d6a-tmp\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.959129 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.957438    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de79c9aa-e167-492c-af48-e818dafe9d6a-etc-tuned\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.959129 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.957528    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/53db7e45-3dab-4292-930c-8628a2f51800-ovn-node-metrics-cert\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.959129 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.958207    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b988618a-9a30-40af-964a-6c68d01dacfb-agent-certs\") pod \"konnectivity-agent-zd5dg\" (UID: \"b988618a-9a30-40af-964a-6c68d01dacfb\") " pod="kube-system/konnectivity-agent-zd5dg"
Jul 02 21:19:00.961176 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:00.960988    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:19:00.961176 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:00.961014    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:19:00.961176 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:00.961035    2564 projected.go:196] Error preparing data for projected volume kube-api-access-slw7v for pod openshift-network-diagnostics/network-check-target-65mqk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:00.961176 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:00.961092    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/238eb474-294d-49e5-8086-6b9cf9711ac6-kube-api-access-slw7v podName:238eb474-294d-49e5-8086-6b9cf9711ac6 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:01.46107373 +0000 UTC m=+2.298313760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-slw7v" (UniqueName: "kubernetes.io/projected/238eb474-294d-49e5-8086-6b9cf9711ac6-kube-api-access-slw7v") pod "network-check-target-65mqk" (UID: "238eb474-294d-49e5-8086-6b9cf9711ac6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:00.963147 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.963121    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwfhq\" (UniqueName: \"kubernetes.io/projected/7fc43c39-ea1c-47da-b658-7834776917a8-kube-api-access-hwfhq\") pod \"network-metrics-daemon-7rzx4\" (UID: \"7fc43c39-ea1c-47da-b658-7834776917a8\") " pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:00.963815 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.963777    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rf9z\" (UniqueName: \"kubernetes.io/projected/53db7e45-3dab-4292-930c-8628a2f51800-kube-api-access-2rf9z\") pod \"ovnkube-node-hlmv7\" (UID: \"53db7e45-3dab-4292-930c-8628a2f51800\") " pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:00.963915 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.963899    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w42mm\" (UniqueName: \"kubernetes.io/projected/188c6c1c-be85-48e2-9cbb-9ab9722bb2b4-kube-api-access-w42mm\") pod \"node-resolver-6k66x\" (UID: \"188c6c1c-be85-48e2-9cbb-9ab9722bb2b4\") " pod="openshift-dns/node-resolver-6k66x"
Jul 02 21:19:00.964078 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.964036    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b5sw\" (UniqueName: \"kubernetes.io/projected/dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd-kube-api-access-6b5sw\") pod \"multus-cg58w\" (UID: \"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd\") " pod="openshift-multus/multus-cg58w"
Jul 02 21:19:00.964547 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.964498    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgrl2\" (UniqueName: \"kubernetes.io/projected/426d3178-3121-44cc-b885-d8b7a3b902a9-kube-api-access-hgrl2\") pod \"node-ca-ctb57\" (UID: \"426d3178-3121-44cc-b885-d8b7a3b902a9\") " pod="openshift-image-registry/node-ca-ctb57"
Jul 02 21:19:00.964644 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.964579    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zmzq\" (UniqueName: \"kubernetes.io/projected/de79c9aa-e167-492c-af48-e818dafe9d6a-kube-api-access-9zmzq\") pod \"tuned-5rn2k\" (UID: \"de79c9aa-e167-492c-af48-e818dafe9d6a\") " pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:00.964905 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.964880    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcsjr\" (UniqueName: \"kubernetes.io/projected/b0f5fcb9-715f-4dc1-915b-ae229225e5c2-kube-api-access-kcsjr\") pod \"multus-additional-cni-plugins-tm8gq\" (UID: \"b0f5fcb9-715f-4dc1-915b-ae229225e5c2\") " pod="openshift-multus/multus-additional-cni-plugins-tm8gq"
Jul 02 21:19:00.965560 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.965541    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9gpb\" (UniqueName: \"kubernetes.io/projected/39ea2cc9-ac41-4474-99dd-1ce8c95d8b87-kube-api-access-s9gpb\") pod \"iptables-alerter-dhs4b\" (UID: \"39ea2cc9-ac41-4474-99dd-1ce8c95d8b87\") " pod="openshift-network-operator/iptables-alerter-dhs4b"
Jul 02 21:19:00.965735 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:00.965719    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-strxr\" (UniqueName: \"kubernetes.io/projected/0a417763-5387-48be-9a81-3055e4a86b7e-kube-api-access-strxr\") pod \"aws-ebs-csi-driver-node-vjbzc\" (UID: \"0a417763-5387-48be-9a81-3055e4a86b7e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc"
Jul 02 21:19:01.138345 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:01.138271    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6k66x"
Jul 02 21:19:01.155476 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:01.155452    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:01.165365 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:01.165343    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zd5dg"
Jul 02 21:19:01.181296 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:01.181269    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ctb57"
Jul 02 21:19:01.197569 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:01.197541    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc"
Jul 02 21:19:01.209591 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:01.209568    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cg58w"
Jul 02 21:19:01.223638 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:01.223614    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5rn2k"
Jul 02 21:19:01.230720 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:01.230683    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dhs4b"
Jul 02 21:19:01.248731 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:01.248708    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tm8gq"
Jul 02 21:19:01.251111 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:01.251071    2564 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = creating pod sandbox with name \"k8s_kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal_openshift-machine-config-operator_9a5d467eae83b5ac2b132e3f387ffbc3_0\": unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 500 Internal Server Error; artifact err: provided artifact is a container image"
Jul 02 21:19:01.251202 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:01.251153    2564 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = creating pod sandbox with name \"k8s_kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal_openshift-machine-config-operator_9a5d467eae83b5ac2b132e3f387ffbc3_0\": unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 500 Internal Server Error; artifact err: provided artifact is a container image" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal"
Jul 02 21:19:01.251202 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:01.251179    2564 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = creating pod sandbox with name \"k8s_kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal_openshift-machine-config-operator_9a5d467eae83b5ac2b132e3f387ffbc3_0\": unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 500 Internal Server Error; artifact err: provided artifact is a container image" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal"
Jul 02 21:19:01.251282 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:01.251248    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal_openshift-machine-config-operator(9a5d467eae83b5ac2b132e3f387ffbc3)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal_openshift-machine-config-operator(9a5d467eae83b5ac2b132e3f387ffbc3)\\\": rpc error: code = Unknown desc = creating pod sandbox with name \\\"k8s_kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal_openshift-machine-config-operator_9a5d467eae83b5ac2b132e3f387ffbc3_0\\\": unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 500 Internal Server Error; artifact err: provided artifact is a container image\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal" podUID="9a5d467eae83b5ac2b132e3f387ffbc3"
Jul 02 21:19:01.289516 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:19:01.289486    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod005b67a3b8a4751eb2ba150e034470d6.slice/crio-f774caa1aff16400a7917043324cba6c05a5d4be6e2938372ccafecdf62460ce WatchSource:0}: Error finding container f774caa1aff16400a7917043324cba6c05a5d4be6e2938372ccafecdf62460ce: Status 404 returned error can't find the container with id f774caa1aff16400a7917043324cba6c05a5d4be6e2938372ccafecdf62460ce
Jul 02 21:19:01.293659 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:01.293642    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 21:19:01.460684 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:01.460604    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fc43c39-ea1c-47da-b658-7834776917a8-metrics-certs\") pod \"network-metrics-daemon-7rzx4\" (UID: \"7fc43c39-ea1c-47da-b658-7834776917a8\") " pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:01.460684 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:01.460646    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/453c93ba-280d-45f1-8ade-994e6242ccc7-original-pull-secret\") pod \"global-pull-secret-syncer-f8nf7\" (UID: \"453c93ba-280d-45f1-8ade-994e6242ccc7\") " pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:01.460866 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:01.460760    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:01.460866 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:01.460766    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:01.460866 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:01.460816    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/453c93ba-280d-45f1-8ade-994e6242ccc7-original-pull-secret podName:453c93ba-280d-45f1-8ade-994e6242ccc7 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:02.460802214 +0000 UTC m=+3.298042242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/453c93ba-280d-45f1-8ade-994e6242ccc7-original-pull-secret") pod "global-pull-secret-syncer-f8nf7" (UID: "453c93ba-280d-45f1-8ade-994e6242ccc7") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:01.460866 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:01.460831    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fc43c39-ea1c-47da-b658-7834776917a8-metrics-certs podName:7fc43c39-ea1c-47da-b658-7834776917a8 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:02.46082384 +0000 UTC m=+3.298063866 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7fc43c39-ea1c-47da-b658-7834776917a8-metrics-certs") pod "network-metrics-daemon-7rzx4" (UID: "7fc43c39-ea1c-47da-b658-7834776917a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:01.482120 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:01.482096    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 21:19:01.500293 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:19:01.500269    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188c6c1c_be85_48e2_9cbb_9ab9722bb2b4.slice/crio-8178e11ce27d4e550681c16a2ed398c2ad36513d249c044bd3d92fcf5e6854e7 WatchSource:0}: Error finding container 8178e11ce27d4e550681c16a2ed398c2ad36513d249c044bd3d92fcf5e6854e7: Status 404 returned error can't find the container with id 8178e11ce27d4e550681c16a2ed398c2ad36513d249c044bd3d92fcf5e6854e7
Jul 02 21:19:01.542576 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:19:01.542539    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53db7e45_3dab_4292_930c_8628a2f51800.slice/crio-17289d4a247c5b0e804ea5e42ee768b7fe1a902f26b9ca707466b0e7d5009ef1 WatchSource:0}: Error finding container 17289d4a247c5b0e804ea5e42ee768b7fe1a902f26b9ca707466b0e7d5009ef1: Status 404 returned error can't find the container with id 17289d4a247c5b0e804ea5e42ee768b7fe1a902f26b9ca707466b0e7d5009ef1
Jul 02 21:19:01.544067 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:19:01.544036    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ea2cc9_ac41_4474_99dd_1ce8c95d8b87.slice/crio-84c32993d508cccbc1425be1b983d4dbd196518e0050c8d45b8ddb6fa66afcea WatchSource:0}: Error finding container 84c32993d508cccbc1425be1b983d4dbd196518e0050c8d45b8ddb6fa66afcea: Status 404 returned error can't find the container with id 84c32993d508cccbc1425be1b983d4dbd196518e0050c8d45b8ddb6fa66afcea
Jul 02 21:19:01.556303 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:19:01.556272    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a417763_5387_48be_9a81_3055e4a86b7e.slice/crio-21f98f3d15515cadf2e5089f12d3b3435fef6afa8d37cf5fc8981c970acc49cc WatchSource:0}: Error finding container 21f98f3d15515cadf2e5089f12d3b3435fef6afa8d37cf5fc8981c970acc49cc: Status 404 returned error can't find the container with id 21f98f3d15515cadf2e5089f12d3b3435fef6afa8d37cf5fc8981c970acc49cc
Jul 02 21:19:01.560914 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:01.560889    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slw7v\" (UniqueName: \"kubernetes.io/projected/238eb474-294d-49e5-8086-6b9cf9711ac6-kube-api-access-slw7v\") pod \"network-check-target-65mqk\" (UID: \"238eb474-294d-49e5-8086-6b9cf9711ac6\") " pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:01.561047 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:01.561032    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:19:01.561125 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:01.561057    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:19:01.561125 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:01.561070    2564 projected.go:196] Error preparing data for projected volume kube-api-access-slw7v for pod openshift-network-diagnostics/network-check-target-65mqk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:01.561232 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:01.561133    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/238eb474-294d-49e5-8086-6b9cf9711ac6-kube-api-access-slw7v podName:238eb474-294d-49e5-8086-6b9cf9711ac6 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:02.561111026 +0000 UTC m=+3.398351073 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-slw7v" (UniqueName: "kubernetes.io/projected/238eb474-294d-49e5-8086-6b9cf9711ac6-kube-api-access-slw7v") pod "network-check-target-65mqk" (UID: "238eb474-294d-49e5-8086-6b9cf9711ac6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:01.589419 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:19:01.589399    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb988618a_9a30_40af_964a_6c68d01dacfb.slice/crio-c3292c37daf8dc438d772acaa357deb4b7e0fbc55939defe95ff8cf2da04661e WatchSource:0}: Error finding container c3292c37daf8dc438d772acaa357deb4b7e0fbc55939defe95ff8cf2da04661e: Status 404 returned error can't find the container with id c3292c37daf8dc438d772acaa357deb4b7e0fbc55939defe95ff8cf2da04661e
Jul 02 21:19:01.620482 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:19:01.620456    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd2b06f_76c0_4b1c_af0d_03f5d780e9dd.slice/crio-fc683f1bb72e721b63b34a3267c534c70a121174d5c114294d32f3b39ce4ee4a WatchSource:0}: Error finding container fc683f1bb72e721b63b34a3267c534c70a121174d5c114294d32f3b39ce4ee4a: Status 404 returned error can't find the container with id fc683f1bb72e721b63b34a3267c534c70a121174d5c114294d32f3b39ce4ee4a
Jul 02 21:19:01.650272 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:19:01.650248    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f5fcb9_715f_4dc1_915b_ae229225e5c2.slice/crio-f85415d6e284f65e5865b687206e20348487c97c519df1853868b50df488b46b WatchSource:0}: Error finding container f85415d6e284f65e5865b687206e20348487c97c519df1853868b50df488b46b: Status 404 returned error can't find the container with id f85415d6e284f65e5865b687206e20348487c97c519df1853868b50df488b46b
Jul 02 21:19:01.654001 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:19:01.653976    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426d3178_3121_44cc_b885_d8b7a3b902a9.slice/crio-4deed4cc0ff8661b51a84cf50ee0263b2436f4648c5ea03e505781593d85435f WatchSource:0}: Error finding container 4deed4cc0ff8661b51a84cf50ee0263b2436f4648c5ea03e505781593d85435f: Status 404 returned error can't find the container with id 4deed4cc0ff8661b51a84cf50ee0263b2436f4648c5ea03e505781593d85435f
Jul 02 21:19:01.660274 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:19:01.660252    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde79c9aa_e167_492c_af48_e818dafe9d6a.slice/crio-9f6280eb87e5b4c4637a33ec52c566af915a818ab403786779ca58da991ef05a WatchSource:0}: Error finding container 9f6280eb87e5b4c4637a33ec52c566af915a818ab403786779ca58da991ef05a: Status 404 returned error can't find the container with id 9f6280eb87e5b4c4637a33ec52c566af915a818ab403786779ca58da991ef05a
Jul 02 21:19:01.896832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:01.896776    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 21:14:01 +0000 UTC" deadline="2028-01-17 04:48:17.11617327 +0000 UTC"
Jul 02 21:19:01.896832 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:01.896822    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13519h29m15.219355931s"
Jul 02 21:19:02.069485 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:02.068852    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:02.069485 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:02.069007    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:02.075483 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:02.075373    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc" event={"ID":"0a417763-5387-48be-9a81-3055e4a86b7e","Type":"ContainerStarted","Data":"21f98f3d15515cadf2e5089f12d3b3435fef6afa8d37cf5fc8981c970acc49cc"}
Jul 02 21:19:02.091095 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:02.090965    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dhs4b" event={"ID":"39ea2cc9-ac41-4474-99dd-1ce8c95d8b87","Type":"ContainerStarted","Data":"84c32993d508cccbc1425be1b983d4dbd196518e0050c8d45b8ddb6fa66afcea"}
Jul 02 21:19:02.096092 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:02.095959    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7" event={"ID":"53db7e45-3dab-4292-930c-8628a2f51800","Type":"ContainerStarted","Data":"17289d4a247c5b0e804ea5e42ee768b7fe1a902f26b9ca707466b0e7d5009ef1"}
Jul 02 21:19:02.098872 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:02.098792    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ctb57" event={"ID":"426d3178-3121-44cc-b885-d8b7a3b902a9","Type":"ContainerStarted","Data":"4deed4cc0ff8661b51a84cf50ee0263b2436f4648c5ea03e505781593d85435f"}
Jul 02 21:19:02.109630 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:02.109579    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6k66x" event={"ID":"188c6c1c-be85-48e2-9cbb-9ab9722bb2b4","Type":"ContainerStarted","Data":"8178e11ce27d4e550681c16a2ed398c2ad36513d249c044bd3d92fcf5e6854e7"}
Jul 02 21:19:02.122923 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:02.122806    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-230.ec2.internal" event={"ID":"005b67a3b8a4751eb2ba150e034470d6","Type":"ContainerStarted","Data":"f774caa1aff16400a7917043324cba6c05a5d4be6e2938372ccafecdf62460ce"}
Jul 02 21:19:02.126959 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:02.126923    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5rn2k" event={"ID":"de79c9aa-e167-492c-af48-e818dafe9d6a","Type":"ContainerStarted","Data":"9f6280eb87e5b4c4637a33ec52c566af915a818ab403786779ca58da991ef05a"}
Jul 02 21:19:02.131382 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:02.131307    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tm8gq" event={"ID":"b0f5fcb9-715f-4dc1-915b-ae229225e5c2","Type":"ContainerStarted","Data":"f85415d6e284f65e5865b687206e20348487c97c519df1853868b50df488b46b"}
Jul 02 21:19:02.137615 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:02.137587    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cg58w" event={"ID":"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd","Type":"ContainerStarted","Data":"fc683f1bb72e721b63b34a3267c534c70a121174d5c114294d32f3b39ce4ee4a"}
Jul 02 21:19:02.142870 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:02.142845    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal"
Jul 02 21:19:02.143157 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:02.143138    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal"
Jul 02 21:19:02.143277 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:02.143259    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zd5dg" event={"ID":"b988618a-9a30-40af-964a-6c68d01dacfb","Type":"ContainerStarted","Data":"c3292c37daf8dc438d772acaa357deb4b7e0fbc55939defe95ff8cf2da04661e"}
Jul 02 21:19:02.179355 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:02.179327    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 21:19:02.230737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:02.230709    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 21:19:02.469289 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:02.469205    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fc43c39-ea1c-47da-b658-7834776917a8-metrics-certs\") pod \"network-metrics-daemon-7rzx4\" (UID: \"7fc43c39-ea1c-47da-b658-7834776917a8\") " pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:02.469289 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:02.469255    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/453c93ba-280d-45f1-8ade-994e6242ccc7-original-pull-secret\") pod \"global-pull-secret-syncer-f8nf7\" (UID: \"453c93ba-280d-45f1-8ade-994e6242ccc7\") " pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:02.469522 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:02.469422    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:02.469522 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:02.469486    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/453c93ba-280d-45f1-8ade-994e6242ccc7-original-pull-secret podName:453c93ba-280d-45f1-8ade-994e6242ccc7 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:04.469465676 +0000 UTC m=+5.306705703 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/453c93ba-280d-45f1-8ade-994e6242ccc7-original-pull-secret") pod "global-pull-secret-syncer-f8nf7" (UID: "453c93ba-280d-45f1-8ade-994e6242ccc7") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:02.469645 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:02.469559    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:02.469645 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:02.469596    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fc43c39-ea1c-47da-b658-7834776917a8-metrics-certs podName:7fc43c39-ea1c-47da-b658-7834776917a8 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:04.469582247 +0000 UTC m=+5.306822280 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7fc43c39-ea1c-47da-b658-7834776917a8-metrics-certs") pod "network-metrics-daemon-7rzx4" (UID: "7fc43c39-ea1c-47da-b658-7834776917a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:02.569930 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:02.569894    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slw7v\" (UniqueName: \"kubernetes.io/projected/238eb474-294d-49e5-8086-6b9cf9711ac6-kube-api-access-slw7v\") pod \"network-check-target-65mqk\" (UID: \"238eb474-294d-49e5-8086-6b9cf9711ac6\") " pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:02.570104 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:02.570079    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:19:02.570104 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:02.570100    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:19:02.570247 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:02.570114    2564 projected.go:196] Error preparing data for projected volume kube-api-access-slw7v for pod openshift-network-diagnostics/network-check-target-65mqk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:02.570247 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:02.570196    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/238eb474-294d-49e5-8086-6b9cf9711ac6-kube-api-access-slw7v podName:238eb474-294d-49e5-8086-6b9cf9711ac6 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:04.570175948 +0000 UTC m=+5.407415985 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-slw7v" (UniqueName: "kubernetes.io/projected/238eb474-294d-49e5-8086-6b9cf9711ac6-kube-api-access-slw7v") pod "network-check-target-65mqk" (UID: "238eb474-294d-49e5-8086-6b9cf9711ac6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:02.897753 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:02.897686    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 21:14:01 +0000 UTC" deadline="2028-03-01 05:48:31.166293909 +0000 UTC"
Jul 02 21:19:02.897753 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:02.897748    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14576h29m28.268549695s"
Jul 02 21:19:03.066627 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:03.066588    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:03.066821 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:03.066729    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65mqk" podUID="238eb474-294d-49e5-8086-6b9cf9711ac6"
Jul 02 21:19:03.067214 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:03.067183    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:03.067321 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:03.067306    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8nf7" podUID="453c93ba-280d-45f1-8ade-994e6242ccc7"
Jul 02 21:19:03.159899 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:03.159737    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal" event={"ID":"9a5d467eae83b5ac2b132e3f387ffbc3","Type":"ContainerStarted","Data":"f9aae14c4f64488b2e09c2c0c8962127ccf0805cf7ee13873fc19bf69d541e27"}
Jul 02 21:19:04.075812 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:04.075731    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:04.076538 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:04.075870    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:04.486142 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:04.486106    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fc43c39-ea1c-47da-b658-7834776917a8-metrics-certs\") pod \"network-metrics-daemon-7rzx4\" (UID: \"7fc43c39-ea1c-47da-b658-7834776917a8\") " pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:04.486349 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:04.486178    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/453c93ba-280d-45f1-8ade-994e6242ccc7-original-pull-secret\") pod \"global-pull-secret-syncer-f8nf7\" (UID: \"453c93ba-280d-45f1-8ade-994e6242ccc7\") " pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:04.486349 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:04.486333    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:04.486479 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:04.486408    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/453c93ba-280d-45f1-8ade-994e6242ccc7-original-pull-secret podName:453c93ba-280d-45f1-8ade-994e6242ccc7 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:08.486387267 +0000 UTC m=+9.323627309 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/453c93ba-280d-45f1-8ade-994e6242ccc7-original-pull-secret") pod "global-pull-secret-syncer-f8nf7" (UID: "453c93ba-280d-45f1-8ade-994e6242ccc7") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:04.486550 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:04.486483    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:04.486550 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:04.486524    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fc43c39-ea1c-47da-b658-7834776917a8-metrics-certs podName:7fc43c39-ea1c-47da-b658-7834776917a8 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:08.486510246 +0000 UTC m=+9.323750279 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7fc43c39-ea1c-47da-b658-7834776917a8-metrics-certs") pod "network-metrics-daemon-7rzx4" (UID: "7fc43c39-ea1c-47da-b658-7834776917a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:04.587923 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:04.587295    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slw7v\" (UniqueName: \"kubernetes.io/projected/238eb474-294d-49e5-8086-6b9cf9711ac6-kube-api-access-slw7v\") pod \"network-check-target-65mqk\" (UID: \"238eb474-294d-49e5-8086-6b9cf9711ac6\") " pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:04.587923 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:04.587455    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:19:04.587923 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:04.587476    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:19:04.587923 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:04.587489    2564 projected.go:196] Error preparing data for projected volume kube-api-access-slw7v for pod openshift-network-diagnostics/network-check-target-65mqk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:04.587923 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:04.587554    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/238eb474-294d-49e5-8086-6b9cf9711ac6-kube-api-access-slw7v podName:238eb474-294d-49e5-8086-6b9cf9711ac6 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:08.587534584 +0000 UTC m=+9.424774612 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-slw7v" (UniqueName: "kubernetes.io/projected/238eb474-294d-49e5-8086-6b9cf9711ac6-kube-api-access-slw7v") pod "network-check-target-65mqk" (UID: "238eb474-294d-49e5-8086-6b9cf9711ac6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:05.066830 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:05.066790    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:05.067041 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:05.066938    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65mqk" podUID="238eb474-294d-49e5-8086-6b9cf9711ac6"
Jul 02 21:19:05.067370 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:05.067347    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:05.067482 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:05.067458    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8nf7" podUID="453c93ba-280d-45f1-8ade-994e6242ccc7"
Jul 02 21:19:06.066936 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:06.066876    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:06.067456 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:06.067021    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:07.066473 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:07.066434    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:07.066672 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:07.066569    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65mqk" podUID="238eb474-294d-49e5-8086-6b9cf9711ac6"
Jul 02 21:19:07.067148 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:07.066992    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:07.067578 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:07.067311    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8nf7" podUID="453c93ba-280d-45f1-8ade-994e6242ccc7"
Jul 02 21:19:08.066225 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:08.066186    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:08.066400 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:08.066352    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:08.524482 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:08.524427    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/453c93ba-280d-45f1-8ade-994e6242ccc7-original-pull-secret\") pod \"global-pull-secret-syncer-f8nf7\" (UID: \"453c93ba-280d-45f1-8ade-994e6242ccc7\") " pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:08.524989 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:08.524534    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fc43c39-ea1c-47da-b658-7834776917a8-metrics-certs\") pod \"network-metrics-daemon-7rzx4\" (UID: \"7fc43c39-ea1c-47da-b658-7834776917a8\") " pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:08.524989 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:08.524587    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:08.524989 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:08.524619    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:08.524989 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:08.524663    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/453c93ba-280d-45f1-8ade-994e6242ccc7-original-pull-secret podName:453c93ba-280d-45f1-8ade-994e6242ccc7 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:16.524641459 +0000 UTC m=+17.361881498 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/453c93ba-280d-45f1-8ade-994e6242ccc7-original-pull-secret") pod "global-pull-secret-syncer-f8nf7" (UID: "453c93ba-280d-45f1-8ade-994e6242ccc7") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:08.524989 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:08.524685    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fc43c39-ea1c-47da-b658-7834776917a8-metrics-certs podName:7fc43c39-ea1c-47da-b658-7834776917a8 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:16.524673245 +0000 UTC m=+17.361913286 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7fc43c39-ea1c-47da-b658-7834776917a8-metrics-certs") pod "network-metrics-daemon-7rzx4" (UID: "7fc43c39-ea1c-47da-b658-7834776917a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:08.625264 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:08.625221    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slw7v\" (UniqueName: \"kubernetes.io/projected/238eb474-294d-49e5-8086-6b9cf9711ac6-kube-api-access-slw7v\") pod \"network-check-target-65mqk\" (UID: \"238eb474-294d-49e5-8086-6b9cf9711ac6\") " pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:08.625455 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:08.625414    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:19:08.625455 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:08.625439    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:19:08.625455 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:08.625449    2564 projected.go:196] Error preparing data for projected volume kube-api-access-slw7v for pod openshift-network-diagnostics/network-check-target-65mqk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:08.625640 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:08.625509    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/238eb474-294d-49e5-8086-6b9cf9711ac6-kube-api-access-slw7v podName:238eb474-294d-49e5-8086-6b9cf9711ac6 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:16.625490782 +0000 UTC m=+17.462730813 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-slw7v" (UniqueName: "kubernetes.io/projected/238eb474-294d-49e5-8086-6b9cf9711ac6-kube-api-access-slw7v") pod "network-check-target-65mqk" (UID: "238eb474-294d-49e5-8086-6b9cf9711ac6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:09.067290 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:09.066688    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:09.067290 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:09.066861    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65mqk" podUID="238eb474-294d-49e5-8086-6b9cf9711ac6"
Jul 02 21:19:09.067569 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:09.067316    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:09.067569 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:09.067470    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8nf7" podUID="453c93ba-280d-45f1-8ade-994e6242ccc7"
Jul 02 21:19:10.067365 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:10.066823    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:10.067365 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:10.066977    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:11.066503 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:11.066467    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:11.066688 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:11.066601    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65mqk" podUID="238eb474-294d-49e5-8086-6b9cf9711ac6"
Jul 02 21:19:11.066764 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:11.066725    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:11.066846 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:11.066817    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8nf7" podUID="453c93ba-280d-45f1-8ade-994e6242ccc7"
Jul 02 21:19:12.066578 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:12.066531    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:12.066993 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:12.066690    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:13.066422 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:13.066384    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:13.066588 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:13.066493    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65mqk" podUID="238eb474-294d-49e5-8086-6b9cf9711ac6"
Jul 02 21:19:13.066588 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:13.066547    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:13.066930 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:13.066654    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8nf7" podUID="453c93ba-280d-45f1-8ade-994e6242ccc7"
Jul 02 21:19:14.066574 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:14.066493    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:14.066724 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:14.066642    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:15.065823 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:15.065790    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:15.066008 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:15.065789    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:15.066008 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:15.065943    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65mqk" podUID="238eb474-294d-49e5-8086-6b9cf9711ac6"
Jul 02 21:19:15.066008 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:15.065996    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8nf7" podUID="453c93ba-280d-45f1-8ade-994e6242ccc7"
Jul 02 21:19:16.067006 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:16.066968    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:16.067479 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:16.067158    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:16.584398 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:16.584362    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fc43c39-ea1c-47da-b658-7834776917a8-metrics-certs\") pod \"network-metrics-daemon-7rzx4\" (UID: \"7fc43c39-ea1c-47da-b658-7834776917a8\") " pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:16.584591 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:16.584408    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/453c93ba-280d-45f1-8ade-994e6242ccc7-original-pull-secret\") pod \"global-pull-secret-syncer-f8nf7\" (UID: \"453c93ba-280d-45f1-8ade-994e6242ccc7\") " pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:16.584591 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:16.584529    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:16.584709 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:16.584611    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fc43c39-ea1c-47da-b658-7834776917a8-metrics-certs podName:7fc43c39-ea1c-47da-b658-7834776917a8 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:32.584590332 +0000 UTC m=+33.421830368 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7fc43c39-ea1c-47da-b658-7834776917a8-metrics-certs") pod "network-metrics-daemon-7rzx4" (UID: "7fc43c39-ea1c-47da-b658-7834776917a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:16.584709 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:16.584536    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:16.584709 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:16.584703    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/453c93ba-280d-45f1-8ade-994e6242ccc7-original-pull-secret podName:453c93ba-280d-45f1-8ade-994e6242ccc7 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:32.584683313 +0000 UTC m=+33.421923342 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/453c93ba-280d-45f1-8ade-994e6242ccc7-original-pull-secret") pod "global-pull-secret-syncer-f8nf7" (UID: "453c93ba-280d-45f1-8ade-994e6242ccc7") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:16.685693 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:16.685656    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slw7v\" (UniqueName: \"kubernetes.io/projected/238eb474-294d-49e5-8086-6b9cf9711ac6-kube-api-access-slw7v\") pod \"network-check-target-65mqk\" (UID: \"238eb474-294d-49e5-8086-6b9cf9711ac6\") " pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:16.685864 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:16.685836    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:19:16.685864 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:16.685863    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:19:16.685972 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:16.685875    2564 projected.go:196] Error preparing data for projected volume kube-api-access-slw7v for pod openshift-network-diagnostics/network-check-target-65mqk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:16.685972 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:16.685939    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/238eb474-294d-49e5-8086-6b9cf9711ac6-kube-api-access-slw7v podName:238eb474-294d-49e5-8086-6b9cf9711ac6 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:32.685920425 +0000 UTC m=+33.523160452 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-slw7v" (UniqueName: "kubernetes.io/projected/238eb474-294d-49e5-8086-6b9cf9711ac6-kube-api-access-slw7v") pod "network-check-target-65mqk" (UID: "238eb474-294d-49e5-8086-6b9cf9711ac6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:17.066028 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:17.065993    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:17.066228 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:17.066033    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:17.066228 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:17.066132    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65mqk" podUID="238eb474-294d-49e5-8086-6b9cf9711ac6"
Jul 02 21:19:17.066337 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:17.066242    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8nf7" podUID="453c93ba-280d-45f1-8ade-994e6242ccc7"
Jul 02 21:19:18.066888 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:18.066855    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:18.067294 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:18.066975    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:19.065804 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:19.065772    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:19.065990 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:19.065812    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:19.065990 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:19.065896    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8nf7" podUID="453c93ba-280d-45f1-8ade-994e6242ccc7"
Jul 02 21:19:19.066146 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:19.066062    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65mqk" podUID="238eb474-294d-49e5-8086-6b9cf9711ac6"
Jul 02 21:19:20.067279 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:20.067228    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:20.067739 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:20.067358    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:21.066531 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:21.066491    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:21.066709 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:21.066491    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:21.066709 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:21.066607    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65mqk" podUID="238eb474-294d-49e5-8086-6b9cf9711ac6"
Jul 02 21:19:21.066803 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:21.066737    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8nf7" podUID="453c93ba-280d-45f1-8ade-994e6242ccc7"
Jul 02 21:19:22.066158 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:22.066107    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:22.066500 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:22.066273    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:23.066516 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:23.066288    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:23.067172 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:23.066621    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65mqk" podUID="238eb474-294d-49e5-8086-6b9cf9711ac6"
Jul 02 21:19:23.067172 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:23.066371    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:23.067172 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:23.066721    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8nf7" podUID="453c93ba-280d-45f1-8ade-994e6242ccc7"
Jul 02 21:19:23.201611 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:23.201535    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-230.ec2.internal" event={"ID":"005b67a3b8a4751eb2ba150e034470d6","Type":"ContainerStarted","Data":"b42a672b75aaacbe45639d0d8b9c8ce7410aa7dfc7ea1bd3d9837776fdd3cd63"}
Jul 02 21:19:23.203012 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:23.202984    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5rn2k" event={"ID":"de79c9aa-e167-492c-af48-e818dafe9d6a","Type":"ContainerStarted","Data":"4d5bcec8c674197f8e7270ee9c3066711aa0ae860e3b42939e050f7effe98e01"}
Jul 02 21:19:23.204383 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:23.204356    2564 generic.go:356] "Generic (PLEG): container finished" podID="b0f5fcb9-715f-4dc1-915b-ae229225e5c2" containerID="57b35dfc14a41367c354c6cbbf2498759b708c45e433b981956975cae0e3cf69" exitCode=0
Jul 02 21:19:23.204463 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:23.204437    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tm8gq" event={"ID":"b0f5fcb9-715f-4dc1-915b-ae229225e5c2","Type":"ContainerDied","Data":"57b35dfc14a41367c354c6cbbf2498759b708c45e433b981956975cae0e3cf69"}
Jul 02 21:19:23.205922 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:23.205900    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cg58w" event={"ID":"dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd","Type":"ContainerStarted","Data":"3fb6a78d8ed1fec450bbf05524015392ccb0d7ace2084a94746bd688342c5293"}
Jul 02 21:19:23.207210 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:23.207192    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zd5dg" event={"ID":"b988618a-9a30-40af-964a-6c68d01dacfb","Type":"ContainerStarted","Data":"24428af9351f37b9d077cfa69d1bf2582ab0ceed1fee21388217fc9220113496"}
Jul 02 21:19:23.208705 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:23.208685    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc" event={"ID":"0a417763-5387-48be-9a81-3055e4a86b7e","Type":"ContainerStarted","Data":"decd47a53a3213f995444ccadda78e9f1fe6fdb399207c21667fb447a1f8dac8"}
Jul 02 21:19:23.211445 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:23.211425    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7" event={"ID":"53db7e45-3dab-4292-930c-8628a2f51800","Type":"ContainerStarted","Data":"cb92ce2e344fb2168b5e7dc0890be412b3c0c06dd9fe563091f24f572d725d40"}
Jul 02 21:19:23.211524 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:23.211495    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7" event={"ID":"53db7e45-3dab-4292-930c-8628a2f51800","Type":"ContainerStarted","Data":"86d70200b3d86559cf0519e85142f87ccb2434b8a85f39d8e15a591f342c3445"}
Jul 02 21:19:23.211595 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:23.211551    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7" event={"ID":"53db7e45-3dab-4292-930c-8628a2f51800","Type":"ContainerStarted","Data":"b87db585c72d9fece218e702ab2b2c8e1bcee2437254f3972ab8d5128263e25c"}
Jul 02 21:19:23.211651 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:23.211601    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7" event={"ID":"53db7e45-3dab-4292-930c-8628a2f51800","Type":"ContainerStarted","Data":"fd6670660b944dd41a930f1fffd65f2b95737717e905504ee8164f119f6572a8"}
Jul 02 21:19:23.211704 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:23.211649    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7" event={"ID":"53db7e45-3dab-4292-930c-8628a2f51800","Type":"ContainerStarted","Data":"4ed55135e3a0d097b957d31a1d56a0527b9f84fa1203a3ff3178802d86da200b"}
Jul 02 21:19:23.211704 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:23.211698    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7" event={"ID":"53db7e45-3dab-4292-930c-8628a2f51800","Type":"ContainerStarted","Data":"6a08e123385a811219f1a5b3aa9d7911a2d180dfb5d295ba4678194d80bf19f1"}
Jul 02 21:19:23.212831 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:23.212812    2564 generic.go:356] "Generic (PLEG): container finished" podID="9a5d467eae83b5ac2b132e3f387ffbc3" containerID="a3baec37a27e4d4c4c1f8ed5afd335f9ae901c2191235fae8a35c47cc99ac228" exitCode=0
Jul 02 21:19:23.212916 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:23.212881    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal" event={"ID":"9a5d467eae83b5ac2b132e3f387ffbc3","Type":"ContainerDied","Data":"a3baec37a27e4d4c4c1f8ed5afd335f9ae901c2191235fae8a35c47cc99ac228"}
Jul 02 21:19:23.214159 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:23.214138    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ctb57" event={"ID":"426d3178-3121-44cc-b885-d8b7a3b902a9","Type":"ContainerStarted","Data":"8dc28db1162793d5012e42d63b9c4899bc68c1ed7a9374fb8759839f0cd2f0ac"}
Jul 02 21:19:23.215379 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:23.215359    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6k66x" event={"ID":"188c6c1c-be85-48e2-9cbb-9ab9722bb2b4","Type":"ContainerStarted","Data":"b7996dfa52cdb75156af29ce1a4cf8c45535ad86c0be33122a9b03e58cdc0b5e"}
Jul 02 21:19:23.235015 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:23.234977    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cg58w" podStartSLOduration=2.7589796460000002 podStartE2EDuration="23.234965708s" totalImagesPullingTime="20.475986062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:19:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:23.234893474 +0000 UTC m=+24.072133516" watchObservedRunningTime="2026-07-02 21:19:23.234965708 +0000 UTC m=+24.072205755"
Jul 02 21:19:23.235482 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:23.235458    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-230.ec2.internal" podStartSLOduration=22.235451997 podStartE2EDuration="22.235451997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:19:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:23.218548925 +0000 UTC m=+24.055788971" watchObservedRunningTime="2026-07-02 21:19:23.235451997 +0000 UTC m=+24.072692085"
Jul 02 21:19:23.280218 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:23.279554    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5rn2k" podStartSLOduration=2.840391015 podStartE2EDuration="23.279537126s" totalImagesPullingTime="20.439146111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:19:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:23.251360834 +0000 UTC m=+24.088600881" watchObservedRunningTime="2026-07-02 21:19:23.279537126 +0000 UTC m=+24.116777175"
Jul 02 21:19:23.301005 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:23.300964    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6k66x" podStartSLOduration=2.866146119 podStartE2EDuration="23.3009507s" totalImagesPullingTime="20.434804581s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:19:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:23.30061159 +0000 UTC m=+24.137851638" watchObservedRunningTime="2026-07-02 21:19:23.3009507 +0000 UTC m=+24.138190746"
Jul 02 21:19:23.318763 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:23.318726    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zd5dg" podStartSLOduration=2.958041968 podStartE2EDuration="23.318713714s" totalImagesPullingTime="20.360671746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:19:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:23.318504424 +0000 UTC m=+24.155744474" watchObservedRunningTime="2026-07-02 21:19:23.318713714 +0000 UTC m=+24.155953760"
Jul 02 21:19:23.334463 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:23.334418    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ctb57" podStartSLOduration=2.893609634 podStartE2EDuration="23.334401021s" totalImagesPullingTime="20.440791387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:19:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:23.334245015 +0000 UTC m=+24.171485062" watchObservedRunningTime="2026-07-02 21:19:23.334401021 +0000 UTC m=+24.171641071"
Jul 02 21:19:24.066709 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:24.066462    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:24.067175 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:24.066806    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:24.219269 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:24.219226    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dhs4b" event={"ID":"39ea2cc9-ac41-4474-99dd-1ce8c95d8b87","Type":"ContainerStarted","Data":"10047ec2e4d3522b3a3c63ccdb96d742c5cb7fc29ad72eb64fcbf6b8b5fa435f"}
Jul 02 21:19:24.268557 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:24.268406    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dhs4b" podStartSLOduration=3.865381426 podStartE2EDuration="24.268381421s" totalImagesPullingTime="20.402999995s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:19:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:24.24991736 +0000 UTC m=+25.087157411" watchObservedRunningTime="2026-07-02 21:19:24.268381421 +0000 UTC m=+25.105621468"
Jul 02 21:19:24.354372 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:24.354348    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 02 21:19:24.940522 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:24.940423    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T21:19:24.354367719Z","UUID":"96ddf957-487c-450c-b737-7e94aec7e91c","Handler":null,"Name":"","Endpoint":""}
Jul 02 21:19:24.943594 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:24.943499    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 02 21:19:24.943594 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:24.943540    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 02 21:19:25.064476 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:25.064452    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6k66x_188c6c1c-be85-48e2-9cbb-9ab9722bb2b4/dns-node-resolver/0.log"
Jul 02 21:19:25.066692 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:25.066666    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:25.066800 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:25.066694    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:25.067077 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:25.066789    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65mqk" podUID="238eb474-294d-49e5-8086-6b9cf9711ac6"
Jul 02 21:19:25.067077 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:25.066882    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8nf7" podUID="453c93ba-280d-45f1-8ade-994e6242ccc7"
Jul 02 21:19:25.225336 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:25.225251    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc" event={"ID":"0a417763-5387-48be-9a81-3055e4a86b7e","Type":"ContainerStarted","Data":"c6560e976ec31583294c35bb749b500c098a3929298cb0098a8f3f365b7542db"}
Jul 02 21:19:25.227316 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:25.227285    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal" event={"ID":"9a5d467eae83b5ac2b132e3f387ffbc3","Type":"ContainerStarted","Data":"94905f6246497494f9613bbcc3888185839fe660462a83655080cebd668f1883"}
Jul 02 21:19:25.267079 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:25.266247    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-230.ec2.internal" podStartSLOduration=24.26622632 podStartE2EDuration="24.26622632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:19:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:25.265532161 +0000 UTC m=+26.102772204" watchObservedRunningTime="2026-07-02 21:19:25.26622632 +0000 UTC m=+26.103466368"
Jul 02 21:19:26.031017 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:26.030986    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ctb57_426d3178-3121-44cc-b885-d8b7a3b902a9/node-ca/0.log"
Jul 02 21:19:26.066471 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:26.066440    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:26.066619 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:26.066581    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:26.232370 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:26.232336    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc" event={"ID":"0a417763-5387-48be-9a81-3055e4a86b7e","Type":"ContainerStarted","Data":"620c3c4cfc4e21357344c30bde32408ea066bb99d488d5d52816d5dc4e138d20"}
Jul 02 21:19:26.235848 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:26.235824    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7" event={"ID":"53db7e45-3dab-4292-930c-8628a2f51800","Type":"ContainerStarted","Data":"9c8a8445c5009d7f00de26675a62ea95115ac5cda92ac565176fe454b612ebd7"}
Jul 02 21:19:26.264198 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:26.264139    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjbzc" podStartSLOduration=3.457299733 podStartE2EDuration="26.264107566s" totalImagesPullingTime="22.806807833s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:19:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:26.263612226 +0000 UTC m=+27.100852273" watchObservedRunningTime="2026-07-02 21:19:26.264107566 +0000 UTC m=+27.101347614"
Jul 02 21:19:26.859558 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:26.859517    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zd5dg"
Jul 02 21:19:26.860180 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:26.860141    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zd5dg"
Jul 02 21:19:27.066825 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:27.066789    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:27.066995 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:27.066914    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65mqk" podUID="238eb474-294d-49e5-8086-6b9cf9711ac6"
Jul 02 21:19:27.066995 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:27.066974    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:27.067089 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:27.067072    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8nf7" podUID="453c93ba-280d-45f1-8ade-994e6242ccc7"
Jul 02 21:19:27.238914 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:27.238837    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zd5dg"
Jul 02 21:19:27.239423 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:27.239402    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zd5dg"
Jul 02 21:19:28.066331 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:28.066161    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:28.066498 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:28.066415    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:28.242598 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:28.242558    2564 generic.go:356] "Generic (PLEG): container finished" podID="b0f5fcb9-715f-4dc1-915b-ae229225e5c2" containerID="5dd1475bb03de88c8874a94992cfc8a6dd2aba0c9f00d771172e8f993365163c" exitCode=0
Jul 02 21:19:28.243331 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:28.242634    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tm8gq" event={"ID":"b0f5fcb9-715f-4dc1-915b-ae229225e5c2","Type":"ContainerDied","Data":"5dd1475bb03de88c8874a94992cfc8a6dd2aba0c9f00d771172e8f993365163c"}
Jul 02 21:19:28.246714 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:28.246688    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7" event={"ID":"53db7e45-3dab-4292-930c-8628a2f51800","Type":"ContainerStarted","Data":"4bdc01c8d56daf4da2790ec5d76c753505365f1c9b8317aab3e0fade20183a82"}
Jul 02 21:19:29.066239 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:29.066208    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:29.066476 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:29.066207    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:29.066476 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:29.066311    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8nf7" podUID="453c93ba-280d-45f1-8ade-994e6242ccc7"
Jul 02 21:19:29.066476 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:29.066406    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65mqk" podUID="238eb474-294d-49e5-8086-6b9cf9711ac6"
Jul 02 21:19:29.249339 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:29.249316    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:29.249751 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:29.249393    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:29.249751 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:29.249444    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:29.266038 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:29.266010    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:29.266198 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:29.266178    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:19:29.298664 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:29.298617    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7" podStartSLOduration=8.552807873 podStartE2EDuration="29.298604164s" totalImagesPullingTime="20.745796291s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:19:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:28.308537997 +0000 UTC m=+29.145778043" watchObservedRunningTime="2026-07-02 21:19:29.298604164 +0000 UTC m=+30.135844240"
Jul 02 21:19:30.067894 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:30.067860    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:30.068054 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:30.067982    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:30.252775 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:30.252740    2564 generic.go:356] "Generic (PLEG): container finished" podID="b0f5fcb9-715f-4dc1-915b-ae229225e5c2" containerID="0558378f32de3861feaeb90ff78b407991279146bf96ec84545bab007291d300" exitCode=0
Jul 02 21:19:30.253158 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:30.252806    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tm8gq" event={"ID":"b0f5fcb9-715f-4dc1-915b-ae229225e5c2","Type":"ContainerDied","Data":"0558378f32de3861feaeb90ff78b407991279146bf96ec84545bab007291d300"}
Jul 02 21:19:31.066518 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:31.066484    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:31.066678 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:31.066499    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:31.066678 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:31.066590    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65mqk" podUID="238eb474-294d-49e5-8086-6b9cf9711ac6"
Jul 02 21:19:31.066678 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:31.066649    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8nf7" podUID="453c93ba-280d-45f1-8ade-994e6242ccc7"
Jul 02 21:19:32.066376 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:32.066196    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:32.066752 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:32.066456    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:32.259368 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:32.259335    2564 generic.go:356] "Generic (PLEG): container finished" podID="b0f5fcb9-715f-4dc1-915b-ae229225e5c2" containerID="07acaed111f966420764649a9b787dc394445798ab604f73cc69331d658a9da0" exitCode=0
Jul 02 21:19:32.259523 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:32.259392    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tm8gq" event={"ID":"b0f5fcb9-715f-4dc1-915b-ae229225e5c2","Type":"ContainerDied","Data":"07acaed111f966420764649a9b787dc394445798ab604f73cc69331d658a9da0"}
Jul 02 21:19:32.602921 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:32.602891    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fc43c39-ea1c-47da-b658-7834776917a8-metrics-certs\") pod \"network-metrics-daemon-7rzx4\" (UID: \"7fc43c39-ea1c-47da-b658-7834776917a8\") " pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:32.602921 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:32.602927    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/453c93ba-280d-45f1-8ade-994e6242ccc7-original-pull-secret\") pod \"global-pull-secret-syncer-f8nf7\" (UID: \"453c93ba-280d-45f1-8ade-994e6242ccc7\") " pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:32.603132 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:32.603030    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:32.603132 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:32.603030    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:32.603132 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:32.603076    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/453c93ba-280d-45f1-8ade-994e6242ccc7-original-pull-secret podName:453c93ba-280d-45f1-8ade-994e6242ccc7 nodeName:}" failed. No retries permitted until 2026-07-02 21:20:04.603063715 +0000 UTC m=+65.440303739 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/453c93ba-280d-45f1-8ade-994e6242ccc7-original-pull-secret") pod "global-pull-secret-syncer-f8nf7" (UID: "453c93ba-280d-45f1-8ade-994e6242ccc7") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:32.603132 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:32.603089    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fc43c39-ea1c-47da-b658-7834776917a8-metrics-certs podName:7fc43c39-ea1c-47da-b658-7834776917a8 nodeName:}" failed. No retries permitted until 2026-07-02 21:20:04.603083094 +0000 UTC m=+65.440323118 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7fc43c39-ea1c-47da-b658-7834776917a8-metrics-certs") pod "network-metrics-daemon-7rzx4" (UID: "7fc43c39-ea1c-47da-b658-7834776917a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:32.703262 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:32.703237    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slw7v\" (UniqueName: \"kubernetes.io/projected/238eb474-294d-49e5-8086-6b9cf9711ac6-kube-api-access-slw7v\") pod \"network-check-target-65mqk\" (UID: \"238eb474-294d-49e5-8086-6b9cf9711ac6\") " pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:32.703416 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:32.703341    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:19:32.703416 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:32.703356    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:19:32.703416 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:32.703365    2564 projected.go:196] Error preparing data for projected volume kube-api-access-slw7v for pod openshift-network-diagnostics/network-check-target-65mqk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:32.703416 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:32.703408    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/238eb474-294d-49e5-8086-6b9cf9711ac6-kube-api-access-slw7v podName:238eb474-294d-49e5-8086-6b9cf9711ac6 nodeName:}" failed. No retries permitted until 2026-07-02 21:20:04.703395701 +0000 UTC m=+65.540635726 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-slw7v" (UniqueName: "kubernetes.io/projected/238eb474-294d-49e5-8086-6b9cf9711ac6-kube-api-access-slw7v") pod "network-check-target-65mqk" (UID: "238eb474-294d-49e5-8086-6b9cf9711ac6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:33.066549 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:33.066512    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:33.066549 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:33.066548    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:33.067010 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:33.066654    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8nf7" podUID="453c93ba-280d-45f1-8ade-994e6242ccc7"
Jul 02 21:19:33.067010 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:33.066799    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65mqk" podUID="238eb474-294d-49e5-8086-6b9cf9711ac6"
Jul 02 21:19:34.066154 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:34.066105    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:34.066320 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:34.066265    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:35.066269 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:35.066234    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:35.066683 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:35.066242    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:35.066683 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:35.066354    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8nf7" podUID="453c93ba-280d-45f1-8ade-994e6242ccc7"
Jul 02 21:19:35.066683 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:35.066430    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65mqk" podUID="238eb474-294d-49e5-8086-6b9cf9711ac6"
Jul 02 21:19:36.066254 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:36.066216    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:36.066427 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:36.066364    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:37.066451 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:37.066417    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:37.066824 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:37.066518    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8nf7" podUID="453c93ba-280d-45f1-8ade-994e6242ccc7"
Jul 02 21:19:37.066824 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:37.066575    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:37.066824 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:37.066666    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65mqk" podUID="238eb474-294d-49e5-8086-6b9cf9711ac6"
Jul 02 21:19:37.273304 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:37.273272    2564 generic.go:356] "Generic (PLEG): container finished" podID="b0f5fcb9-715f-4dc1-915b-ae229225e5c2" containerID="a4585dd490274b6e2574746efa50880cad11ec1fdc77c00ef0c18dfaaf5441ee" exitCode=0
Jul 02 21:19:37.273464 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:37.273316    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tm8gq" event={"ID":"b0f5fcb9-715f-4dc1-915b-ae229225e5c2","Type":"ContainerDied","Data":"a4585dd490274b6e2574746efa50880cad11ec1fdc77c00ef0c18dfaaf5441ee"}
Jul 02 21:19:38.065996 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:38.065960    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:38.066192 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:38.066074    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:38.278014 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:38.277979    2564 generic.go:356] "Generic (PLEG): container finished" podID="b0f5fcb9-715f-4dc1-915b-ae229225e5c2" containerID="0d4d75d6a94775f8d1eabceafcb7ace8a6deeb1305fb050ce3a31bd6e90e74ce" exitCode=0
Jul 02 21:19:38.278488 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:38.278040    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tm8gq" event={"ID":"b0f5fcb9-715f-4dc1-915b-ae229225e5c2","Type":"ContainerDied","Data":"0d4d75d6a94775f8d1eabceafcb7ace8a6deeb1305fb050ce3a31bd6e90e74ce"}
Jul 02 21:19:39.066546 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:39.066512    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:39.066743 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:39.066511    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:39.066743 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:39.066638    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8nf7" podUID="453c93ba-280d-45f1-8ade-994e6242ccc7"
Jul 02 21:19:39.066743 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:39.066676    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65mqk" podUID="238eb474-294d-49e5-8086-6b9cf9711ac6"
Jul 02 21:19:39.283173 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:39.283142    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tm8gq" event={"ID":"b0f5fcb9-715f-4dc1-915b-ae229225e5c2","Type":"ContainerStarted","Data":"e0a8592ad0a5ebb180cbbebd741d9b97419e55a2600e0cbde84378efa22e88c8"}
Jul 02 21:19:39.307345 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:39.307297    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tm8gq" podStartSLOduration=8.772479864 podStartE2EDuration="39.30728283s" totalImagesPullingTime="30.534802966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:19:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:39.30663914 +0000 UTC m=+40.143879188" watchObservedRunningTime="2026-07-02 21:19:39.30728283 +0000 UTC m=+40.144522877"
Jul 02 21:19:40.068037 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:40.067999    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:40.068194 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:40.068149    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:41.065957 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:41.065928    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:41.066357 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:41.065928    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:41.066357 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:41.066025    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8nf7" podUID="453c93ba-280d-45f1-8ade-994e6242ccc7"
Jul 02 21:19:41.066357 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:41.066091    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65mqk" podUID="238eb474-294d-49e5-8086-6b9cf9711ac6"
Jul 02 21:19:42.066676 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:42.066619    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:42.067040 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:42.066751    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:43.066052 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:43.066026    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:43.066208 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:43.066026    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:43.066208 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:43.066152    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8nf7" podUID="453c93ba-280d-45f1-8ade-994e6242ccc7"
Jul 02 21:19:43.066298 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:43.066207    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65mqk" podUID="238eb474-294d-49e5-8086-6b9cf9711ac6"
Jul 02 21:19:44.066406 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:44.066374    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:44.066758 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:44.066485    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:45.066142 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:45.066097    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:45.066331 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:45.066212    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65mqk" podUID="238eb474-294d-49e5-8086-6b9cf9711ac6"
Jul 02 21:19:45.066331 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:45.066274    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:45.066447 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:45.066391    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8nf7" podUID="453c93ba-280d-45f1-8ade-994e6242ccc7"
Jul 02 21:19:46.065939 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:46.065909    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:46.066240 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:46.066033    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:47.066525 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:47.066489    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:47.066525 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:47.066527    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:47.066925 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:47.066611    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65mqk" podUID="238eb474-294d-49e5-8086-6b9cf9711ac6"
Jul 02 21:19:47.066925 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:47.066728    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8nf7" podUID="453c93ba-280d-45f1-8ade-994e6242ccc7"
Jul 02 21:19:48.066095 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:48.066061    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:48.066283 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:48.066194    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:48.582774 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:48.582567    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f8nf7"]
Jul 02 21:19:48.583222 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:48.583155    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:48.583300 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:48.583275    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8nf7" podUID="453c93ba-280d-45f1-8ade-994e6242ccc7"
Jul 02 21:19:48.584238 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:48.584210    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7rzx4"]
Jul 02 21:19:48.584427 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:48.584306    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:48.584427 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:48.584396    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:48.585025 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:48.585001    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-65mqk"]
Jul 02 21:19:48.585126 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:48.585082    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:48.585205 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:48.585184    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65mqk" podUID="238eb474-294d-49e5-8086-6b9cf9711ac6"
Jul 02 21:19:50.067057 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:50.067026    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:50.067454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:50.067097    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:50.067454 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:50.067123    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:50.067454 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:50.067203    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65mqk" podUID="238eb474-294d-49e5-8086-6b9cf9711ac6"
Jul 02 21:19:50.067454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:50.067247    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:50.067454 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:50.067297    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8nf7" podUID="453c93ba-280d-45f1-8ade-994e6242ccc7"
Jul 02 21:19:52.066565 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:52.066531    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:52.067076 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:52.066531    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:52.067076 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:52.066643    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8nf7" podUID="453c93ba-280d-45f1-8ade-994e6242ccc7"
Jul 02 21:19:52.067076 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:52.066748    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rzx4" podUID="7fc43c39-ea1c-47da-b658-7834776917a8"
Jul 02 21:19:52.067076 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:52.066531    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:52.067076 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:19:52.066853    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65mqk" podUID="238eb474-294d-49e5-8086-6b9cf9711ac6"
Jul 02 21:19:52.903747 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:52.903715    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-230.ec2.internal" event="NodeReady"
Jul 02 21:19:52.903926 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:52.903862    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 02 21:19:52.958748 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:52.958675    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4scqp"]
Jul 02 21:19:52.973738 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:52.973716    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kjzfh"]
Jul 02 21:19:52.973857 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:52.973847    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4scqp"
Jul 02 21:19:52.979881 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:52.979843    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bftgm\""
Jul 02 21:19:52.979881 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:52.979876    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 02 21:19:52.980128 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:52.979926    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 02 21:19:52.982464 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:52.982440    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 02 21:19:52.995669 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:52.995643    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4scqp"]
Jul 02 21:19:52.995669 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:52.995670    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kjzfh"]
Jul 02 21:19:52.995849 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:52.995778    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kjzfh"
Jul 02 21:19:53.000230 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.000207    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 02 21:19:53.000230 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.000221    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 02 21:19:53.000397 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.000220    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 02 21:19:53.000397 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.000214    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t9dsl\""
Jul 02 21:19:53.000734 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.000714    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tcvc8"]
Jul 02 21:19:53.006088 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.006073    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 02 21:19:53.018007 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.017990    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tcvc8"]
Jul 02 21:19:53.018105 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.018072    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tcvc8"
Jul 02 21:19:53.025252 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.025235    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 02 21:19:53.025391 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.025376    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-69xqz\""
Jul 02 21:19:53.027480 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.027465    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 02 21:19:53.058479 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.058451    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ecf34a8b-a6e9-48cf-9097-74ac6ab7c362-data-volume\") pod \"insights-runtime-extractor-kjzfh\" (UID: \"ecf34a8b-a6e9-48cf-9097-74ac6ab7c362\") " pod="openshift-insights/insights-runtime-extractor-kjzfh"
Jul 02 21:19:53.058614 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.058482    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgdwv\" (UniqueName: \"kubernetes.io/projected/ecf34a8b-a6e9-48cf-9097-74ac6ab7c362-kube-api-access-sgdwv\") pod \"insights-runtime-extractor-kjzfh\" (UID: \"ecf34a8b-a6e9-48cf-9097-74ac6ab7c362\") " pod="openshift-insights/insights-runtime-extractor-kjzfh"
Jul 02 21:19:53.058614 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.058527    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ecf34a8b-a6e9-48cf-9097-74ac6ab7c362-crio-socket\") pod \"insights-runtime-extractor-kjzfh\" (UID: \"ecf34a8b-a6e9-48cf-9097-74ac6ab7c362\") " pod="openshift-insights/insights-runtime-extractor-kjzfh"
Jul 02 21:19:53.058614 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.058587    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e7ae876-0cf7-4171-a256-8205bbd810f7-metrics-tls\") pod \"dns-default-tcvc8\" (UID: \"4e7ae876-0cf7-4171-a256-8205bbd810f7\") " pod="openshift-dns/dns-default-tcvc8"
Jul 02 21:19:53.058741 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.058613    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ecf34a8b-a6e9-48cf-9097-74ac6ab7c362-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kjzfh\" (UID: \"ecf34a8b-a6e9-48cf-9097-74ac6ab7c362\") " pod="openshift-insights/insights-runtime-extractor-kjzfh"
Jul 02 21:19:53.058741 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.058632    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65655d15-640b-4caf-8715-750aeb8b1887-cert\") pod \"ingress-canary-4scqp\" (UID: \"65655d15-640b-4caf-8715-750aeb8b1887\") " pod="openshift-ingress-canary/ingress-canary-4scqp"
Jul 02 21:19:53.058741 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.058665    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e7ae876-0cf7-4171-a256-8205bbd810f7-tmp-dir\") pod \"dns-default-tcvc8\" (UID: \"4e7ae876-0cf7-4171-a256-8205bbd810f7\") " pod="openshift-dns/dns-default-tcvc8"
Jul 02 21:19:53.058741 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.058691    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbb46\" (UniqueName: \"kubernetes.io/projected/4e7ae876-0cf7-4171-a256-8205bbd810f7-kube-api-access-vbb46\") pod \"dns-default-tcvc8\" (UID: \"4e7ae876-0cf7-4171-a256-8205bbd810f7\") " pod="openshift-dns/dns-default-tcvc8"
Jul 02 21:19:53.058741 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.058731    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7ae876-0cf7-4171-a256-8205bbd810f7-config-volume\") pod \"dns-default-tcvc8\" (UID: \"4e7ae876-0cf7-4171-a256-8205bbd810f7\") " pod="openshift-dns/dns-default-tcvc8"
Jul 02 21:19:53.058919 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.058749    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ecf34a8b-a6e9-48cf-9097-74ac6ab7c362-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kjzfh\" (UID: \"ecf34a8b-a6e9-48cf-9097-74ac6ab7c362\") " pod="openshift-insights/insights-runtime-extractor-kjzfh"
Jul 02 21:19:53.058919 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.058764    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxsjm\" (UniqueName: \"kubernetes.io/projected/65655d15-640b-4caf-8715-750aeb8b1887-kube-api-access-zxsjm\") pod \"ingress-canary-4scqp\" (UID: \"65655d15-640b-4caf-8715-750aeb8b1887\") " pod="openshift-ingress-canary/ingress-canary-4scqp"
Jul 02 21:19:53.159329 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.159299    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ecf34a8b-a6e9-48cf-9097-74ac6ab7c362-data-volume\") pod \"insights-runtime-extractor-kjzfh\" (UID: \"ecf34a8b-a6e9-48cf-9097-74ac6ab7c362\") " pod="openshift-insights/insights-runtime-extractor-kjzfh"
Jul 02 21:19:53.159329 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.159331    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgdwv\" (UniqueName: \"kubernetes.io/projected/ecf34a8b-a6e9-48cf-9097-74ac6ab7c362-kube-api-access-sgdwv\") pod \"insights-runtime-extractor-kjzfh\" (UID: \"ecf34a8b-a6e9-48cf-9097-74ac6ab7c362\") " pod="openshift-insights/insights-runtime-extractor-kjzfh"
Jul 02 21:19:53.159787 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.159354    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ecf34a8b-a6e9-48cf-9097-74ac6ab7c362-crio-socket\") pod \"insights-runtime-extractor-kjzfh\" (UID: \"ecf34a8b-a6e9-48cf-9097-74ac6ab7c362\") " pod="openshift-insights/insights-runtime-extractor-kjzfh"
Jul 02 21:19:53.159787 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.159431    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ecf34a8b-a6e9-48cf-9097-74ac6ab7c362-crio-socket\") pod \"insights-runtime-extractor-kjzfh\" (UID: \"ecf34a8b-a6e9-48cf-9097-74ac6ab7c362\") " pod="openshift-insights/insights-runtime-extractor-kjzfh"
Jul 02 21:19:53.159787 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.159496    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e7ae876-0cf7-4171-a256-8205bbd810f7-metrics-tls\") pod \"dns-default-tcvc8\" (UID: \"4e7ae876-0cf7-4171-a256-8205bbd810f7\") " pod="openshift-dns/dns-default-tcvc8"
Jul 02 21:19:53.159787 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.159523    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ecf34a8b-a6e9-48cf-9097-74ac6ab7c362-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kjzfh\" (UID: \"ecf34a8b-a6e9-48cf-9097-74ac6ab7c362\") " pod="openshift-insights/insights-runtime-extractor-kjzfh"
Jul 02 21:19:53.159787 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.159542    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65655d15-640b-4caf-8715-750aeb8b1887-cert\") pod \"ingress-canary-4scqp\" (UID: \"65655d15-640b-4caf-8715-750aeb8b1887\") " pod="openshift-ingress-canary/ingress-canary-4scqp"
Jul 02 21:19:53.159787 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.159562    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e7ae876-0cf7-4171-a256-8205bbd810f7-tmp-dir\") pod \"dns-default-tcvc8\" (UID: \"4e7ae876-0cf7-4171-a256-8205bbd810f7\") " pod="openshift-dns/dns-default-tcvc8"
Jul 02 21:19:53.159787 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.159589    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbb46\" (UniqueName: \"kubernetes.io/projected/4e7ae876-0cf7-4171-a256-8205bbd810f7-kube-api-access-vbb46\") pod \"dns-default-tcvc8\" (UID: \"4e7ae876-0cf7-4171-a256-8205bbd810f7\") " pod="openshift-dns/dns-default-tcvc8"
Jul 02 21:19:53.159787 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.159617    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7ae876-0cf7-4171-a256-8205bbd810f7-config-volume\") pod \"dns-default-tcvc8\" (UID: \"4e7ae876-0cf7-4171-a256-8205bbd810f7\") " pod="openshift-dns/dns-default-tcvc8"
Jul 02 21:19:53.159787 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.159635    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ecf34a8b-a6e9-48cf-9097-74ac6ab7c362-data-volume\") pod \"insights-runtime-extractor-kjzfh\" (UID: \"ecf34a8b-a6e9-48cf-9097-74ac6ab7c362\") " pod="openshift-insights/insights-runtime-extractor-kjzfh"
Jul 02 21:19:53.159787 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.159657    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ecf34a8b-a6e9-48cf-9097-74ac6ab7c362-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kjzfh\" (UID: \"ecf34a8b-a6e9-48cf-9097-74ac6ab7c362\") " pod="openshift-insights/insights-runtime-extractor-kjzfh"
Jul 02 21:19:53.160306 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.159681    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxsjm\" (UniqueName: \"kubernetes.io/projected/65655d15-640b-4caf-8715-750aeb8b1887-kube-api-access-zxsjm\") pod \"ingress-canary-4scqp\" (UID: \"65655d15-640b-4caf-8715-750aeb8b1887\") " pod="openshift-ingress-canary/ingress-canary-4scqp"
Jul 02 21:19:53.160306 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.159950    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e7ae876-0cf7-4171-a256-8205bbd810f7-tmp-dir\") pod \"dns-default-tcvc8\" (UID: \"4e7ae876-0cf7-4171-a256-8205bbd810f7\") " pod="openshift-dns/dns-default-tcvc8"
Jul 02 21:19:53.160306 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.160197    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ecf34a8b-a6e9-48cf-9097-74ac6ab7c362-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kjzfh\" (UID: \"ecf34a8b-a6e9-48cf-9097-74ac6ab7c362\") " pod="openshift-insights/insights-runtime-extractor-kjzfh"
Jul 02 21:19:53.160306 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.160285    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7ae876-0cf7-4171-a256-8205bbd810f7-config-volume\") pod \"dns-default-tcvc8\" (UID: \"4e7ae876-0cf7-4171-a256-8205bbd810f7\") " pod="openshift-dns/dns-default-tcvc8"
Jul 02 21:19:53.163426 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.163403    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ecf34a8b-a6e9-48cf-9097-74ac6ab7c362-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kjzfh\" (UID: \"ecf34a8b-a6e9-48cf-9097-74ac6ab7c362\") " pod="openshift-insights/insights-runtime-extractor-kjzfh"
Jul 02 21:19:53.163532 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.163516    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e7ae876-0cf7-4171-a256-8205bbd810f7-metrics-tls\") pod \"dns-default-tcvc8\" (UID: \"4e7ae876-0cf7-4171-a256-8205bbd810f7\") " pod="openshift-dns/dns-default-tcvc8"
Jul 02 21:19:53.163626 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.163610    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65655d15-640b-4caf-8715-750aeb8b1887-cert\") pod \"ingress-canary-4scqp\" (UID: \"65655d15-640b-4caf-8715-750aeb8b1887\") " pod="openshift-ingress-canary/ingress-canary-4scqp"
Jul 02 21:19:53.171710 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.171682    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxsjm\" (UniqueName: \"kubernetes.io/projected/65655d15-640b-4caf-8715-750aeb8b1887-kube-api-access-zxsjm\") pod \"ingress-canary-4scqp\" (UID: \"65655d15-640b-4caf-8715-750aeb8b1887\") " pod="openshift-ingress-canary/ingress-canary-4scqp"
Jul 02 21:19:53.172164 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.172138    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgdwv\" (UniqueName: \"kubernetes.io/projected/ecf34a8b-a6e9-48cf-9097-74ac6ab7c362-kube-api-access-sgdwv\") pod \"insights-runtime-extractor-kjzfh\" (UID: \"ecf34a8b-a6e9-48cf-9097-74ac6ab7c362\") " pod="openshift-insights/insights-runtime-extractor-kjzfh"
Jul 02 21:19:53.173933 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.173910    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbb46\" (UniqueName: \"kubernetes.io/projected/4e7ae876-0cf7-4171-a256-8205bbd810f7-kube-api-access-vbb46\") pod \"dns-default-tcvc8\" (UID: \"4e7ae876-0cf7-4171-a256-8205bbd810f7\") " pod="openshift-dns/dns-default-tcvc8"
Jul 02 21:19:53.287203 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.287177    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4scqp"
Jul 02 21:19:53.304424 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.304396    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kjzfh"
Jul 02 21:19:53.326811 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.326792    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tcvc8"
Jul 02 21:19:53.495962 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.495920    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4scqp"]
Jul 02 21:19:53.499654 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:19:53.499621    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65655d15_640b_4caf_8715_750aeb8b1887.slice/crio-62758139fa8141abf83cdbb8c47f00b42bb1cb9a9f5257d1c8dc2506c6e7b639 WatchSource:0}: Error finding container 62758139fa8141abf83cdbb8c47f00b42bb1cb9a9f5257d1c8dc2506c6e7b639: Status 404 returned error can't find the container with id 62758139fa8141abf83cdbb8c47f00b42bb1cb9a9f5257d1c8dc2506c6e7b639
Jul 02 21:19:53.520325 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.520303    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kjzfh"]
Jul 02 21:19:53.522551 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:19:53.522531    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf34a8b_a6e9_48cf_9097_74ac6ab7c362.slice/crio-07f25cbc6b9bda736ef5a2a1c545c657b3c8e7d7e7a8cf42f259738e5b89db42 WatchSource:0}: Error finding container 07f25cbc6b9bda736ef5a2a1c545c657b3c8e7d7e7a8cf42f259738e5b89db42: Status 404 returned error can't find the container with id 07f25cbc6b9bda736ef5a2a1c545c657b3c8e7d7e7a8cf42f259738e5b89db42
Jul 02 21:19:53.538632 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:53.538555    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tcvc8"]
Jul 02 21:19:53.541833 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:19:53.541803    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7ae876_0cf7_4171_a256_8205bbd810f7.slice/crio-0c5e5b27ab48d10db990809acb5828023dd7cac4021a5675491aaee7708209d2 WatchSource:0}: Error finding container 0c5e5b27ab48d10db990809acb5828023dd7cac4021a5675491aaee7708209d2: Status 404 returned error can't find the container with id 0c5e5b27ab48d10db990809acb5828023dd7cac4021a5675491aaee7708209d2
Jul 02 21:19:54.068216 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:54.067015    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:19:54.068216 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:54.067343    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:19:54.068216 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:54.067869    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:19:54.073104 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:54.070089    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 21:19:54.073104 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:54.072238    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 21:19:54.073104 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:54.072461    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-r4vsj\""
Jul 02 21:19:54.073104 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:54.072685    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 21:19:54.073104 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:54.072942    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 21:19:54.073104 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:54.073033    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lxff2\""
Jul 02 21:19:54.317241 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:54.317202    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tcvc8" event={"ID":"4e7ae876-0cf7-4171-a256-8205bbd810f7","Type":"ContainerStarted","Data":"0c5e5b27ab48d10db990809acb5828023dd7cac4021a5675491aaee7708209d2"}
Jul 02 21:19:54.318853 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:54.318789    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kjzfh" event={"ID":"ecf34a8b-a6e9-48cf-9097-74ac6ab7c362","Type":"ContainerStarted","Data":"07f25cbc6b9bda736ef5a2a1c545c657b3c8e7d7e7a8cf42f259738e5b89db42"}
Jul 02 21:19:54.321369 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:54.321342    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4scqp" event={"ID":"65655d15-640b-4caf-8715-750aeb8b1887","Type":"ContainerStarted","Data":"62758139fa8141abf83cdbb8c47f00b42bb1cb9a9f5257d1c8dc2506c6e7b639"}
Jul 02 21:19:55.939587 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:55.939417    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-sxkx5"]
Jul 02 21:19:55.962423 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:55.962381    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-sxkx5"]
Jul 02 21:19:55.962580 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:55.962548    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-sxkx5"
Jul 02 21:19:55.967954 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:55.967928    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-k544r\""
Jul 02 21:19:55.968105 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:55.967938    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 02 21:19:55.968355 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:55.968334    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 02 21:19:55.968809 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:55.968792    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 02 21:19:55.969375 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:55.969158    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 02 21:19:55.969375 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:55.969182    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 02 21:19:55.985864 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:55.985829    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14f107b6-85d1-4377-912f-442b949a0b1b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-sxkx5\" (UID: \"14f107b6-85d1-4377-912f-442b949a0b1b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-sxkx5"
Jul 02 21:19:55.985976 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:55.985947    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7g52\" (UniqueName: \"kubernetes.io/projected/14f107b6-85d1-4377-912f-442b949a0b1b-kube-api-access-d7g52\") pod \"prometheus-operator-567f7b8b7c-sxkx5\" (UID: \"14f107b6-85d1-4377-912f-442b949a0b1b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-sxkx5"
Jul 02 21:19:55.986048 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:55.986026    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14f107b6-85d1-4377-912f-442b949a0b1b-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-sxkx5\" (UID: \"14f107b6-85d1-4377-912f-442b949a0b1b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-sxkx5"
Jul 02 21:19:55.986130 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:55.986086    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/14f107b6-85d1-4377-912f-442b949a0b1b-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-sxkx5\" (UID: \"14f107b6-85d1-4377-912f-442b949a0b1b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-sxkx5"
Jul 02 21:19:56.086448 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:56.086417    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14f107b6-85d1-4377-912f-442b949a0b1b-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-sxkx5\" (UID: \"14f107b6-85d1-4377-912f-442b949a0b1b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-sxkx5"
Jul 02 21:19:56.086647 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:56.086487    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/14f107b6-85d1-4377-912f-442b949a0b1b-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-sxkx5\" (UID: \"14f107b6-85d1-4377-912f-442b949a0b1b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-sxkx5"
Jul 02 21:19:56.086647 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:56.086528    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14f107b6-85d1-4377-912f-442b949a0b1b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-sxkx5\" (UID: \"14f107b6-85d1-4377-912f-442b949a0b1b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-sxkx5"
Jul 02 21:19:56.086647 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:56.086585    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7g52\" (UniqueName: \"kubernetes.io/projected/14f107b6-85d1-4377-912f-442b949a0b1b-kube-api-access-d7g52\") pod \"prometheus-operator-567f7b8b7c-sxkx5\" (UID: \"14f107b6-85d1-4377-912f-442b949a0b1b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-sxkx5"
Jul 02 21:19:56.087189 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:56.087166    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14f107b6-85d1-4377-912f-442b949a0b1b-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-sxkx5\" (UID: \"14f107b6-85d1-4377-912f-442b949a0b1b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-sxkx5"
Jul 02 21:19:56.090089 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:56.090064    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14f107b6-85d1-4377-912f-442b949a0b1b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-sxkx5\" (UID: \"14f107b6-85d1-4377-912f-442b949a0b1b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-sxkx5"
Jul 02 21:19:56.090199 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:56.090168    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/14f107b6-85d1-4377-912f-442b949a0b1b-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-sxkx5\" (UID: \"14f107b6-85d1-4377-912f-442b949a0b1b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-sxkx5"
Jul 02 21:19:56.100165 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:56.100142    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7g52\" (UniqueName: \"kubernetes.io/projected/14f107b6-85d1-4377-912f-442b949a0b1b-kube-api-access-d7g52\") pod \"prometheus-operator-567f7b8b7c-sxkx5\" (UID: \"14f107b6-85d1-4377-912f-442b949a0b1b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-sxkx5"
Jul 02 21:19:56.273733 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:56.273695    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-sxkx5"
Jul 02 21:19:59.248741 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:59.248676    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-sxkx5"]
Jul 02 21:19:59.337412 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:59.337358    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4scqp" event={"ID":"65655d15-640b-4caf-8715-750aeb8b1887","Type":"ContainerStarted","Data":"0f645fc4f1419bed91cea0c67987ef96fd02cf893c8a0adf0e883cbf997f19dc"}
Jul 02 21:19:59.338598 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:59.338547    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-sxkx5" event={"ID":"14f107b6-85d1-4377-912f-442b949a0b1b","Type":"ContainerStarted","Data":"77a5bffbcb42d2021f47648125460c433e2076207aa656b0c062d8d96c8a8576"}
Jul 02 21:19:59.340370 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:59.340350    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kjzfh" event={"ID":"ecf34a8b-a6e9-48cf-9097-74ac6ab7c362","Type":"ContainerStarted","Data":"e0cb864bda098b871220cec2279d0912433d06fc80dbf0d6d24973bc9fe244bc"}
Jul 02 21:19:59.346352 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:59.346312    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tcvc8" event={"ID":"4e7ae876-0cf7-4171-a256-8205bbd810f7","Type":"ContainerStarted","Data":"8d3e5111d76741d9b312de69f7e5f7a9ae057abced8d99dabd1ab137da6f9c1e"}
Jul 02 21:19:59.364375 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:19:59.364255    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4scqp" podStartSLOduration=1.727685844 podStartE2EDuration="7.364235347s" totalImagesPullingTime="5.636549503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:59.362989082 +0000 UTC m=+60.200229128" watchObservedRunningTime="2026-07-02 21:19:59.364235347 +0000 UTC m=+60.201475396"
Jul 02 21:20:00.350789 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:00.350753    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kjzfh" event={"ID":"ecf34a8b-a6e9-48cf-9097-74ac6ab7c362","Type":"ContainerStarted","Data":"5b559365b7b8daa65fb726cf9af51e2f4e8b2fd03030b5b45a3eba77477cbe84"}
Jul 02 21:20:00.352698 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:00.352667    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tcvc8" event={"ID":"4e7ae876-0cf7-4171-a256-8205bbd810f7","Type":"ContainerStarted","Data":"55f10c1892538d375e32b3b8871482a29c1d1a2d1dfc32fbe4c7bff551af3114"}
Jul 02 21:20:00.374745 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:00.374692    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tcvc8" podStartSLOduration=2.805941148 podStartE2EDuration="8.374673069s" totalImagesPullingTime="5.568731921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:00.374022002 +0000 UTC m=+61.211262049" watchObservedRunningTime="2026-07-02 21:20:00.374673069 +0000 UTC m=+61.211913118"
Jul 02 21:20:01.266239 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:01.266208    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hlmv7"
Jul 02 21:20:01.362596 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:01.362559    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-sxkx5" event={"ID":"14f107b6-85d1-4377-912f-442b949a0b1b","Type":"ContainerStarted","Data":"d31376ea11314c4c41192a92e752f5e3aa09c123f17d2a2bda2020d17bcc6c0c"}
Jul 02 21:20:01.363202 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:01.362622    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-sxkx5" event={"ID":"14f107b6-85d1-4377-912f-442b949a0b1b","Type":"ContainerStarted","Data":"1342d43e31b559d9d7851944e44cd2d320c2086f295a58ea1fb4adbb4782e2e9"}
Jul 02 21:20:01.363202 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:01.362653    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tcvc8"
Jul 02 21:20:01.402987 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:01.402941    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-sxkx5" podStartSLOduration=5.237022433 podStartE2EDuration="6.402922873s" totalImagesPullingTime="1.16590044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:01.392987355 +0000 UTC m=+62.230227404" watchObservedRunningTime="2026-07-02 21:20:01.402922873 +0000 UTC m=+62.240162919"
Jul 02 21:20:03.370649 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:03.370612    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kjzfh" event={"ID":"ecf34a8b-a6e9-48cf-9097-74ac6ab7c362","Type":"ContainerStarted","Data":"180a925519a75804430db4199ae33b3df3447da8b16de84e3189e8bbaeeac67b"}
Jul 02 21:20:03.423836 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:03.423785    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kjzfh" podStartSLOduration=2.641202584 podStartE2EDuration="11.423766289s" totalImagesPullingTime="8.782563705s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:19:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:03.423667274 +0000 UTC m=+64.260907322" watchObservedRunningTime="2026-07-02 21:20:03.423766289 +0000 UTC m=+64.261006337"
Jul 02 21:20:04.044635 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.044596    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd"]
Jul 02 21:20:04.048197 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.048163    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd"
Jul 02 21:20:04.051716 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.051694    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 02 21:20:04.051860 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.051812    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4gjnh\""
Jul 02 21:20:04.054988 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.054734    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 02 21:20:04.055983 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.055966    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 02 21:20:04.064202 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.064179    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-zr8d2"]
Jul 02 21:20:04.066237 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.066222    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mc95h"]
Jul 02 21:20:04.066346 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.066332    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-zr8d2"
Jul 02 21:20:04.068235 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.068213    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.074383 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.074364    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jmtvw\""
Jul 02 21:20:04.074470 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.074456    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 02 21:20:04.075159 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.075142    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 02 21:20:04.078356 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.078338    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 02 21:20:04.089062 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.089044    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 02 21:20:04.089582 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.089564    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-plzdj\""
Jul 02 21:20:04.089625 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.089567    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 02 21:20:04.147830 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.147791    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd"]
Jul 02 21:20:04.149542 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.149524    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c00f52df-cbd9-4514-9124-c4bf250131b9-node-exporter-textfile\") pod \"node-exporter-mc95h\" (UID: \"c00f52df-cbd9-4514-9124-c4bf250131b9\") " pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.149624 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.149551    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c00f52df-cbd9-4514-9124-c4bf250131b9-node-exporter-tls\") pod \"node-exporter-mc95h\" (UID: \"c00f52df-cbd9-4514-9124-c4bf250131b9\") " pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.149624 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.149574    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c00f52df-cbd9-4514-9124-c4bf250131b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mc95h\" (UID: \"c00f52df-cbd9-4514-9124-c4bf250131b9\") " pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.149624 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.149619    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32e0b40a-797e-4f77-addc-87a3076aef35-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-zr8d2\" (UID: \"32e0b40a-797e-4f77-addc-87a3076aef35\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-zr8d2"
Jul 02 21:20:04.149742 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.149649    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4a25465f-d7ff-4356-8bb1-ab8ef7a166a7-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-4ghfd\" (UID: \"4a25465f-d7ff-4356-8bb1-ab8ef7a166a7\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd"
Jul 02 21:20:04.149742 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.149665    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ccp9\" (UniqueName: \"kubernetes.io/projected/c00f52df-cbd9-4514-9124-c4bf250131b9-kube-api-access-4ccp9\") pod \"node-exporter-mc95h\" (UID: \"c00f52df-cbd9-4514-9124-c4bf250131b9\") " pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.149742 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.149723    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c00f52df-cbd9-4514-9124-c4bf250131b9-node-exporter-wtmp\") pod \"node-exporter-mc95h\" (UID: \"c00f52df-cbd9-4514-9124-c4bf250131b9\") " pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.149835 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.149763    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22qsr\" (UniqueName: \"kubernetes.io/projected/32e0b40a-797e-4f77-addc-87a3076aef35-kube-api-access-22qsr\") pod \"openshift-state-metrics-54cf96dfcf-zr8d2\" (UID: \"32e0b40a-797e-4f77-addc-87a3076aef35\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-zr8d2"
Jul 02 21:20:04.149835 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.149784    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a25465f-d7ff-4356-8bb1-ab8ef7a166a7-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-4ghfd\" (UID: \"4a25465f-d7ff-4356-8bb1-ab8ef7a166a7\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd"
Jul 02 21:20:04.149835 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.149802    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32e0b40a-797e-4f77-addc-87a3076aef35-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-zr8d2\" (UID: \"32e0b40a-797e-4f77-addc-87a3076aef35\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-zr8d2"
Jul 02 21:20:04.149934 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.149876    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c00f52df-cbd9-4514-9124-c4bf250131b9-sys\") pod \"node-exporter-mc95h\" (UID: \"c00f52df-cbd9-4514-9124-c4bf250131b9\") " pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.149934 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.149899    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a25465f-d7ff-4356-8bb1-ab8ef7a166a7-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-4ghfd\" (UID: \"4a25465f-d7ff-4356-8bb1-ab8ef7a166a7\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd"
Jul 02 21:20:04.149993 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.149934    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/32e0b40a-797e-4f77-addc-87a3076aef35-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-zr8d2\" (UID: \"32e0b40a-797e-4f77-addc-87a3076aef35\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-zr8d2"
Jul 02 21:20:04.149993 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.149956    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v69gf\" (UniqueName: \"kubernetes.io/projected/4a25465f-d7ff-4356-8bb1-ab8ef7a166a7-kube-api-access-v69gf\") pod \"kube-state-metrics-7c9dbb8b7d-4ghfd\" (UID: \"4a25465f-d7ff-4356-8bb1-ab8ef7a166a7\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd"
Jul 02 21:20:04.150064 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.149996    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c00f52df-cbd9-4514-9124-c4bf250131b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-mc95h\" (UID: \"c00f52df-cbd9-4514-9124-c4bf250131b9\") " pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.150064 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.150018    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c00f52df-cbd9-4514-9124-c4bf250131b9-root\") pod \"node-exporter-mc95h\" (UID: \"c00f52df-cbd9-4514-9124-c4bf250131b9\") " pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.150064 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.150053    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4a25465f-d7ff-4356-8bb1-ab8ef7a166a7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-4ghfd\" (UID: \"4a25465f-d7ff-4356-8bb1-ab8ef7a166a7\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd"
Jul 02 21:20:04.150231 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.150132    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a25465f-d7ff-4356-8bb1-ab8ef7a166a7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-4ghfd\" (UID: \"4a25465f-d7ff-4356-8bb1-ab8ef7a166a7\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd"
Jul 02 21:20:04.150231 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.150173    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c00f52df-cbd9-4514-9124-c4bf250131b9-metrics-client-ca\") pod \"node-exporter-mc95h\" (UID: \"c00f52df-cbd9-4514-9124-c4bf250131b9\") " pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.222608 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.222575    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-zr8d2"]
Jul 02 21:20:04.250865 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.250829    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32e0b40a-797e-4f77-addc-87a3076aef35-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-zr8d2\" (UID: \"32e0b40a-797e-4f77-addc-87a3076aef35\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-zr8d2"
Jul 02 21:20:04.251058 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.250884    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c00f52df-cbd9-4514-9124-c4bf250131b9-sys\") pod \"node-exporter-mc95h\" (UID: \"c00f52df-cbd9-4514-9124-c4bf250131b9\") " pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.251058 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.250903    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a25465f-d7ff-4356-8bb1-ab8ef7a166a7-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-4ghfd\" (UID: \"4a25465f-d7ff-4356-8bb1-ab8ef7a166a7\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd"
Jul 02 21:20:04.251058 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.250923    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/32e0b40a-797e-4f77-addc-87a3076aef35-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-zr8d2\" (UID: \"32e0b40a-797e-4f77-addc-87a3076aef35\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-zr8d2"
Jul 02 21:20:04.251058 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.250948    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v69gf\" (UniqueName: \"kubernetes.io/projected/4a25465f-d7ff-4356-8bb1-ab8ef7a166a7-kube-api-access-v69gf\") pod \"kube-state-metrics-7c9dbb8b7d-4ghfd\" (UID: \"4a25465f-d7ff-4356-8bb1-ab8ef7a166a7\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd"
Jul 02 21:20:04.251058 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.250962    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c00f52df-cbd9-4514-9124-c4bf250131b9-sys\") pod \"node-exporter-mc95h\" (UID: \"c00f52df-cbd9-4514-9124-c4bf250131b9\") " pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.251058 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.250972    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c00f52df-cbd9-4514-9124-c4bf250131b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-mc95h\" (UID: \"c00f52df-cbd9-4514-9124-c4bf250131b9\") " pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.251058 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.251057    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c00f52df-cbd9-4514-9124-c4bf250131b9-root\") pod \"node-exporter-mc95h\" (UID: \"c00f52df-cbd9-4514-9124-c4bf250131b9\") " pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.251466 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.251169    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c00f52df-cbd9-4514-9124-c4bf250131b9-root\") pod \"node-exporter-mc95h\" (UID: \"c00f52df-cbd9-4514-9124-c4bf250131b9\") " pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.251466 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.251092    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4a25465f-d7ff-4356-8bb1-ab8ef7a166a7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-4ghfd\" (UID: \"4a25465f-d7ff-4356-8bb1-ab8ef7a166a7\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd"
Jul 02 21:20:04.251466 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.251412    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a25465f-d7ff-4356-8bb1-ab8ef7a166a7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-4ghfd\" (UID: \"4a25465f-d7ff-4356-8bb1-ab8ef7a166a7\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd"
Jul 02 21:20:04.251466 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.251443    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c00f52df-cbd9-4514-9124-c4bf250131b9-metrics-client-ca\") pod \"node-exporter-mc95h\" (UID: \"c00f52df-cbd9-4514-9124-c4bf250131b9\") " pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.251657 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.251473    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c00f52df-cbd9-4514-9124-c4bf250131b9-node-exporter-textfile\") pod \"node-exporter-mc95h\" (UID: \"c00f52df-cbd9-4514-9124-c4bf250131b9\") " pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.251657 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.251500    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c00f52df-cbd9-4514-9124-c4bf250131b9-node-exporter-tls\") pod \"node-exporter-mc95h\" (UID: \"c00f52df-cbd9-4514-9124-c4bf250131b9\") " pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.251657 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.251536    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c00f52df-cbd9-4514-9124-c4bf250131b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mc95h\" (UID: \"c00f52df-cbd9-4514-9124-c4bf250131b9\") " pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.251657 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.251611    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32e0b40a-797e-4f77-addc-87a3076aef35-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-zr8d2\" (UID: \"32e0b40a-797e-4f77-addc-87a3076aef35\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-zr8d2"
Jul 02 21:20:04.251657 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.251614    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c00f52df-cbd9-4514-9124-c4bf250131b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-mc95h\" (UID: \"c00f52df-cbd9-4514-9124-c4bf250131b9\") " pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.251657 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.251655    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4a25465f-d7ff-4356-8bb1-ab8ef7a166a7-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-4ghfd\" (UID: \"4a25465f-d7ff-4356-8bb1-ab8ef7a166a7\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd"
Jul 02 21:20:04.251965 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.251684    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ccp9\" (UniqueName: \"kubernetes.io/projected/c00f52df-cbd9-4514-9124-c4bf250131b9-kube-api-access-4ccp9\") pod \"node-exporter-mc95h\" (UID: \"c00f52df-cbd9-4514-9124-c4bf250131b9\") " pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.251965 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.251719    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c00f52df-cbd9-4514-9124-c4bf250131b9-node-exporter-wtmp\") pod \"node-exporter-mc95h\" (UID: \"c00f52df-cbd9-4514-9124-c4bf250131b9\") " pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.251965 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.251752    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22qsr\" (UniqueName: \"kubernetes.io/projected/32e0b40a-797e-4f77-addc-87a3076aef35-kube-api-access-22qsr\") pod \"openshift-state-metrics-54cf96dfcf-zr8d2\" (UID: \"32e0b40a-797e-4f77-addc-87a3076aef35\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-zr8d2"
Jul 02 21:20:04.251965 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.251779    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a25465f-d7ff-4356-8bb1-ab8ef7a166a7-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-4ghfd\" (UID: \"4a25465f-d7ff-4356-8bb1-ab8ef7a166a7\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd"
Jul 02 21:20:04.251965 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.251903    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4a25465f-d7ff-4356-8bb1-ab8ef7a166a7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-4ghfd\" (UID: \"4a25465f-d7ff-4356-8bb1-ab8ef7a166a7\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd"
Jul 02 21:20:04.252245 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.252231    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c00f52df-cbd9-4514-9124-c4bf250131b9-node-exporter-wtmp\") pod \"node-exporter-mc95h\" (UID: \"c00f52df-cbd9-4514-9124-c4bf250131b9\") " pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.252462 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.252435    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a25465f-d7ff-4356-8bb1-ab8ef7a166a7-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-4ghfd\" (UID: \"4a25465f-d7ff-4356-8bb1-ab8ef7a166a7\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd"
Jul 02 21:20:04.252653 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.252626    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c00f52df-cbd9-4514-9124-c4bf250131b9-metrics-client-ca\") pod \"node-exporter-mc95h\" (UID: \"c00f52df-cbd9-4514-9124-c4bf250131b9\") " pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.252828 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.252773    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32e0b40a-797e-4f77-addc-87a3076aef35-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-zr8d2\" (UID: \"32e0b40a-797e-4f77-addc-87a3076aef35\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-zr8d2"
Jul 02 21:20:04.252960 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.252926    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4a25465f-d7ff-4356-8bb1-ab8ef7a166a7-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-4ghfd\" (UID: \"4a25465f-d7ff-4356-8bb1-ab8ef7a166a7\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd"
Jul 02 21:20:04.253073 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.253039    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c00f52df-cbd9-4514-9124-c4bf250131b9-node-exporter-textfile\") pod \"node-exporter-mc95h\" (UID: \"c00f52df-cbd9-4514-9124-c4bf250131b9\") " pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.253937 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.253905    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a25465f-d7ff-4356-8bb1-ab8ef7a166a7-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-4ghfd\" (UID: \"4a25465f-d7ff-4356-8bb1-ab8ef7a166a7\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd"
Jul 02 21:20:04.254033 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.253941    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/32e0b40a-797e-4f77-addc-87a3076aef35-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-zr8d2\" (UID: \"32e0b40a-797e-4f77-addc-87a3076aef35\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-zr8d2"
Jul 02 21:20:04.254033 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.253994    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32e0b40a-797e-4f77-addc-87a3076aef35-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-zr8d2\" (UID: \"32e0b40a-797e-4f77-addc-87a3076aef35\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-zr8d2"
Jul 02 21:20:04.254632 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.254610    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a25465f-d7ff-4356-8bb1-ab8ef7a166a7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-4ghfd\" (UID: \"4a25465f-d7ff-4356-8bb1-ab8ef7a166a7\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd"
Jul 02 21:20:04.254727 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.254680    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c00f52df-cbd9-4514-9124-c4bf250131b9-node-exporter-tls\") pod \"node-exporter-mc95h\" (UID: \"c00f52df-cbd9-4514-9124-c4bf250131b9\") " pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.254913 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.254898    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c00f52df-cbd9-4514-9124-c4bf250131b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mc95h\" (UID: \"c00f52df-cbd9-4514-9124-c4bf250131b9\") " pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.275591 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.275564    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ccp9\" (UniqueName: \"kubernetes.io/projected/c00f52df-cbd9-4514-9124-c4bf250131b9-kube-api-access-4ccp9\") pod \"node-exporter-mc95h\" (UID: \"c00f52df-cbd9-4514-9124-c4bf250131b9\") " pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.276881 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.276854    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22qsr\" (UniqueName: \"kubernetes.io/projected/32e0b40a-797e-4f77-addc-87a3076aef35-kube-api-access-22qsr\") pod \"openshift-state-metrics-54cf96dfcf-zr8d2\" (UID: \"32e0b40a-797e-4f77-addc-87a3076aef35\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-zr8d2"
Jul 02 21:20:04.276965 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.276865    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v69gf\" (UniqueName: \"kubernetes.io/projected/4a25465f-d7ff-4356-8bb1-ab8ef7a166a7-kube-api-access-v69gf\") pod \"kube-state-metrics-7c9dbb8b7d-4ghfd\" (UID: \"4a25465f-d7ff-4356-8bb1-ab8ef7a166a7\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd"
Jul 02 21:20:04.357824 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.357744    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd"
Jul 02 21:20:04.376387 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.376361    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-zr8d2"
Jul 02 21:20:04.380928 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.380906    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mc95h"
Jul 02 21:20:04.389735 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:20:04.389699    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00f52df_cbd9_4514_9124_c4bf250131b9.slice/crio-2f11d987692dace8cb0ae12a2869f19a103df291321d50dd12ab8f1c06d931a8 WatchSource:0}: Error finding container 2f11d987692dace8cb0ae12a2869f19a103df291321d50dd12ab8f1c06d931a8: Status 404 returned error can't find the container with id 2f11d987692dace8cb0ae12a2869f19a103df291321d50dd12ab8f1c06d931a8
Jul 02 21:20:04.516599 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.516574    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd"]
Jul 02 21:20:04.518345 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:20:04.518318    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a25465f_d7ff_4356_8bb1_ab8ef7a166a7.slice/crio-8a75b2f34acf7f21daeea02d362b85d3d88f2e9047a5d656f0dc8f7a19f73d43 WatchSource:0}: Error finding container 8a75b2f34acf7f21daeea02d362b85d3d88f2e9047a5d656f0dc8f7a19f73d43: Status 404 returned error can't find the container with id 8a75b2f34acf7f21daeea02d362b85d3d88f2e9047a5d656f0dc8f7a19f73d43
Jul 02 21:20:04.529511 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.529489    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-zr8d2"]
Jul 02 21:20:04.532287 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:20:04.532265    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e0b40a_797e_4f77_addc_87a3076aef35.slice/crio-b1752db41798715a00ddf6b7d96f06b8174e8d7cb9d591fd077b8f8f8c1ba8dd WatchSource:0}: Error finding container b1752db41798715a00ddf6b7d96f06b8174e8d7cb9d591fd077b8f8f8c1ba8dd: Status 404 returned error can't find the container with id b1752db41798715a00ddf6b7d96f06b8174e8d7cb9d591fd077b8f8f8c1ba8dd
Jul 02 21:20:04.655574 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.655544    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/453c93ba-280d-45f1-8ade-994e6242ccc7-original-pull-secret\") pod \"global-pull-secret-syncer-f8nf7\" (UID: \"453c93ba-280d-45f1-8ade-994e6242ccc7\") " pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:20:04.655706 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.655580    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fc43c39-ea1c-47da-b658-7834776917a8-metrics-certs\") pod \"network-metrics-daemon-7rzx4\" (UID: \"7fc43c39-ea1c-47da-b658-7834776917a8\") " pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:20:04.658662 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.658643    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 21:20:04.659520 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.659504    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 21:20:04.669317 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.669291    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/453c93ba-280d-45f1-8ade-994e6242ccc7-original-pull-secret\") pod \"global-pull-secret-syncer-f8nf7\" (UID: \"453c93ba-280d-45f1-8ade-994e6242ccc7\") " pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:20:04.669437 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.669356    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fc43c39-ea1c-47da-b658-7834776917a8-metrics-certs\") pod \"network-metrics-daemon-7rzx4\" (UID: \"7fc43c39-ea1c-47da-b658-7834776917a8\") " pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:20:04.718414 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.718379    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 21:20:04.720976 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.720955    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.726396 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.726371    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 02 21:20:04.726838 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.726371    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 02 21:20:04.726838 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.726373    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 02 21:20:04.726838 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.726469    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 02 21:20:04.726838 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.726378    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 02 21:20:04.726838 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.726820    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 02 21:20:04.727337 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.726896    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ksr88\""
Jul 02 21:20:04.727500 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.727485    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 02 21:20:04.727552 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.727527    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 02 21:20:04.727695 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.727682    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 02 21:20:04.756353 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.756321    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/822b29a4-70ae-4bde-9c8a-3ef180c23036-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.756353 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.756354    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svcn8\" (UniqueName: \"kubernetes.io/projected/822b29a4-70ae-4bde-9c8a-3ef180c23036-kube-api-access-svcn8\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.756570 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.756381    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.756570 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.756397    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-web-config\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.756570 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.756412    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/822b29a4-70ae-4bde-9c8a-3ef180c23036-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.756570 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.756439    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/822b29a4-70ae-4bde-9c8a-3ef180c23036-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.756570 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.756457    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/822b29a4-70ae-4bde-9c8a-3ef180c23036-tls-assets\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.756570 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.756470    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/822b29a4-70ae-4bde-9c8a-3ef180c23036-config-out\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.756570 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.756487    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.756570 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.756555    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-config-volume\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.756826 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.756611    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slw7v\" (UniqueName: \"kubernetes.io/projected/238eb474-294d-49e5-8086-6b9cf9711ac6-kube-api-access-slw7v\") pod \"network-check-target-65mqk\" (UID: \"238eb474-294d-49e5-8086-6b9cf9711ac6\") " pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:20:04.756826 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.756636    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.756826 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.756665    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.756826 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.756721    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.789068 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.788378    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 21:20:04.789068 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.788671    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 21:20:04.810987 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.810959    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 21:20:04.822552 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.822518    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slw7v\" (UniqueName: \"kubernetes.io/projected/238eb474-294d-49e5-8086-6b9cf9711ac6-kube-api-access-slw7v\") pod \"network-check-target-65mqk\" (UID: \"238eb474-294d-49e5-8086-6b9cf9711ac6\") " pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:20:04.857636 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.857604    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-config-volume\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.857788 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.857646    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.857788 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.857671    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.857788 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.857698    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.857897 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.857818    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/822b29a4-70ae-4bde-9c8a-3ef180c23036-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.857897 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.857854    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svcn8\" (UniqueName: \"kubernetes.io/projected/822b29a4-70ae-4bde-9c8a-3ef180c23036-kube-api-access-svcn8\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.857994 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.857896    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.858191 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.858164    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-web-config\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.858435 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.858349    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/822b29a4-70ae-4bde-9c8a-3ef180c23036-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.858435 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.858387    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/822b29a4-70ae-4bde-9c8a-3ef180c23036-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.858634 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.858416    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/822b29a4-70ae-4bde-9c8a-3ef180c23036-tls-assets\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.858751 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.858734    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/822b29a4-70ae-4bde-9c8a-3ef180c23036-config-out\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.858892 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.858877    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.859348 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.859315    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/822b29a4-70ae-4bde-9c8a-3ef180c23036-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.859429 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.858253    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/822b29a4-70ae-4bde-9c8a-3ef180c23036-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.861364 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.861058    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-config-volume\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.861364 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.861059    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.861364 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.861227    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.861364 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.861235    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-web-config\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.861364 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.861346    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.861685 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.861377    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.861742 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.861715    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.861793 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.861734    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/822b29a4-70ae-4bde-9c8a-3ef180c23036-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.861840 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.861819    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/822b29a4-70ae-4bde-9c8a-3ef180c23036-tls-assets\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.862918 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.862900    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/822b29a4-70ae-4bde-9c8a-3ef180c23036-config-out\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.879086 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.879038    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svcn8\" (UniqueName: \"kubernetes.io/projected/822b29a4-70ae-4bde-9c8a-3ef180c23036-kube-api-access-svcn8\") pod \"alertmanager-main-0\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:04.886872 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.886852    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8nf7"
Jul 02 21:20:04.898756 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.898734    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-r4vsj\""
Jul 02 21:20:04.906068 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.906039    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:20:04.906351 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.906328    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lxff2\""
Jul 02 21:20:04.914671 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:04.914593    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rzx4"
Jul 02 21:20:05.030330 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:05.030086    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:20:05.048017 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:05.047879    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f8nf7"]
Jul 02 21:20:05.050953 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:20:05.050924    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453c93ba_280d_45f1_8ade_994e6242ccc7.slice/crio-5dfc0ff44c1d29ef84f7598bec3d584e040509dbe45681e5f4c331e09cb6121f WatchSource:0}: Error finding container 5dfc0ff44c1d29ef84f7598bec3d584e040509dbe45681e5f4c331e09cb6121f: Status 404 returned error can't find the container with id 5dfc0ff44c1d29ef84f7598bec3d584e040509dbe45681e5f4c331e09cb6121f
Jul 02 21:20:05.076598 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:05.076520    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-65mqk"]
Jul 02 21:20:05.080422 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:20:05.080389    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod238eb474_294d_49e5_8086_6b9cf9711ac6.slice/crio-29edb1b629668c6994b4ff101db189e65004ebf0ce77f2f7bd8208a8f26e4b74 WatchSource:0}: Error finding container 29edb1b629668c6994b4ff101db189e65004ebf0ce77f2f7bd8208a8f26e4b74: Status 404 returned error can't find the container with id 29edb1b629668c6994b4ff101db189e65004ebf0ce77f2f7bd8208a8f26e4b74
Jul 02 21:20:05.098684 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:05.098592    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7rzx4"]
Jul 02 21:20:05.104012 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:20:05.103976    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc43c39_ea1c_47da_b658_7834776917a8.slice/crio-ac5a5ebb50c3ad4d97faf9287150c6a11f535d813c88993568192bb18c3e4234 WatchSource:0}: Error finding container ac5a5ebb50c3ad4d97faf9287150c6a11f535d813c88993568192bb18c3e4234: Status 404 returned error can't find the container with id ac5a5ebb50c3ad4d97faf9287150c6a11f535d813c88993568192bb18c3e4234
Jul 02 21:20:05.220465 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:05.220359    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 21:20:05.224836 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:20:05.224099    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822b29a4_70ae_4bde_9c8a_3ef180c23036.slice/crio-beb330a54ddf379d31d63055e8e53d7e61ecd920a5127c384d342c32d8543151 WatchSource:0}: Error finding container beb330a54ddf379d31d63055e8e53d7e61ecd920a5127c384d342c32d8543151: Status 404 returned error can't find the container with id beb330a54ddf379d31d63055e8e53d7e61ecd920a5127c384d342c32d8543151
Jul 02 21:20:05.380615 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:05.380569    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f8nf7" event={"ID":"453c93ba-280d-45f1-8ade-994e6242ccc7","Type":"ContainerStarted","Data":"5dfc0ff44c1d29ef84f7598bec3d584e040509dbe45681e5f4c331e09cb6121f"}
Jul 02 21:20:05.383441 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:05.383293    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-zr8d2" event={"ID":"32e0b40a-797e-4f77-addc-87a3076aef35","Type":"ContainerStarted","Data":"b4acd368245b608067b780b42a627b38e99e82ecb19d776a003a1b142339ecc1"}
Jul 02 21:20:05.383441 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:05.383359    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-zr8d2" event={"ID":"32e0b40a-797e-4f77-addc-87a3076aef35","Type":"ContainerStarted","Data":"124774bc3e19edb6f3046a8d847777099352dd163bff41969ecc726559b48486"}
Jul 02 21:20:05.383441 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:05.383390    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-zr8d2" event={"ID":"32e0b40a-797e-4f77-addc-87a3076aef35","Type":"ContainerStarted","Data":"b1752db41798715a00ddf6b7d96f06b8174e8d7cb9d591fd077b8f8f8c1ba8dd"}
Jul 02 21:20:05.385082 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:05.385054    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"822b29a4-70ae-4bde-9c8a-3ef180c23036","Type":"ContainerStarted","Data":"beb330a54ddf379d31d63055e8e53d7e61ecd920a5127c384d342c32d8543151"}
Jul 02 21:20:05.386585 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:05.386540    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7rzx4" event={"ID":"7fc43c39-ea1c-47da-b658-7834776917a8","Type":"ContainerStarted","Data":"ac5a5ebb50c3ad4d97faf9287150c6a11f535d813c88993568192bb18c3e4234"}
Jul 02 21:20:05.388072 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:05.388018    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-65mqk" event={"ID":"238eb474-294d-49e5-8086-6b9cf9711ac6","Type":"ContainerStarted","Data":"29edb1b629668c6994b4ff101db189e65004ebf0ce77f2f7bd8208a8f26e4b74"}
Jul 02 21:20:05.389548 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:05.389473    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd" event={"ID":"4a25465f-d7ff-4356-8bb1-ab8ef7a166a7","Type":"ContainerStarted","Data":"8a75b2f34acf7f21daeea02d362b85d3d88f2e9047a5d656f0dc8f7a19f73d43"}
Jul 02 21:20:05.391034 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:05.390951    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mc95h" event={"ID":"c00f52df-cbd9-4514-9124-c4bf250131b9","Type":"ContainerStarted","Data":"2f11d987692dace8cb0ae12a2869f19a103df291321d50dd12ab8f1c06d931a8"}
Jul 02 21:20:06.400604 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.400549    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd" event={"ID":"4a25465f-d7ff-4356-8bb1-ab8ef7a166a7","Type":"ContainerStarted","Data":"61a77a80740cc6435f5eec45a1cc41fe034bc923e517a49b2ad4875ec8f8b4a8"}
Jul 02 21:20:06.403375 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.403303    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mc95h" event={"ID":"c00f52df-cbd9-4514-9124-c4bf250131b9","Type":"ContainerStarted","Data":"a12e1cac22facc3864057fb5d3d8a0b6cf81bfa9fb74c39d51277dbe731d89aa"}
Jul 02 21:20:06.649328 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.649233    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-78949869bb-xq265"]
Jul 02 21:20:06.653766 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.653741    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:06.658398 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.657706    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fkn0d2b9lk73l\""
Jul 02 21:20:06.658398 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.657729    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 02 21:20:06.658398 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.657912    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 02 21:20:06.658398 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.657975    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 02 21:20:06.658398 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.658215    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 02 21:20:06.658398 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.658281    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7vzfk\""
Jul 02 21:20:06.658901 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.658478    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 02 21:20:06.670257 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.670178    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-78949869bb-xq265"]
Jul 02 21:20:06.776163 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.775768    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f95a85ad-d07b-4346-8f57-1afa13eb330e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78949869bb-xq265\" (UID: \"f95a85ad-d07b-4346-8f57-1afa13eb330e\") " pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:06.776163 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.775822    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f95a85ad-d07b-4346-8f57-1afa13eb330e-secret-thanos-querier-tls\") pod \"thanos-querier-78949869bb-xq265\" (UID: \"f95a85ad-d07b-4346-8f57-1afa13eb330e\") " pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:06.776163 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.775853    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f95a85ad-d07b-4346-8f57-1afa13eb330e-metrics-client-ca\") pod \"thanos-querier-78949869bb-xq265\" (UID: \"f95a85ad-d07b-4346-8f57-1afa13eb330e\") " pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:06.776163 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.775881    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f95a85ad-d07b-4346-8f57-1afa13eb330e-secret-grpc-tls\") pod \"thanos-querier-78949869bb-xq265\" (UID: \"f95a85ad-d07b-4346-8f57-1afa13eb330e\") " pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:06.776163 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.775920    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f95a85ad-d07b-4346-8f57-1afa13eb330e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78949869bb-xq265\" (UID: \"f95a85ad-d07b-4346-8f57-1afa13eb330e\") " pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:06.776163 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.775955    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f95a85ad-d07b-4346-8f57-1afa13eb330e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78949869bb-xq265\" (UID: \"f95a85ad-d07b-4346-8f57-1afa13eb330e\") " pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:06.776163 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.775988    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f95a85ad-d07b-4346-8f57-1afa13eb330e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78949869bb-xq265\" (UID: \"f95a85ad-d07b-4346-8f57-1afa13eb330e\") " pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:06.776163 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.776032    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wxvk\" (UniqueName: \"kubernetes.io/projected/f95a85ad-d07b-4346-8f57-1afa13eb330e-kube-api-access-9wxvk\") pod \"thanos-querier-78949869bb-xq265\" (UID: \"f95a85ad-d07b-4346-8f57-1afa13eb330e\") " pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:06.877731 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.877282    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f95a85ad-d07b-4346-8f57-1afa13eb330e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78949869bb-xq265\" (UID: \"f95a85ad-d07b-4346-8f57-1afa13eb330e\") " pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:06.877731 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.877346    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wxvk\" (UniqueName: \"kubernetes.io/projected/f95a85ad-d07b-4346-8f57-1afa13eb330e-kube-api-access-9wxvk\") pod \"thanos-querier-78949869bb-xq265\" (UID: \"f95a85ad-d07b-4346-8f57-1afa13eb330e\") " pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:06.877912 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.877761    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f95a85ad-d07b-4346-8f57-1afa13eb330e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78949869bb-xq265\" (UID: \"f95a85ad-d07b-4346-8f57-1afa13eb330e\") " pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:06.877912 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.877824    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f95a85ad-d07b-4346-8f57-1afa13eb330e-secret-thanos-querier-tls\") pod \"thanos-querier-78949869bb-xq265\" (UID: \"f95a85ad-d07b-4346-8f57-1afa13eb330e\") " pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:06.877912 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.877863    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f95a85ad-d07b-4346-8f57-1afa13eb330e-metrics-client-ca\") pod \"thanos-querier-78949869bb-xq265\" (UID: \"f95a85ad-d07b-4346-8f57-1afa13eb330e\") " pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:06.877912 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.877892    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f95a85ad-d07b-4346-8f57-1afa13eb330e-secret-grpc-tls\") pod \"thanos-querier-78949869bb-xq265\" (UID: \"f95a85ad-d07b-4346-8f57-1afa13eb330e\") " pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:06.878131 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.877933    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f95a85ad-d07b-4346-8f57-1afa13eb330e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78949869bb-xq265\" (UID: \"f95a85ad-d07b-4346-8f57-1afa13eb330e\") " pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:06.878131 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.877971    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f95a85ad-d07b-4346-8f57-1afa13eb330e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78949869bb-xq265\" (UID: \"f95a85ad-d07b-4346-8f57-1afa13eb330e\") " pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:06.879088 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.878843    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f95a85ad-d07b-4346-8f57-1afa13eb330e-metrics-client-ca\") pod \"thanos-querier-78949869bb-xq265\" (UID: \"f95a85ad-d07b-4346-8f57-1afa13eb330e\") " pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:06.880898 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.880865    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f95a85ad-d07b-4346-8f57-1afa13eb330e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78949869bb-xq265\" (UID: \"f95a85ad-d07b-4346-8f57-1afa13eb330e\") " pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:06.882540 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.882485    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f95a85ad-d07b-4346-8f57-1afa13eb330e-secret-grpc-tls\") pod \"thanos-querier-78949869bb-xq265\" (UID: \"f95a85ad-d07b-4346-8f57-1afa13eb330e\") " pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:06.883057 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.883009    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f95a85ad-d07b-4346-8f57-1afa13eb330e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78949869bb-xq265\" (UID: \"f95a85ad-d07b-4346-8f57-1afa13eb330e\") " pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:06.884371 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.884343    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f95a85ad-d07b-4346-8f57-1afa13eb330e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78949869bb-xq265\" (UID: \"f95a85ad-d07b-4346-8f57-1afa13eb330e\") " pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:06.889773 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.889451    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f95a85ad-d07b-4346-8f57-1afa13eb330e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78949869bb-xq265\" (UID: \"f95a85ad-d07b-4346-8f57-1afa13eb330e\") " pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:06.889773 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.889703    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f95a85ad-d07b-4346-8f57-1afa13eb330e-secret-thanos-querier-tls\") pod \"thanos-querier-78949869bb-xq265\" (UID: \"f95a85ad-d07b-4346-8f57-1afa13eb330e\") " pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:06.895476 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.895373    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wxvk\" (UniqueName: \"kubernetes.io/projected/f95a85ad-d07b-4346-8f57-1afa13eb330e-kube-api-access-9wxvk\") pod \"thanos-querier-78949869bb-xq265\" (UID: \"f95a85ad-d07b-4346-8f57-1afa13eb330e\") " pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:06.969034 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:06.968855    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:07.156270 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:07.156160    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-78949869bb-xq265"]
Jul 02 21:20:07.410917 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:07.410836    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7rzx4" event={"ID":"7fc43c39-ea1c-47da-b658-7834776917a8","Type":"ContainerStarted","Data":"c25180dff56634636a8cb14b6c94f92a0e873ca1d0708279e231e67c33ae1145"}
Jul 02 21:20:07.410917 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:07.410912    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7rzx4" event={"ID":"7fc43c39-ea1c-47da-b658-7834776917a8","Type":"ContainerStarted","Data":"ee8bceabc399d16ddc04844298af52cab8eb35ac2dd40ef564385f0f03aa901a"}
Jul 02 21:20:07.413345 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:20:07.413314    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf95a85ad_d07b_4346_8f57_1afa13eb330e.slice/crio-6897971c395e99fe9c04550c1c4f281c4a9386fc3dddc6e72f798644e745f09b WatchSource:0}: Error finding container 6897971c395e99fe9c04550c1c4f281c4a9386fc3dddc6e72f798644e745f09b: Status 404 returned error can't find the container with id 6897971c395e99fe9c04550c1c4f281c4a9386fc3dddc6e72f798644e745f09b
Jul 02 21:20:07.414429 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:07.414402    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd" event={"ID":"4a25465f-d7ff-4356-8bb1-ab8ef7a166a7","Type":"ContainerStarted","Data":"a36f17e04bbbfb5f24ca508142ba756e85023db7c354b3ca4f7a61d5e12c29e9"}
Jul 02 21:20:07.414496 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:07.414469    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd" event={"ID":"4a25465f-d7ff-4356-8bb1-ab8ef7a166a7","Type":"ContainerStarted","Data":"f59234ab956317de11797c16d5658460ea507ad69c8892c0f2fb9cc44d6c9587"}
Jul 02 21:20:07.419004 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:07.417692    2564 generic.go:356] "Generic (PLEG): container finished" podID="c00f52df-cbd9-4514-9124-c4bf250131b9" containerID="a12e1cac22facc3864057fb5d3d8a0b6cf81bfa9fb74c39d51277dbe731d89aa" exitCode=0
Jul 02 21:20:07.419004 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:07.417826    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mc95h" event={"ID":"c00f52df-cbd9-4514-9124-c4bf250131b9","Type":"ContainerDied","Data":"a12e1cac22facc3864057fb5d3d8a0b6cf81bfa9fb74c39d51277dbe731d89aa"}
Jul 02 21:20:07.422221 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:07.422072    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-zr8d2" event={"ID":"32e0b40a-797e-4f77-addc-87a3076aef35","Type":"ContainerStarted","Data":"12108d045fd399b3804e3a20b7079eec5cf135d3d02d4dbc3b8654b82f3fdf15"}
Jul 02 21:20:07.435509 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:07.435462    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7rzx4" podStartSLOduration=65.772369793 podStartE2EDuration="1m7.435441951s" totalImagesPullingTime="1.663072158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:19:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:07.434756597 +0000 UTC m=+68.271996644" watchObservedRunningTime="2026-07-02 21:20:07.435441951 +0000 UTC m=+68.272682000"
Jul 02 21:20:07.465474 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:07.465429    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-4ghfd" podStartSLOduration=2.702986732 podStartE2EDuration="4.465410298s" totalImagesPullingTime="1.762423566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:20:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:07.463716097 +0000 UTC m=+68.300956145" watchObservedRunningTime="2026-07-02 21:20:07.465410298 +0000 UTC m=+68.302650349"
Jul 02 21:20:07.492503 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:07.491229    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-zr8d2" podStartSLOduration=2.844529038 podStartE2EDuration="4.491206928s" totalImagesPullingTime="1.64667789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:20:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:07.491039979 +0000 UTC m=+68.328280026" watchObservedRunningTime="2026-07-02 21:20:07.491206928 +0000 UTC m=+68.328446977"
Jul 02 21:20:08.103181 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:08.101979    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-vqqmh"]
Jul 02 21:20:08.105560 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:08.105525    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-vqqmh"
Jul 02 21:20:08.108382 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:08.108354    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 02 21:20:08.108690 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:08.108357    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-9856q\""
Jul 02 21:20:08.125318 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:08.125279    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-vqqmh"]
Jul 02 21:20:08.190289 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:08.190154    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68st7\" (UniqueName: \"kubernetes.io/projected/1a350267-d9ef-4568-9cd2-027f8dbc7448-kube-api-access-68st7\") pod \"monitoring-plugin-d5dcff7cf-vqqmh\" (UID: \"1a350267-d9ef-4568-9cd2-027f8dbc7448\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-vqqmh"
Jul 02 21:20:08.190289 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:08.190220    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1a350267-d9ef-4568-9cd2-027f8dbc7448-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-vqqmh\" (UID: \"1a350267-d9ef-4568-9cd2-027f8dbc7448\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-vqqmh"
Jul 02 21:20:08.291273 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:08.290836    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1a350267-d9ef-4568-9cd2-027f8dbc7448-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-vqqmh\" (UID: \"1a350267-d9ef-4568-9cd2-027f8dbc7448\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-vqqmh"
Jul 02 21:20:08.291273 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:08.290969    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68st7\" (UniqueName: \"kubernetes.io/projected/1a350267-d9ef-4568-9cd2-027f8dbc7448-kube-api-access-68st7\") pod \"monitoring-plugin-d5dcff7cf-vqqmh\" (UID: \"1a350267-d9ef-4568-9cd2-027f8dbc7448\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-vqqmh"
Jul 02 21:20:08.294204 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:08.294146    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1a350267-d9ef-4568-9cd2-027f8dbc7448-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-vqqmh\" (UID: \"1a350267-d9ef-4568-9cd2-027f8dbc7448\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-vqqmh"
Jul 02 21:20:08.301185 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:08.301100    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68st7\" (UniqueName: \"kubernetes.io/projected/1a350267-d9ef-4568-9cd2-027f8dbc7448-kube-api-access-68st7\") pod \"monitoring-plugin-d5dcff7cf-vqqmh\" (UID: \"1a350267-d9ef-4568-9cd2-027f8dbc7448\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-vqqmh"
Jul 02 21:20:08.419391 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:08.419301    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-vqqmh"
Jul 02 21:20:08.428963 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:08.428931    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78949869bb-xq265" event={"ID":"f95a85ad-d07b-4346-8f57-1afa13eb330e","Type":"ContainerStarted","Data":"6897971c395e99fe9c04550c1c4f281c4a9386fc3dddc6e72f798644e745f09b"}
Jul 02 21:20:08.430774 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:08.430749    2564 generic.go:356] "Generic (PLEG): container finished" podID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerID="5516927a8028d2d82cd117b7ee5b9736a08ccf98c05732f794b5ccdba0eceba5" exitCode=0
Jul 02 21:20:08.430881 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:08.430858    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"822b29a4-70ae-4bde-9c8a-3ef180c23036","Type":"ContainerDied","Data":"5516927a8028d2d82cd117b7ee5b9736a08ccf98c05732f794b5ccdba0eceba5"}
Jul 02 21:20:08.434445 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:08.434294    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mc95h" event={"ID":"c00f52df-cbd9-4514-9124-c4bf250131b9","Type":"ContainerStarted","Data":"caaabef24a99bf2f139070389cc08d87b5a6f4abb075ce786aaadb3e10fafb54"}
Jul 02 21:20:08.434445 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:08.434352    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mc95h" event={"ID":"c00f52df-cbd9-4514-9124-c4bf250131b9","Type":"ContainerStarted","Data":"fbc4d90cc4ea06521b01e4b52e863d80cd1aa13cceb753b1b0c95d80afbeff4a"}
Jul 02 21:20:08.486614 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:08.486549    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mc95h" podStartSLOduration=3.49348341 podStartE2EDuration="5.486531782s" totalImagesPullingTime="1.888875069s" totalInitContainerRuntime="104.173303ms" isStatefulPod=true podCreationTimestamp="2026-07-02 21:20:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:08.484992597 +0000 UTC m=+69.322232645" watchObservedRunningTime="2026-07-02 21:20:08.486531782 +0000 UTC m=+69.323771808"
Jul 02 21:20:09.097943 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.097908    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"]
Jul 02 21:20:09.112636 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.112553    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"
Jul 02 21:20:09.113240 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.113210    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"]
Jul 02 21:20:09.115087 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.115068    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 02 21:20:09.115185 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.115104    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-skr68\""
Jul 02 21:20:09.115381 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.115358    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 02 21:20:09.115470 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.115394    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 02 21:20:09.115696 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.115652    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 02 21:20:09.115810 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.115722    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 02 21:20:09.120655 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.120381    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 02 21:20:09.199824 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.199784    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6599453e-5370-4a05-a765-52ad046d1955-serving-certs-ca-bundle\") pod \"telemeter-client-6dc9c85ff5-grzn4\" (UID: \"6599453e-5370-4a05-a765-52ad046d1955\") " pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"
Jul 02 21:20:09.199824 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.199824    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6599453e-5370-4a05-a765-52ad046d1955-telemeter-client-tls\") pod \"telemeter-client-6dc9c85ff5-grzn4\" (UID: \"6599453e-5370-4a05-a765-52ad046d1955\") " pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"
Jul 02 21:20:09.200060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.199849    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsj25\" (UniqueName: \"kubernetes.io/projected/6599453e-5370-4a05-a765-52ad046d1955-kube-api-access-dsj25\") pod \"telemeter-client-6dc9c85ff5-grzn4\" (UID: \"6599453e-5370-4a05-a765-52ad046d1955\") " pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"
Jul 02 21:20:09.200060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.199873    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6599453e-5370-4a05-a765-52ad046d1955-secret-telemeter-client\") pod \"telemeter-client-6dc9c85ff5-grzn4\" (UID: \"6599453e-5370-4a05-a765-52ad046d1955\") " pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"
Jul 02 21:20:09.200060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.199953    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6599453e-5370-4a05-a765-52ad046d1955-federate-client-tls\") pod \"telemeter-client-6dc9c85ff5-grzn4\" (UID: \"6599453e-5370-4a05-a765-52ad046d1955\") " pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"
Jul 02 21:20:09.200060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.200012    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6599453e-5370-4a05-a765-52ad046d1955-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dc9c85ff5-grzn4\" (UID: \"6599453e-5370-4a05-a765-52ad046d1955\") " pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"
Jul 02 21:20:09.200290 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.200076    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6599453e-5370-4a05-a765-52ad046d1955-metrics-client-ca\") pod \"telemeter-client-6dc9c85ff5-grzn4\" (UID: \"6599453e-5370-4a05-a765-52ad046d1955\") " pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"
Jul 02 21:20:09.200290 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.200127    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6599453e-5370-4a05-a765-52ad046d1955-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dc9c85ff5-grzn4\" (UID: \"6599453e-5370-4a05-a765-52ad046d1955\") " pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"
Jul 02 21:20:09.300833 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.300798    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsj25\" (UniqueName: \"kubernetes.io/projected/6599453e-5370-4a05-a765-52ad046d1955-kube-api-access-dsj25\") pod \"telemeter-client-6dc9c85ff5-grzn4\" (UID: \"6599453e-5370-4a05-a765-52ad046d1955\") " pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"
Jul 02 21:20:09.300833 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.300840    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6599453e-5370-4a05-a765-52ad046d1955-secret-telemeter-client\") pod \"telemeter-client-6dc9c85ff5-grzn4\" (UID: \"6599453e-5370-4a05-a765-52ad046d1955\") " pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"
Jul 02 21:20:09.301097 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.300864    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6599453e-5370-4a05-a765-52ad046d1955-federate-client-tls\") pod \"telemeter-client-6dc9c85ff5-grzn4\" (UID: \"6599453e-5370-4a05-a765-52ad046d1955\") " pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"
Jul 02 21:20:09.301097 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.300884    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6599453e-5370-4a05-a765-52ad046d1955-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dc9c85ff5-grzn4\" (UID: \"6599453e-5370-4a05-a765-52ad046d1955\") " pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"
Jul 02 21:20:09.301097 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.300939    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6599453e-5370-4a05-a765-52ad046d1955-metrics-client-ca\") pod \"telemeter-client-6dc9c85ff5-grzn4\" (UID: \"6599453e-5370-4a05-a765-52ad046d1955\") " pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"
Jul 02 21:20:09.301097 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.300960    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6599453e-5370-4a05-a765-52ad046d1955-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dc9c85ff5-grzn4\" (UID: \"6599453e-5370-4a05-a765-52ad046d1955\") " pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"
Jul 02 21:20:09.301097 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.301007    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6599453e-5370-4a05-a765-52ad046d1955-serving-certs-ca-bundle\") pod \"telemeter-client-6dc9c85ff5-grzn4\" (UID: \"6599453e-5370-4a05-a765-52ad046d1955\") " pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"
Jul 02 21:20:09.301097 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.301025    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6599453e-5370-4a05-a765-52ad046d1955-telemeter-client-tls\") pod \"telemeter-client-6dc9c85ff5-grzn4\" (UID: \"6599453e-5370-4a05-a765-52ad046d1955\") " pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"
Jul 02 21:20:09.301974 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.301911    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6599453e-5370-4a05-a765-52ad046d1955-serving-certs-ca-bundle\") pod \"telemeter-client-6dc9c85ff5-grzn4\" (UID: \"6599453e-5370-4a05-a765-52ad046d1955\") " pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"
Jul 02 21:20:09.301974 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.301911    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6599453e-5370-4a05-a765-52ad046d1955-metrics-client-ca\") pod \"telemeter-client-6dc9c85ff5-grzn4\" (UID: \"6599453e-5370-4a05-a765-52ad046d1955\") " pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"
Jul 02 21:20:09.302550 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.302493    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6599453e-5370-4a05-a765-52ad046d1955-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dc9c85ff5-grzn4\" (UID: \"6599453e-5370-4a05-a765-52ad046d1955\") " pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"
Jul 02 21:20:09.303978 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.303953    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6599453e-5370-4a05-a765-52ad046d1955-federate-client-tls\") pod \"telemeter-client-6dc9c85ff5-grzn4\" (UID: \"6599453e-5370-4a05-a765-52ad046d1955\") " pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"
Jul 02 21:20:09.304182 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.304156    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6599453e-5370-4a05-a765-52ad046d1955-telemeter-client-tls\") pod \"telemeter-client-6dc9c85ff5-grzn4\" (UID: \"6599453e-5370-4a05-a765-52ad046d1955\") " pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"
Jul 02 21:20:09.304298 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.304215    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6599453e-5370-4a05-a765-52ad046d1955-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dc9c85ff5-grzn4\" (UID: \"6599453e-5370-4a05-a765-52ad046d1955\") " pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"
Jul 02 21:20:09.304391 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.304369    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6599453e-5370-4a05-a765-52ad046d1955-secret-telemeter-client\") pod \"telemeter-client-6dc9c85ff5-grzn4\" (UID: \"6599453e-5370-4a05-a765-52ad046d1955\") " pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"
Jul 02 21:20:09.311702 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.311676    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsj25\" (UniqueName: \"kubernetes.io/projected/6599453e-5370-4a05-a765-52ad046d1955-kube-api-access-dsj25\") pod \"telemeter-client-6dc9c85ff5-grzn4\" (UID: \"6599453e-5370-4a05-a765-52ad046d1955\") " pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"
Jul 02 21:20:09.423258 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.423175    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"
Jul 02 21:20:09.902160 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.902107    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 21:20:09.918719 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.918683    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:09.924898 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.924512    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 02 21:20:09.924898 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.924743    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 02 21:20:09.925830 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.925748    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 02 21:20:09.926014 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.925989    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cv89tloccb7c6\""
Jul 02 21:20:09.926014 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.926008    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 02 21:20:09.927007 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.926985    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 02 21:20:09.927100 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.927068    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 02 21:20:09.927385 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.927367    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 02 21:20:09.930464 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.930138    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 02 21:20:09.930464 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.930379    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8vhlb\""
Jul 02 21:20:09.930612 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.930465    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 02 21:20:09.930612 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.930587    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 02 21:20:09.930716 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.930642    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 02 21:20:09.930857 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.930829    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 02 21:20:09.931352 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.931245    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 02 21:20:09.933857 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.933834    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 02 21:20:09.942011 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.941987    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 21:20:09.961132 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:09.961082    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 02 21:20:10.012864 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.012758    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.012864 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.012804    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.012864 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.012831    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-web-config\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.013179 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.012891    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-config\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.013179 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.012926    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.013179 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.012971    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.013179 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.012998    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.013179 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.013021    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.013179 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.013048    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.013179 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.013093    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.013179 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.013156    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.013499 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.013218    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.013499 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.013249    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.013499 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.013274    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.013499 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.013305    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.013499 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.013389    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.013499 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.013418    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-config-out\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.013499 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.013448    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.013499 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.013472    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.013890 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.013506    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2dk9\" (UniqueName: \"kubernetes.io/projected/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-kube-api-access-z2dk9\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.114356 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.114325    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.114530 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.114370    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.114530 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.114391    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.114530 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.114414    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.114530 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.114442    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.114753 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.114568    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.114753 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.114596    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.114753 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.114642    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.114753 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.114669    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.114753 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.114695    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.114753 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.114730    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.115032 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.114775    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.115257 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.115226    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.115360 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.115263    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.115421 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.115394    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.115478 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.115445    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-config-out\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.115529 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.115488    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.115529 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.115521    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.115643 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.115562    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2dk9\" (UniqueName: \"kubernetes.io/projected/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-kube-api-access-z2dk9\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.115643 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.115595    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.115643 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.115623    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.115779 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.115648    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-web-config\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.115779 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.115677    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-config\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.118312 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.118101    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.118312 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.118130    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.118312 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.118236    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.118560 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.118330    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-config\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.118842 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.118791    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.119611 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.119163    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.119611 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.119252    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.119611 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.119260    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.119797 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.119767    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.120043 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.120016    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.120218 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.120063    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.120529 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.120505    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.121146 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.121101    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-web-config\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.121599 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.121543    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.121840 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.121789    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-config-out\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.122063 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.122041    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.142012 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.141986    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2dk9\" (UniqueName: \"kubernetes.io/projected/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-kube-api-access-z2dk9\") pod \"prometheus-k8s-0\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:10.234604 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:10.234570    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:11.368307 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:11.368274    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tcvc8"
Jul 02 21:20:11.625243 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:11.624855    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4"]
Jul 02 21:20:11.630989 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:20:11.630938    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6599453e_5370_4a05_a765_52ad046d1955.slice/crio-de824d82c643d3f1a10a9b2fd0141cb958ac9854d36993aca09d35da5c8ba0dd WatchSource:0}: Error finding container de824d82c643d3f1a10a9b2fd0141cb958ac9854d36993aca09d35da5c8ba0dd: Status 404 returned error can't find the container with id de824d82c643d3f1a10a9b2fd0141cb958ac9854d36993aca09d35da5c8ba0dd
Jul 02 21:20:11.655218 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:11.655137    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-vqqmh"]
Jul 02 21:20:11.659556 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:20:11.659529    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a350267_d9ef_4568_9cd2_027f8dbc7448.slice/crio-7298a0774e84a79a641f0f4f48d273b4d5d0b7bbab64eadec5e9400f35af1953 WatchSource:0}: Error finding container 7298a0774e84a79a641f0f4f48d273b4d5d0b7bbab64eadec5e9400f35af1953: Status 404 returned error can't find the container with id 7298a0774e84a79a641f0f4f48d273b4d5d0b7bbab64eadec5e9400f35af1953
Jul 02 21:20:11.694667 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:11.693996    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 21:20:11.704771 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:20:11.704740    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2ed5a9_3ec9_4f8f_a3ba_3858705e3f18.slice/crio-d62a55b6428629d3bb817edb136929d9144a8d9d4cbcd2d52edd6d179ae966ec WatchSource:0}: Error finding container d62a55b6428629d3bb817edb136929d9144a8d9d4cbcd2d52edd6d179ae966ec: Status 404 returned error can't find the container with id d62a55b6428629d3bb817edb136929d9144a8d9d4cbcd2d52edd6d179ae966ec
Jul 02 21:20:12.461394 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:12.461271    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-vqqmh" event={"ID":"1a350267-d9ef-4568-9cd2-027f8dbc7448","Type":"ContainerStarted","Data":"7298a0774e84a79a641f0f4f48d273b4d5d0b7bbab64eadec5e9400f35af1953"}
Jul 02 21:20:12.465263 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:12.465226    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78949869bb-xq265" event={"ID":"f95a85ad-d07b-4346-8f57-1afa13eb330e","Type":"ContainerStarted","Data":"f890ca4d2790589719d3ff00da808aa01009ecbf9d36b915c9526ad865318157"}
Jul 02 21:20:12.465417 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:12.465308    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78949869bb-xq265" event={"ID":"f95a85ad-d07b-4346-8f57-1afa13eb330e","Type":"ContainerStarted","Data":"747d3ce569d7ada27fc0c23fb2b6d1d12e43a9efecebda3339e7ff61a47cb951"}
Jul 02 21:20:12.465417 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:12.465348    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78949869bb-xq265" event={"ID":"f95a85ad-d07b-4346-8f57-1afa13eb330e","Type":"ContainerStarted","Data":"7a8b84b0d47867e74a9072ef624588fc2d7c8dafc7e60b576f48171babe4c008"}
Jul 02 21:20:12.467507 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:12.467182    2564 generic.go:356] "Generic (PLEG): container finished" podID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerID="7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b" exitCode=0
Jul 02 21:20:12.467507 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:12.467300    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18","Type":"ContainerDied","Data":"7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b"}
Jul 02 21:20:12.467507 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:12.467362    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18","Type":"ContainerStarted","Data":"d62a55b6428629d3bb817edb136929d9144a8d9d4cbcd2d52edd6d179ae966ec"}
Jul 02 21:20:12.470092 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:12.469515    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f8nf7" event={"ID":"453c93ba-280d-45f1-8ade-994e6242ccc7","Type":"ContainerStarted","Data":"dc3b59d82c1ae243a0515ad248bf5876899d2fb802807d2ab335fe752cfaa5a1"}
Jul 02 21:20:12.475100 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:12.474776    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"822b29a4-70ae-4bde-9c8a-3ef180c23036","Type":"ContainerStarted","Data":"c7780c16c5e7eeb60f348a971cc4a624734870cf856888f4d7207690d1096800"}
Jul 02 21:20:12.475100 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:12.474923    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"822b29a4-70ae-4bde-9c8a-3ef180c23036","Type":"ContainerStarted","Data":"89fb8f4645f6a31b70212711ba2e664f617296b83118dbea3d2ab160b6bd099a"}
Jul 02 21:20:12.475100 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:12.474969    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"822b29a4-70ae-4bde-9c8a-3ef180c23036","Type":"ContainerStarted","Data":"56de277195c242fe421c6afa4e10985056af8b100d2cd1efac44418b64853187"}
Jul 02 21:20:12.475100 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:12.475009    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"822b29a4-70ae-4bde-9c8a-3ef180c23036","Type":"ContainerStarted","Data":"a0ab031aa535d5b8ad7a5a7ccc5044e7a86ad300a952aa8a6c1dd55ae5914c95"}
Jul 02 21:20:12.476678 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:12.475845    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"822b29a4-70ae-4bde-9c8a-3ef180c23036","Type":"ContainerStarted","Data":"ce71768e87f632700e2928147a4d9dc5a4e369bdf419e3da1bf40a5162e5aa16"}
Jul 02 21:20:12.488084 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:12.488055    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-65mqk" event={"ID":"238eb474-294d-49e5-8086-6b9cf9711ac6","Type":"ContainerStarted","Data":"fb7dd41edf4c9e15a9dfd178d303d99d04fea29d2911a63b6c9ae72f8210269a"}
Jul 02 21:20:12.488206 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:12.488169    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:20:12.489854 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:12.489833    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4" event={"ID":"6599453e-5370-4a05-a765-52ad046d1955","Type":"ContainerStarted","Data":"de824d82c643d3f1a10a9b2fd0141cb958ac9854d36993aca09d35da5c8ba0dd"}
Jul 02 21:20:12.522383 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:12.522330    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-f8nf7" podStartSLOduration=66.166052133 podStartE2EDuration="1m12.522316211s" totalImagesPullingTime="6.356264078s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:19:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:12.521419893 +0000 UTC m=+73.358659943" watchObservedRunningTime="2026-07-02 21:20:12.522316211 +0000 UTC m=+73.359556258"
Jul 02 21:20:12.548715 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:12.548653    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-65mqk" podStartSLOduration=66.229240731 podStartE2EDuration="1m12.54863784s" totalImagesPullingTime="6.319397109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:19:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:12.546843849 +0000 UTC m=+73.384083888" watchObservedRunningTime="2026-07-02 21:20:12.54863784 +0000 UTC m=+73.385877887"
Jul 02 21:20:13.500070 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:13.500030    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"822b29a4-70ae-4bde-9c8a-3ef180c23036","Type":"ContainerStarted","Data":"717208fe958b4e153530b5be9b1963adb73c99373b03658cefa784ee2d13a116"}
Jul 02 21:20:13.502000 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:13.501969    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-vqqmh" event={"ID":"1a350267-d9ef-4568-9cd2-027f8dbc7448","Type":"ContainerStarted","Data":"e2937f8c7a3fe60cc6ced819656e10090c243ef50c0f7545a74a7c7d49701496"}
Jul 02 21:20:13.502231 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:13.502186    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-vqqmh"
Jul 02 21:20:13.506700 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:13.506195    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78949869bb-xq265" event={"ID":"f95a85ad-d07b-4346-8f57-1afa13eb330e","Type":"ContainerStarted","Data":"a8c836b6aa5c6b5615c9ce613a9f65bf5d18fd8a9f1f9d31de166616487a82e8"}
Jul 02 21:20:13.506700 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:13.506279    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78949869bb-xq265" event={"ID":"f95a85ad-d07b-4346-8f57-1afa13eb330e","Type":"ContainerStarted","Data":"2bdce073e7b533f186441b997c02ec42f4c9c42a157c2c54a085f484af1be1fa"}
Jul 02 21:20:13.506700 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:13.506320    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78949869bb-xq265" event={"ID":"f95a85ad-d07b-4346-8f57-1afa13eb330e","Type":"ContainerStarted","Data":"4646fc0dae186934903a45bc5647d52ff49d13dc9f85d600a8e536aa427bbe3c"}
Jul 02 21:20:13.508554 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:13.508504    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-vqqmh"
Jul 02 21:20:13.538239 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:13.538189    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.994558154 podStartE2EDuration="9.538172701s" totalImagesPullingTime="6.525265793s" totalInitContainerRuntime="18.348754ms" isStatefulPod=false podCreationTimestamp="2026-07-02 21:20:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:13.536661295 +0000 UTC m=+74.373901345" watchObservedRunningTime="2026-07-02 21:20:13.538172701 +0000 UTC m=+74.375412754"
Jul 02 21:20:13.561829 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:13.561768    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-78949869bb-xq265" podStartSLOduration=2.29271794 podStartE2EDuration="7.561747033s" totalImagesPullingTime="5.269029093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:20:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:13.560744605 +0000 UTC m=+74.397984687" watchObservedRunningTime="2026-07-02 21:20:13.561747033 +0000 UTC m=+74.398987081"
Jul 02 21:20:13.585639 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:13.585569    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-vqqmh" podStartSLOduration=4.196665486 podStartE2EDuration="5.585545635s" totalImagesPullingTime="1.388880149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:20:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:13.583914456 +0000 UTC m=+74.421154504" watchObservedRunningTime="2026-07-02 21:20:13.585545635 +0000 UTC m=+74.422785682"
Jul 02 21:20:14.512921 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:14.512881    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4" event={"ID":"6599453e-5370-4a05-a765-52ad046d1955","Type":"ContainerStarted","Data":"643d3e8f0f0d53f91916f9bc755d0653783fc4a8ffd011eb222bd7360a598333"}
Jul 02 21:20:14.513321 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:14.512976    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4" event={"ID":"6599453e-5370-4a05-a765-52ad046d1955","Type":"ContainerStarted","Data":"84f26e29f02248f3b4e6277fc6568d7dba64aa39fbdf15fc9a92783f30c61046"}
Jul 02 21:20:14.513321 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:14.512998    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4" event={"ID":"6599453e-5370-4a05-a765-52ad046d1955","Type":"ContainerStarted","Data":"35cd2f1e41afed2fce5488686a396af83cfbfab3ae3c82e3e832fc2394ce0708"}
Jul 02 21:20:14.513750 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:14.513730    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:14.543160 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:14.543072    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6dc9c85ff5-grzn4" podStartSLOduration=3.264813128 podStartE2EDuration="5.543051221s" totalImagesPullingTime="2.278238093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:20:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:14.541108157 +0000 UTC m=+75.378348205" watchObservedRunningTime="2026-07-02 21:20:14.543051221 +0000 UTC m=+75.380291268"
Jul 02 21:20:16.525365 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:16.525326    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18","Type":"ContainerStarted","Data":"47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec"}
Jul 02 21:20:16.525733 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:16.525443    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18","Type":"ContainerStarted","Data":"d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f"}
Jul 02 21:20:16.525733 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:16.525485    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18","Type":"ContainerStarted","Data":"359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491"}
Jul 02 21:20:16.525733 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:16.525528    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18","Type":"ContainerStarted","Data":"34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291"}
Jul 02 21:20:16.525733 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:16.525564    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18","Type":"ContainerStarted","Data":"9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b"}
Jul 02 21:20:16.525733 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:16.525616    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18","Type":"ContainerStarted","Data":"d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865"}
Jul 02 21:20:16.571638 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:16.571580    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.14657768 podStartE2EDuration="7.571558507s" totalImagesPullingTime="3.406203999s" totalInitContainerRuntime="18.776828ms" isStatefulPod=false podCreationTimestamp="2026-07-02 21:20:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:16.568583085 +0000 UTC m=+77.405823134" watchObservedRunningTime="2026-07-02 21:20:16.571558507 +0000 UTC m=+77.408798553"
Jul 02 21:20:20.234968 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:20.234927    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:20:20.523892 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:20.523810    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-78949869bb-xq265"
Jul 02 21:20:35.257347 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:35.257233    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_822b29a4-70ae-4bde-9c8a-3ef180c23036/init-config-reloader/0.log"
Jul 02 21:20:35.265235 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:35.265215    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_822b29a4-70ae-4bde-9c8a-3ef180c23036/alertmanager/0.log"
Jul 02 21:20:35.274056 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:35.274033    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_822b29a4-70ae-4bde-9c8a-3ef180c23036/config-reloader/0.log"
Jul 02 21:20:35.457641 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:35.457602    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_822b29a4-70ae-4bde-9c8a-3ef180c23036/kube-rbac-proxy-web/0.log"
Jul 02 21:20:35.657367 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:35.657293    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_822b29a4-70ae-4bde-9c8a-3ef180c23036/kube-rbac-proxy/0.log"
Jul 02 21:20:35.856752 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:35.856725    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_822b29a4-70ae-4bde-9c8a-3ef180c23036/kube-rbac-proxy-metric/0.log"
Jul 02 21:20:36.058213 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:36.058188    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_822b29a4-70ae-4bde-9c8a-3ef180c23036/prom-label-proxy/0.log"
Jul 02 21:20:36.459045 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:36.458974    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-4ghfd_4a25465f-d7ff-4356-8bb1-ab8ef7a166a7/kube-state-metrics/0.log"
Jul 02 21:20:36.656841 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:36.656813    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-4ghfd_4a25465f-d7ff-4356-8bb1-ab8ef7a166a7/kube-rbac-proxy-main/0.log"
Jul 02 21:20:36.862745 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:36.862712    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-4ghfd_4a25465f-d7ff-4356-8bb1-ab8ef7a166a7/kube-rbac-proxy-self/0.log"
Jul 02 21:20:37.259336 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:37.259300    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-vqqmh_1a350267-d9ef-4568-9cd2-027f8dbc7448/monitoring-plugin/0.log"
Jul 02 21:20:38.662559 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:38.662531    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mc95h_c00f52df-cbd9-4514-9124-c4bf250131b9/init-textfile/0.log"
Jul 02 21:20:38.859666 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:38.859636    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mc95h_c00f52df-cbd9-4514-9124-c4bf250131b9/node-exporter/0.log"
Jul 02 21:20:39.056985 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:39.056956    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mc95h_c00f52df-cbd9-4514-9124-c4bf250131b9/kube-rbac-proxy/0.log"
Jul 02 21:20:39.256414 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:39.256381    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-zr8d2_32e0b40a-797e-4f77-addc-87a3076aef35/kube-rbac-proxy-main/0.log"
Jul 02 21:20:39.462920 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:39.462810    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-zr8d2_32e0b40a-797e-4f77-addc-87a3076aef35/kube-rbac-proxy-self/0.log"
Jul 02 21:20:39.660678 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:39.660652    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-zr8d2_32e0b40a-797e-4f77-addc-87a3076aef35/openshift-state-metrics/0.log"
Jul 02 21:20:39.864280 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:39.864254    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18/init-config-reloader/0.log"
Jul 02 21:20:40.057998 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:40.057975    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18/prometheus/0.log"
Jul 02 21:20:40.257766 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:40.257740    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18/config-reloader/0.log"
Jul 02 21:20:40.457318 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:40.457289    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18/thanos-sidecar/0.log"
Jul 02 21:20:40.656768 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:40.656688    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18/kube-rbac-proxy-web/0.log"
Jul 02 21:20:40.857230 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:40.857198    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18/kube-rbac-proxy/0.log"
Jul 02 21:20:41.056904 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:41.056880    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18/kube-rbac-proxy-thanos/0.log"
Jul 02 21:20:41.258105 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:41.258080    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-sxkx5_14f107b6-85d1-4377-912f-442b949a0b1b/prometheus-operator/0.log"
Jul 02 21:20:41.457161 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:41.457067    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-sxkx5_14f107b6-85d1-4377-912f-442b949a0b1b/kube-rbac-proxy/0.log"
Jul 02 21:20:41.859160 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:41.859137    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dc9c85ff5-grzn4_6599453e-5370-4a05-a765-52ad046d1955/telemeter-client/0.log"
Jul 02 21:20:42.056637 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:42.056612    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dc9c85ff5-grzn4_6599453e-5370-4a05-a765-52ad046d1955/reload/0.log"
Jul 02 21:20:42.256892 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:42.256867    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dc9c85ff5-grzn4_6599453e-5370-4a05-a765-52ad046d1955/kube-rbac-proxy/0.log"
Jul 02 21:20:42.458963 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:42.458934    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78949869bb-xq265_f95a85ad-d07b-4346-8f57-1afa13eb330e/thanos-query/0.log"
Jul 02 21:20:42.656935 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:42.656861    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78949869bb-xq265_f95a85ad-d07b-4346-8f57-1afa13eb330e/kube-rbac-proxy-web/0.log"
Jul 02 21:20:42.857505 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:42.857477    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78949869bb-xq265_f95a85ad-d07b-4346-8f57-1afa13eb330e/kube-rbac-proxy/0.log"
Jul 02 21:20:43.056540 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:43.056516    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78949869bb-xq265_f95a85ad-d07b-4346-8f57-1afa13eb330e/prom-label-proxy/0.log"
Jul 02 21:20:43.256224 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:43.256197    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78949869bb-xq265_f95a85ad-d07b-4346-8f57-1afa13eb330e/kube-rbac-proxy-rules/0.log"
Jul 02 21:20:43.457024 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:43.456952    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78949869bb-xq265_f95a85ad-d07b-4346-8f57-1afa13eb330e/kube-rbac-proxy-metrics/0.log"
Jul 02 21:20:43.508373 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:20:43.508349    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-65mqk"
Jul 02 21:21:10.234751 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:10.234711    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:10.254085 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:10.254058    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:10.759373 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:10.759347    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:24.017487 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:24.017454    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 21:21:24.017995 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:24.017766    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerName="alertmanager" containerID="cri-o://ce71768e87f632700e2928147a4d9dc5a4e369bdf419e3da1bf40a5162e5aa16" gracePeriod=120
Jul 02 21:21:24.017995 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:24.017834    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerName="kube-rbac-proxy-metric" containerID="cri-o://c7780c16c5e7eeb60f348a971cc4a624734870cf856888f4d7207690d1096800" gracePeriod=120
Jul 02 21:21:24.017995 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:24.017884    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerName="kube-rbac-proxy-web" containerID="cri-o://56de277195c242fe421c6afa4e10985056af8b100d2cd1efac44418b64853187" gracePeriod=120
Jul 02 21:21:24.017995 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:24.017919    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerName="prom-label-proxy" containerID="cri-o://717208fe958b4e153530b5be9b1963adb73c99373b03658cefa784ee2d13a116" gracePeriod=120
Jul 02 21:21:24.017995 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:24.017962    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerName="config-reloader" containerID="cri-o://a0ab031aa535d5b8ad7a5a7ccc5044e7a86ad300a952aa8a6c1dd55ae5914c95" gracePeriod=120
Jul 02 21:21:24.018303 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:24.018023    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerName="kube-rbac-proxy" containerID="cri-o://89fb8f4645f6a31b70212711ba2e664f617296b83118dbea3d2ab160b6bd099a" gracePeriod=120
Jul 02 21:21:24.800281 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:24.800250    2564 generic.go:356] "Generic (PLEG): container finished" podID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerID="717208fe958b4e153530b5be9b1963adb73c99373b03658cefa784ee2d13a116" exitCode=0
Jul 02 21:21:24.800281 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:24.800274    2564 generic.go:356] "Generic (PLEG): container finished" podID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerID="89fb8f4645f6a31b70212711ba2e664f617296b83118dbea3d2ab160b6bd099a" exitCode=0
Jul 02 21:21:24.800281 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:24.800280    2564 generic.go:356] "Generic (PLEG): container finished" podID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerID="a0ab031aa535d5b8ad7a5a7ccc5044e7a86ad300a952aa8a6c1dd55ae5914c95" exitCode=0
Jul 02 21:21:24.800281 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:24.800286    2564 generic.go:356] "Generic (PLEG): container finished" podID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerID="ce71768e87f632700e2928147a4d9dc5a4e369bdf419e3da1bf40a5162e5aa16" exitCode=0
Jul 02 21:21:24.800541 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:24.800324    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"822b29a4-70ae-4bde-9c8a-3ef180c23036","Type":"ContainerDied","Data":"717208fe958b4e153530b5be9b1963adb73c99373b03658cefa784ee2d13a116"}
Jul 02 21:21:24.800541 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:24.800392    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"822b29a4-70ae-4bde-9c8a-3ef180c23036","Type":"ContainerDied","Data":"89fb8f4645f6a31b70212711ba2e664f617296b83118dbea3d2ab160b6bd099a"}
Jul 02 21:21:24.800541 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:24.800418    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"822b29a4-70ae-4bde-9c8a-3ef180c23036","Type":"ContainerDied","Data":"a0ab031aa535d5b8ad7a5a7ccc5044e7a86ad300a952aa8a6c1dd55ae5914c95"}
Jul 02 21:21:24.800541 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:24.800443    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"822b29a4-70ae-4bde-9c8a-3ef180c23036","Type":"ContainerDied","Data":"ce71768e87f632700e2928147a4d9dc5a4e369bdf419e3da1bf40a5162e5aa16"}
Jul 02 21:21:25.261453 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.261428    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:25.356658 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.356575    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-web-config\") pod \"822b29a4-70ae-4bde-9c8a-3ef180c23036\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") "
Jul 02 21:21:25.356658 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.356612    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-cluster-tls-config\") pod \"822b29a4-70ae-4bde-9c8a-3ef180c23036\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") "
Jul 02 21:21:25.356658 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.356646    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/822b29a4-70ae-4bde-9c8a-3ef180c23036-config-out\") pod \"822b29a4-70ae-4bde-9c8a-3ef180c23036\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") "
Jul 02 21:21:25.356931 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.356672    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/822b29a4-70ae-4bde-9c8a-3ef180c23036-alertmanager-main-db\") pod \"822b29a4-70ae-4bde-9c8a-3ef180c23036\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") "
Jul 02 21:21:25.356931 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.356690    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-secret-alertmanager-kube-rbac-proxy\") pod \"822b29a4-70ae-4bde-9c8a-3ef180c23036\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") "
Jul 02 21:21:25.356931 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.356717    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-secret-alertmanager-main-tls\") pod \"822b29a4-70ae-4bde-9c8a-3ef180c23036\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") "
Jul 02 21:21:25.356931 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.356749    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/822b29a4-70ae-4bde-9c8a-3ef180c23036-metrics-client-ca\") pod \"822b29a4-70ae-4bde-9c8a-3ef180c23036\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") "
Jul 02 21:21:25.356931 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.356794    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-secret-alertmanager-kube-rbac-proxy-metric\") pod \"822b29a4-70ae-4bde-9c8a-3ef180c23036\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") "
Jul 02 21:21:25.356931 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.356835    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svcn8\" (UniqueName: \"kubernetes.io/projected/822b29a4-70ae-4bde-9c8a-3ef180c23036-kube-api-access-svcn8\") pod \"822b29a4-70ae-4bde-9c8a-3ef180c23036\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") "
Jul 02 21:21:25.356931 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.356875    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/822b29a4-70ae-4bde-9c8a-3ef180c23036-tls-assets\") pod \"822b29a4-70ae-4bde-9c8a-3ef180c23036\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") "
Jul 02 21:21:25.356931 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.356914    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-config-volume\") pod \"822b29a4-70ae-4bde-9c8a-3ef180c23036\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") "
Jul 02 21:21:25.357339 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.356950    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-secret-alertmanager-kube-rbac-proxy-web\") pod \"822b29a4-70ae-4bde-9c8a-3ef180c23036\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") "
Jul 02 21:21:25.357339 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.356980    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/822b29a4-70ae-4bde-9c8a-3ef180c23036-alertmanager-trusted-ca-bundle\") pod \"822b29a4-70ae-4bde-9c8a-3ef180c23036\" (UID: \"822b29a4-70ae-4bde-9c8a-3ef180c23036\") "
Jul 02 21:21:25.357339 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.357024    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/822b29a4-70ae-4bde-9c8a-3ef180c23036-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "822b29a4-70ae-4bde-9c8a-3ef180c23036" (UID: "822b29a4-70ae-4bde-9c8a-3ef180c23036"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:21:25.357339 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.357279    2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/822b29a4-70ae-4bde-9c8a-3ef180c23036-alertmanager-main-db\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:25.357555 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.357403    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/822b29a4-70ae-4bde-9c8a-3ef180c23036-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "822b29a4-70ae-4bde-9c8a-3ef180c23036" (UID: "822b29a4-70ae-4bde-9c8a-3ef180c23036"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:21:25.358027 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.357988    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/822b29a4-70ae-4bde-9c8a-3ef180c23036-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "822b29a4-70ae-4bde-9c8a-3ef180c23036" (UID: "822b29a4-70ae-4bde-9c8a-3ef180c23036"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:21:25.360368 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.360324    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "822b29a4-70ae-4bde-9c8a-3ef180c23036" (UID: "822b29a4-70ae-4bde-9c8a-3ef180c23036"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:25.360537 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.360508    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "822b29a4-70ae-4bde-9c8a-3ef180c23036" (UID: "822b29a4-70ae-4bde-9c8a-3ef180c23036"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:25.360743 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.360705    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "822b29a4-70ae-4bde-9c8a-3ef180c23036" (UID: "822b29a4-70ae-4bde-9c8a-3ef180c23036"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:25.360966 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.360942    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/822b29a4-70ae-4bde-9c8a-3ef180c23036-config-out" (OuterVolumeSpecName: "config-out") pod "822b29a4-70ae-4bde-9c8a-3ef180c23036" (UID: "822b29a4-70ae-4bde-9c8a-3ef180c23036"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:21:25.361040 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.360989    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-config-volume" (OuterVolumeSpecName: "config-volume") pod "822b29a4-70ae-4bde-9c8a-3ef180c23036" (UID: "822b29a4-70ae-4bde-9c8a-3ef180c23036"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:25.361040 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.360992    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822b29a4-70ae-4bde-9c8a-3ef180c23036-kube-api-access-svcn8" (OuterVolumeSpecName: "kube-api-access-svcn8") pod "822b29a4-70ae-4bde-9c8a-3ef180c23036" (UID: "822b29a4-70ae-4bde-9c8a-3ef180c23036"). InnerVolumeSpecName "kube-api-access-svcn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:21:25.361186 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.361085    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "822b29a4-70ae-4bde-9c8a-3ef180c23036" (UID: "822b29a4-70ae-4bde-9c8a-3ef180c23036"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:25.361307 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.361288    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822b29a4-70ae-4bde-9c8a-3ef180c23036-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "822b29a4-70ae-4bde-9c8a-3ef180c23036" (UID: "822b29a4-70ae-4bde-9c8a-3ef180c23036"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:21:25.365254 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.365094    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "822b29a4-70ae-4bde-9c8a-3ef180c23036" (UID: "822b29a4-70ae-4bde-9c8a-3ef180c23036"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:25.371338 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.371315    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-web-config" (OuterVolumeSpecName: "web-config") pod "822b29a4-70ae-4bde-9c8a-3ef180c23036" (UID: "822b29a4-70ae-4bde-9c8a-3ef180c23036"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:25.458473 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.458437    2564 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-config-volume\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:25.458473 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.458468    2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:25.458473 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.458479    2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/822b29a4-70ae-4bde-9c8a-3ef180c23036-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:25.458698 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.458487    2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-web-config\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:25.458698 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.458496    2564 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-cluster-tls-config\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:25.458698 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.458506    2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/822b29a4-70ae-4bde-9c8a-3ef180c23036-config-out\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:25.458698 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.458515    2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:25.458698 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.458523    2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-secret-alertmanager-main-tls\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:25.458698 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.458531    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/822b29a4-70ae-4bde-9c8a-3ef180c23036-metrics-client-ca\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:25.458698 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.458540    2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/822b29a4-70ae-4bde-9c8a-3ef180c23036-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:25.458698 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.458548    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svcn8\" (UniqueName: \"kubernetes.io/projected/822b29a4-70ae-4bde-9c8a-3ef180c23036-kube-api-access-svcn8\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:25.458698 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.458557    2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/822b29a4-70ae-4bde-9c8a-3ef180c23036-tls-assets\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:25.807810 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.807779    2564 generic.go:356] "Generic (PLEG): container finished" podID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerID="c7780c16c5e7eeb60f348a971cc4a624734870cf856888f4d7207690d1096800" exitCode=0
Jul 02 21:21:25.807810 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.807804    2564 generic.go:356] "Generic (PLEG): container finished" podID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerID="56de277195c242fe421c6afa4e10985056af8b100d2cd1efac44418b64853187" exitCode=0
Jul 02 21:21:25.808017 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.807854    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:25.808017 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.807851    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"822b29a4-70ae-4bde-9c8a-3ef180c23036","Type":"ContainerDied","Data":"c7780c16c5e7eeb60f348a971cc4a624734870cf856888f4d7207690d1096800"}
Jul 02 21:21:25.808017 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.807924    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"822b29a4-70ae-4bde-9c8a-3ef180c23036","Type":"ContainerDied","Data":"56de277195c242fe421c6afa4e10985056af8b100d2cd1efac44418b64853187"}
Jul 02 21:21:25.808017 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.807951    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"822b29a4-70ae-4bde-9c8a-3ef180c23036","Type":"ContainerDied","Data":"beb330a54ddf379d31d63055e8e53d7e61ecd920a5127c384d342c32d8543151"}
Jul 02 21:21:25.808017 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.807992    2564 scope.go:117] "RemoveContainer" containerID="717208fe958b4e153530b5be9b1963adb73c99373b03658cefa784ee2d13a116"
Jul 02 21:21:25.816225 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.816205    2564 scope.go:117] "RemoveContainer" containerID="c7780c16c5e7eeb60f348a971cc4a624734870cf856888f4d7207690d1096800"
Jul 02 21:21:25.823773 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.823756    2564 scope.go:117] "RemoveContainer" containerID="89fb8f4645f6a31b70212711ba2e664f617296b83118dbea3d2ab160b6bd099a"
Jul 02 21:21:25.831892 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.831824    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 21:21:25.832102 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.832081    2564 scope.go:117] "RemoveContainer" containerID="56de277195c242fe421c6afa4e10985056af8b100d2cd1efac44418b64853187"
Jul 02 21:21:25.835651 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.835629    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 21:21:25.839796 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.839777    2564 scope.go:117] "RemoveContainer" containerID="a0ab031aa535d5b8ad7a5a7ccc5044e7a86ad300a952aa8a6c1dd55ae5914c95"
Jul 02 21:21:25.846830 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.846813    2564 scope.go:117] "RemoveContainer" containerID="ce71768e87f632700e2928147a4d9dc5a4e369bdf419e3da1bf40a5162e5aa16"
Jul 02 21:21:25.853910 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.853886    2564 scope.go:117] "RemoveContainer" containerID="5516927a8028d2d82cd117b7ee5b9736a08ccf98c05732f794b5ccdba0eceba5"
Jul 02 21:21:25.861036 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.860940    2564 scope.go:117] "RemoveContainer" containerID="717208fe958b4e153530b5be9b1963adb73c99373b03658cefa784ee2d13a116"
Jul 02 21:21:25.861382 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:21:25.861308    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"717208fe958b4e153530b5be9b1963adb73c99373b03658cefa784ee2d13a116\": container with ID starting with 717208fe958b4e153530b5be9b1963adb73c99373b03658cefa784ee2d13a116 not found: ID does not exist" containerID="717208fe958b4e153530b5be9b1963adb73c99373b03658cefa784ee2d13a116"
Jul 02 21:21:25.861382 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.861345    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"717208fe958b4e153530b5be9b1963adb73c99373b03658cefa784ee2d13a116"} err="failed to get container status \"717208fe958b4e153530b5be9b1963adb73c99373b03658cefa784ee2d13a116\": rpc error: code = NotFound desc = could not find container \"717208fe958b4e153530b5be9b1963adb73c99373b03658cefa784ee2d13a116\": container with ID starting with 717208fe958b4e153530b5be9b1963adb73c99373b03658cefa784ee2d13a116 not found: ID does not exist"
Jul 02 21:21:25.861580 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.861392    2564 scope.go:117] "RemoveContainer" containerID="c7780c16c5e7eeb60f348a971cc4a624734870cf856888f4d7207690d1096800"
Jul 02 21:21:25.861785 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:21:25.861677    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7780c16c5e7eeb60f348a971cc4a624734870cf856888f4d7207690d1096800\": container with ID starting with c7780c16c5e7eeb60f348a971cc4a624734870cf856888f4d7207690d1096800 not found: ID does not exist" containerID="c7780c16c5e7eeb60f348a971cc4a624734870cf856888f4d7207690d1096800"
Jul 02 21:21:25.861785 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.861718    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7780c16c5e7eeb60f348a971cc4a624734870cf856888f4d7207690d1096800"} err="failed to get container status \"c7780c16c5e7eeb60f348a971cc4a624734870cf856888f4d7207690d1096800\": rpc error: code = NotFound desc = could not find container \"c7780c16c5e7eeb60f348a971cc4a624734870cf856888f4d7207690d1096800\": container with ID starting with c7780c16c5e7eeb60f348a971cc4a624734870cf856888f4d7207690d1096800 not found: ID does not exist"
Jul 02 21:21:25.861785 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.861744    2564 scope.go:117] "RemoveContainer" containerID="89fb8f4645f6a31b70212711ba2e664f617296b83118dbea3d2ab160b6bd099a"
Jul 02 21:21:25.862083 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:21:25.862062    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89fb8f4645f6a31b70212711ba2e664f617296b83118dbea3d2ab160b6bd099a\": container with ID starting with 89fb8f4645f6a31b70212711ba2e664f617296b83118dbea3d2ab160b6bd099a not found: ID does not exist" containerID="89fb8f4645f6a31b70212711ba2e664f617296b83118dbea3d2ab160b6bd099a"
Jul 02 21:21:25.862171 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.862088    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89fb8f4645f6a31b70212711ba2e664f617296b83118dbea3d2ab160b6bd099a"} err="failed to get container status \"89fb8f4645f6a31b70212711ba2e664f617296b83118dbea3d2ab160b6bd099a\": rpc error: code = NotFound desc = could not find container \"89fb8f4645f6a31b70212711ba2e664f617296b83118dbea3d2ab160b6bd099a\": container with ID starting with 89fb8f4645f6a31b70212711ba2e664f617296b83118dbea3d2ab160b6bd099a not found: ID does not exist"
Jul 02 21:21:25.862171 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.862104    2564 scope.go:117] "RemoveContainer" containerID="56de277195c242fe421c6afa4e10985056af8b100d2cd1efac44418b64853187"
Jul 02 21:21:25.862494 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:21:25.862472    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56de277195c242fe421c6afa4e10985056af8b100d2cd1efac44418b64853187\": container with ID starting with 56de277195c242fe421c6afa4e10985056af8b100d2cd1efac44418b64853187 not found: ID does not exist" containerID="56de277195c242fe421c6afa4e10985056af8b100d2cd1efac44418b64853187"
Jul 02 21:21:25.862555 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.862502    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56de277195c242fe421c6afa4e10985056af8b100d2cd1efac44418b64853187"} err="failed to get container status \"56de277195c242fe421c6afa4e10985056af8b100d2cd1efac44418b64853187\": rpc error: code = NotFound desc = could not find container \"56de277195c242fe421c6afa4e10985056af8b100d2cd1efac44418b64853187\": container with ID starting with 56de277195c242fe421c6afa4e10985056af8b100d2cd1efac44418b64853187 not found: ID does not exist"
Jul 02 21:21:25.862555 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.862521    2564 scope.go:117] "RemoveContainer" containerID="a0ab031aa535d5b8ad7a5a7ccc5044e7a86ad300a952aa8a6c1dd55ae5914c95"
Jul 02 21:21:25.862798 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:21:25.862777    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0ab031aa535d5b8ad7a5a7ccc5044e7a86ad300a952aa8a6c1dd55ae5914c95\": container with ID starting with a0ab031aa535d5b8ad7a5a7ccc5044e7a86ad300a952aa8a6c1dd55ae5914c95 not found: ID does not exist" containerID="a0ab031aa535d5b8ad7a5a7ccc5044e7a86ad300a952aa8a6c1dd55ae5914c95"
Jul 02 21:21:25.862880 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.862802    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0ab031aa535d5b8ad7a5a7ccc5044e7a86ad300a952aa8a6c1dd55ae5914c95"} err="failed to get container status \"a0ab031aa535d5b8ad7a5a7ccc5044e7a86ad300a952aa8a6c1dd55ae5914c95\": rpc error: code = NotFound desc = could not find container \"a0ab031aa535d5b8ad7a5a7ccc5044e7a86ad300a952aa8a6c1dd55ae5914c95\": container with ID starting with a0ab031aa535d5b8ad7a5a7ccc5044e7a86ad300a952aa8a6c1dd55ae5914c95 not found: ID does not exist"
Jul 02 21:21:25.862880 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.862818    2564 scope.go:117] "RemoveContainer" containerID="ce71768e87f632700e2928147a4d9dc5a4e369bdf419e3da1bf40a5162e5aa16"
Jul 02 21:21:25.863127 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:21:25.863091    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce71768e87f632700e2928147a4d9dc5a4e369bdf419e3da1bf40a5162e5aa16\": container with ID starting with ce71768e87f632700e2928147a4d9dc5a4e369bdf419e3da1bf40a5162e5aa16 not found: ID does not exist" containerID="ce71768e87f632700e2928147a4d9dc5a4e369bdf419e3da1bf40a5162e5aa16"
Jul 02 21:21:25.863207 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.863130    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce71768e87f632700e2928147a4d9dc5a4e369bdf419e3da1bf40a5162e5aa16"} err="failed to get container status \"ce71768e87f632700e2928147a4d9dc5a4e369bdf419e3da1bf40a5162e5aa16\": rpc error: code = NotFound desc = could not find container \"ce71768e87f632700e2928147a4d9dc5a4e369bdf419e3da1bf40a5162e5aa16\": container with ID starting with ce71768e87f632700e2928147a4d9dc5a4e369bdf419e3da1bf40a5162e5aa16 not found: ID does not exist"
Jul 02 21:21:25.863207 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.863145    2564 scope.go:117] "RemoveContainer" containerID="5516927a8028d2d82cd117b7ee5b9736a08ccf98c05732f794b5ccdba0eceba5"
Jul 02 21:21:25.863318 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.863262    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 21:21:25.863408 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:21:25.863393    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5516927a8028d2d82cd117b7ee5b9736a08ccf98c05732f794b5ccdba0eceba5\": container with ID starting with 5516927a8028d2d82cd117b7ee5b9736a08ccf98c05732f794b5ccdba0eceba5 not found: ID does not exist" containerID="5516927a8028d2d82cd117b7ee5b9736a08ccf98c05732f794b5ccdba0eceba5"
Jul 02 21:21:25.863451 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.863411    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5516927a8028d2d82cd117b7ee5b9736a08ccf98c05732f794b5ccdba0eceba5"} err="failed to get container status \"5516927a8028d2d82cd117b7ee5b9736a08ccf98c05732f794b5ccdba0eceba5\": rpc error: code = NotFound desc = could not find container \"5516927a8028d2d82cd117b7ee5b9736a08ccf98c05732f794b5ccdba0eceba5\": container with ID starting with 5516927a8028d2d82cd117b7ee5b9736a08ccf98c05732f794b5ccdba0eceba5 not found: ID does not exist"
Jul 02 21:21:25.863451 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.863423    2564 scope.go:117] "RemoveContainer" containerID="717208fe958b4e153530b5be9b1963adb73c99373b03658cefa784ee2d13a116"
Jul 02 21:21:25.863586 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.863571    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerName="kube-rbac-proxy-metric"
Jul 02 21:21:25.863631 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.863590    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerName="kube-rbac-proxy-metric"
Jul 02 21:21:25.863631 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.863603    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerName="prom-label-proxy"
Jul 02 21:21:25.863631 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.863610    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerName="prom-label-proxy"
Jul 02 21:21:25.863631 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.863619    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerName="config-reloader"
Jul 02 21:21:25.863631 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.863628    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerName="config-reloader"
Jul 02 21:21:25.863770 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.863637    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerName="kube-rbac-proxy"
Jul 02 21:21:25.863770 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.863643    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerName="kube-rbac-proxy"
Jul 02 21:21:25.863770 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.863661    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerName="init-config-reloader"
Jul 02 21:21:25.863770 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.863672    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerName="init-config-reloader"
Jul 02 21:21:25.863770 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.863681    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerName="kube-rbac-proxy-web"
Jul 02 21:21:25.863770 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.863686    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerName="kube-rbac-proxy-web"
Jul 02 21:21:25.863770 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.863696    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerName="alertmanager"
Jul 02 21:21:25.863770 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.863701    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerName="alertmanager"
Jul 02 21:21:25.863770 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.863657    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"717208fe958b4e153530b5be9b1963adb73c99373b03658cefa784ee2d13a116"} err="failed to get container status \"717208fe958b4e153530b5be9b1963adb73c99373b03658cefa784ee2d13a116\": rpc error: code = NotFound desc = could not find container \"717208fe958b4e153530b5be9b1963adb73c99373b03658cefa784ee2d13a116\": container with ID starting with 717208fe958b4e153530b5be9b1963adb73c99373b03658cefa784ee2d13a116 not found: ID does not exist"
Jul 02 21:21:25.863770 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.863742    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerName="kube-rbac-proxy"
Jul 02 21:21:25.863770 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.863750    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerName="kube-rbac-proxy-metric"
Jul 02 21:21:25.863770 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.863756    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerName="kube-rbac-proxy-web"
Jul 02 21:21:25.863770 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.863764    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerName="prom-label-proxy"
Jul 02 21:21:25.863770 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.863770    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerName="alertmanager"
Jul 02 21:21:25.864273 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.863777    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="822b29a4-70ae-4bde-9c8a-3ef180c23036" containerName="config-reloader"
Jul 02 21:21:25.864273 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.863748    2564 scope.go:117] "RemoveContainer" containerID="c7780c16c5e7eeb60f348a971cc4a624734870cf856888f4d7207690d1096800"
Jul 02 21:21:25.864273 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.864062    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7780c16c5e7eeb60f348a971cc4a624734870cf856888f4d7207690d1096800"} err="failed to get container status \"c7780c16c5e7eeb60f348a971cc4a624734870cf856888f4d7207690d1096800\": rpc error: code = NotFound desc = could not find container \"c7780c16c5e7eeb60f348a971cc4a624734870cf856888f4d7207690d1096800\": container with ID starting with c7780c16c5e7eeb60f348a971cc4a624734870cf856888f4d7207690d1096800 not found: ID does not exist"
Jul 02 21:21:25.864273 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.864079    2564 scope.go:117] "RemoveContainer" containerID="89fb8f4645f6a31b70212711ba2e664f617296b83118dbea3d2ab160b6bd099a"
Jul 02 21:21:25.864436 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.864376    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89fb8f4645f6a31b70212711ba2e664f617296b83118dbea3d2ab160b6bd099a"} err="failed to get container status \"89fb8f4645f6a31b70212711ba2e664f617296b83118dbea3d2ab160b6bd099a\": rpc error: code = NotFound desc = could not find container \"89fb8f4645f6a31b70212711ba2e664f617296b83118dbea3d2ab160b6bd099a\": container with ID starting with 89fb8f4645f6a31b70212711ba2e664f617296b83118dbea3d2ab160b6bd099a not found: ID does not exist"
Jul 02 21:21:25.864436 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.864392    2564 scope.go:117] "RemoveContainer" containerID="56de277195c242fe421c6afa4e10985056af8b100d2cd1efac44418b64853187"
Jul 02 21:21:25.864644 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.864627    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56de277195c242fe421c6afa4e10985056af8b100d2cd1efac44418b64853187"} err="failed to get container status \"56de277195c242fe421c6afa4e10985056af8b100d2cd1efac44418b64853187\": rpc error: code = NotFound desc = could not find container \"56de277195c242fe421c6afa4e10985056af8b100d2cd1efac44418b64853187\": container with ID starting with 56de277195c242fe421c6afa4e10985056af8b100d2cd1efac44418b64853187 not found: ID does not exist"
Jul 02 21:21:25.864694 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.864643    2564 scope.go:117] "RemoveContainer" containerID="a0ab031aa535d5b8ad7a5a7ccc5044e7a86ad300a952aa8a6c1dd55ae5914c95"
Jul 02 21:21:25.864885 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.864868    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0ab031aa535d5b8ad7a5a7ccc5044e7a86ad300a952aa8a6c1dd55ae5914c95"} err="failed to get container status \"a0ab031aa535d5b8ad7a5a7ccc5044e7a86ad300a952aa8a6c1dd55ae5914c95\": rpc error: code = NotFound desc = could not find container \"a0ab031aa535d5b8ad7a5a7ccc5044e7a86ad300a952aa8a6c1dd55ae5914c95\": container with ID starting with a0ab031aa535d5b8ad7a5a7ccc5044e7a86ad300a952aa8a6c1dd55ae5914c95 not found: ID does not exist"
Jul 02 21:21:25.864931 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.864885    2564 scope.go:117] "RemoveContainer" containerID="ce71768e87f632700e2928147a4d9dc5a4e369bdf419e3da1bf40a5162e5aa16"
Jul 02 21:21:25.865143 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.865106    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce71768e87f632700e2928147a4d9dc5a4e369bdf419e3da1bf40a5162e5aa16"} err="failed to get container status \"ce71768e87f632700e2928147a4d9dc5a4e369bdf419e3da1bf40a5162e5aa16\": rpc error: code = NotFound desc = could not find container \"ce71768e87f632700e2928147a4d9dc5a4e369bdf419e3da1bf40a5162e5aa16\": container with ID starting with ce71768e87f632700e2928147a4d9dc5a4e369bdf419e3da1bf40a5162e5aa16 not found: ID does not exist"
Jul 02 21:21:25.865143 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.865143    2564 scope.go:117] "RemoveContainer" containerID="5516927a8028d2d82cd117b7ee5b9736a08ccf98c05732f794b5ccdba0eceba5"
Jul 02 21:21:25.865367 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.865343    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5516927a8028d2d82cd117b7ee5b9736a08ccf98c05732f794b5ccdba0eceba5"} err="failed to get container status \"5516927a8028d2d82cd117b7ee5b9736a08ccf98c05732f794b5ccdba0eceba5\": rpc error: code = NotFound desc = could not find container \"5516927a8028d2d82cd117b7ee5b9736a08ccf98c05732f794b5ccdba0eceba5\": container with ID starting with 5516927a8028d2d82cd117b7ee5b9736a08ccf98c05732f794b5ccdba0eceba5 not found: ID does not exist"
Jul 02 21:21:25.866137 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.866097    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:25.868699 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.868675    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ksr88\""
Jul 02 21:21:25.869093 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.869043    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 02 21:21:25.869216 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.869075    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 02 21:21:25.869216 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.869098    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 02 21:21:25.869216 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.869169    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 02 21:21:25.869216 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.869098    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 02 21:21:25.869408 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.869099    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 02 21:21:25.869732 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.869684    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 02 21:21:25.869939 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.869882    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 02 21:21:25.877894 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.877869    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 02 21:21:25.882742 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.882718    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 21:21:25.961869 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.961825    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-config-out\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:25.962057 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.961884    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:25.962057 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.961914    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:25.962057 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.961936    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:25.962057 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.962037    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:25.962306 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.962078    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-config-volume\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:25.962306 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.962105    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:25.962306 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.962210    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-web-config\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:25.962306 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.962238    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:25.962306 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.962259    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:25.962306 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.962279    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:25.962306 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.962297    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:25.962529 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:25.962329    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2f4w\" (UniqueName: \"kubernetes.io/projected/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-kube-api-access-t2f4w\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.063258 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.063157    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2f4w\" (UniqueName: \"kubernetes.io/projected/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-kube-api-access-t2f4w\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.063258 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.063212    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-config-out\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.063491 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.063332    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.063491 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.063376    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.063491 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.063403    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.063491 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.063450    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.063677 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.063592    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-config-volume\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.063677 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.063627    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.063777 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.063696    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-web-config\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.063777 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.063724    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.063777 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.063756    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.063942 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.063788    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.063942 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.063819    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.064597 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.064248    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.064597 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.064507    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.066331 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.065999    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-config-out\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.066331 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.066270    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-config-volume\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.066487 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.066370    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.066595 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.066572    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.066662 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.066592    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.066662 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.066642    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.066910 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.066892    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.067195 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.067174    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-web-config\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.067268 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.067174    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.068227 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.068206    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.071590 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.071559    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="822b29a4-70ae-4bde-9c8a-3ef180c23036" path="/var/lib/kubelet/pods/822b29a4-70ae-4bde-9c8a-3ef180c23036/volumes"
Jul 02 21:21:26.072181 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.072165    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2f4w\" (UniqueName: \"kubernetes.io/projected/fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae-kube-api-access-t2f4w\") pod \"alertmanager-main-0\" (UID: \"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.180534 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.180499    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 02 21:21:26.313505 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.313481    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 21:21:26.315555 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:21:26.315527    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb3f3b2_f215_464e_83a4_9ab4ca8d15ae.slice/crio-e1dafda1b4254ae3654f036c7f1fce2de3ddff97dfbb56b1fed336b27749e66f WatchSource:0}: Error finding container e1dafda1b4254ae3654f036c7f1fce2de3ddff97dfbb56b1fed336b27749e66f: Status 404 returned error can't find the container with id e1dafda1b4254ae3654f036c7f1fce2de3ddff97dfbb56b1fed336b27749e66f
Jul 02 21:21:26.814169 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.814135    2564 generic.go:356] "Generic (PLEG): container finished" podID="fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae" containerID="45b547bff81047f12649cf090a9f4a9176028a99f17f30cc881d2c9dbb9363ec" exitCode=0
Jul 02 21:21:26.814361 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.814206    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae","Type":"ContainerDied","Data":"45b547bff81047f12649cf090a9f4a9176028a99f17f30cc881d2c9dbb9363ec"}
Jul 02 21:21:26.814361 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:26.814276    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae","Type":"ContainerStarted","Data":"e1dafda1b4254ae3654f036c7f1fce2de3ddff97dfbb56b1fed336b27749e66f"}
Jul 02 21:21:27.821702 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:27.821663    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae","Type":"ContainerStarted","Data":"ca3b07dd613f5925d791301b74ec6f931fc15f8e627743ae32122597d8660870"}
Jul 02 21:21:27.822185 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:27.821733    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae","Type":"ContainerStarted","Data":"04aa5d964197366aa761309ad9bebe6dba47f4c8a6a0625951ced57b9e48f85d"}
Jul 02 21:21:27.822185 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:27.821761    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae","Type":"ContainerStarted","Data":"828dc2195e6e9b6372c38dafa4fe790f3b8227676e8be17215ef560327b6a838"}
Jul 02 21:21:27.822185 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:27.821787    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae","Type":"ContainerStarted","Data":"89583e734e5863dab3c35728fd65677115e4ab850fac7eab5138a87b639010f2"}
Jul 02 21:21:27.822185 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:27.821810    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae","Type":"ContainerStarted","Data":"02cc9d3f6e534cd84b785f0cdf5782e33f698f9fca9d804751bb500153a617d9"}
Jul 02 21:21:27.822185 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:27.821832    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae","Type":"ContainerStarted","Data":"4d52ddc2fedf8ab8d81103624f1abd08d920712fcaf204328148814980082f40"}
Jul 02 21:21:27.857834 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:27.857774    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.842265459 podStartE2EDuration="2.857757108s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.491649ms" isStatefulPod=false podCreationTimestamp="2026-07-02 21:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:21:27.857478491 +0000 UTC m=+148.694718539" watchObservedRunningTime="2026-07-02 21:21:27.857757108 +0000 UTC m=+148.694997154"
Jul 02 21:21:28.257672 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.257632    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 21:21:28.257949 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.257912    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerName="prometheus" containerID="cri-o://d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865" gracePeriod=600
Jul 02 21:21:28.258095 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.257975    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerName="thanos-sidecar" containerID="cri-o://34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291" gracePeriod=600
Jul 02 21:21:28.258095 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.257956    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerName="kube-rbac-proxy" containerID="cri-o://d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f" gracePeriod=600
Jul 02 21:21:28.258095 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.258016    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerName="kube-rbac-proxy-thanos" containerID="cri-o://47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec" gracePeriod=600
Jul 02 21:21:28.258095 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.258000    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerName="kube-rbac-proxy-web" containerID="cri-o://359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491" gracePeriod=600
Jul 02 21:21:28.258095 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.257982    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerName="config-reloader" containerID="cri-o://9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b" gracePeriod=600
Jul 02 21:21:28.511781 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.511720    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:28.693370 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.693326    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") "
Jul 02 21:21:28.693370 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.693381    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-grpc-tls\") pod \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") "
Jul 02 21:21:28.693635 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.693404    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-configmap-kubelet-serving-ca-bundle\") pod \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") "
Jul 02 21:21:28.693635 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.693434    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-kube-rbac-proxy\") pod \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") "
Jul 02 21:21:28.693635 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.693456    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-thanos-prometheus-http-client-file\") pod \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") "
Jul 02 21:21:28.693635 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.693477    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-config\") pod \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") "
Jul 02 21:21:28.693635 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.693497    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-config-out\") pod \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") "
Jul 02 21:21:28.693635 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.693519    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-k8s-rulefiles-1\") pod \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") "
Jul 02 21:21:28.693935 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.693903    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" (UID: "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:21:28.693987 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.693959    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-web-config\") pod \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") "
Jul 02 21:21:28.694036 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.694010    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-prometheus-k8s-tls\") pod \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") "
Jul 02 21:21:28.694091 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.694049    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-configmap-metrics-client-ca\") pod \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") "
Jul 02 21:21:28.694091 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.694082    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-tls-assets\") pod \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") "
Jul 02 21:21:28.694225 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.694152    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-k8s-db\") pod \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") "
Jul 02 21:21:28.694225 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.694165    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" (UID: "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:21:28.694225 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.694188    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-trusted-ca-bundle\") pod \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") "
Jul 02 21:21:28.694371 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.694227    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-k8s-rulefiles-0\") pod \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") "
Jul 02 21:21:28.694371 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.694260    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") "
Jul 02 21:21:28.694371 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.694289    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-metrics-client-certs\") pod \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") "
Jul 02 21:21:28.694371 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.694320    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-configmap-serving-certs-ca-bundle\") pod \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") "
Jul 02 21:21:28.694371 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.694358    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-k8s-rulefiles-2\") pod \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") "
Jul 02 21:21:28.694614 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.694384    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2dk9\" (UniqueName: \"kubernetes.io/projected/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-kube-api-access-z2dk9\") pod \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\" (UID: \"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18\") "
Jul 02 21:21:28.694614 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.694444    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" (UID: "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:21:28.694710 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.694684    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:28.694760 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.694707    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:28.694760 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.694723    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-configmap-metrics-client-ca\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:28.696336 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.696198    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" (UID: "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:28.696336 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.696251    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-config-out" (OuterVolumeSpecName: "config-out") pod "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" (UID: "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:21:28.696336 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.696319    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" (UID: "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:28.696556 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.696459    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" (UID: "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:28.696615 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.696577    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-config" (OuterVolumeSpecName: "config") pod "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" (UID: "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:28.697376 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.696750    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-kube-api-access-z2dk9" (OuterVolumeSpecName: "kube-api-access-z2dk9") pod "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" (UID: "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18"). InnerVolumeSpecName "kube-api-access-z2dk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:21:28.697376 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.697045    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" (UID: "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:21:28.697376 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.697084    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" (UID: "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:21:28.697376 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.697104    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" (UID: "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:28.697613 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.697471    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" (UID: "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:21:28.697826 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.697790    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" (UID: "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:21:28.697923 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.697823    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" (UID: "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:21:28.698554 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.698525    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" (UID: "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:28.698834 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.698808    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" (UID: "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:28.699351 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.699325    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" (UID: "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:21:28.699607 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.699588    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" (UID: "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:28.707848 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.707820    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-web-config" (OuterVolumeSpecName: "web-config") pod "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" (UID: "4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:28.795535 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.795445    2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-web-config\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:28.795535 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.795477    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:28.795535 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.795487    2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-tls-assets\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:28.795535 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.795496    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-k8s-db\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:28.795535 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.795505    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:28.795535 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.795514    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:28.795535 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.795523    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:28.795535 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.795532    2564 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-metrics-client-certs\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:28.795535 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.795543    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:28.795914 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.795551    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:28.795914 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.795559    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2dk9\" (UniqueName: \"kubernetes.io/projected/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-kube-api-access-z2dk9\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:28.795914 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.795568    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:28.795914 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.795577    2564 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-grpc-tls\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:28.795914 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.795585    2564 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-secret-kube-rbac-proxy\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:28.795914 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.795593    2564 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:28.795914 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.795601    2564 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-config\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:28.795914 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.795608    2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18-config-out\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:21:28.828954 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.828922    2564 generic.go:356] "Generic (PLEG): container finished" podID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerID="47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec" exitCode=0
Jul 02 21:21:28.828954 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.828944    2564 generic.go:356] "Generic (PLEG): container finished" podID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerID="d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f" exitCode=0
Jul 02 21:21:28.828954 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.828952    2564 generic.go:356] "Generic (PLEG): container finished" podID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerID="359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491" exitCode=0
Jul 02 21:21:28.828954 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.828959    2564 generic.go:356] "Generic (PLEG): container finished" podID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerID="34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291" exitCode=0
Jul 02 21:21:28.829562 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.828966    2564 generic.go:356] "Generic (PLEG): container finished" podID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerID="9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b" exitCode=0
Jul 02 21:21:28.829562 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.828974    2564 generic.go:356] "Generic (PLEG): container finished" podID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerID="d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865" exitCode=0
Jul 02 21:21:28.829562 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.829017    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:28.829562 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.829016    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18","Type":"ContainerDied","Data":"47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec"}
Jul 02 21:21:28.829562 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.829105    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18","Type":"ContainerDied","Data":"d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f"}
Jul 02 21:21:28.829562 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.829137    2564 scope.go:117] "RemoveContainer" containerID="47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec"
Jul 02 21:21:28.829562 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.829168    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18","Type":"ContainerDied","Data":"359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491"}
Jul 02 21:21:28.829562 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.829212    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18","Type":"ContainerDied","Data":"34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291"}
Jul 02 21:21:28.829562 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.829254    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18","Type":"ContainerDied","Data":"9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b"}
Jul 02 21:21:28.829562 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.829298    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18","Type":"ContainerDied","Data":"d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865"}
Jul 02 21:21:28.829562 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.829340    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18","Type":"ContainerDied","Data":"d62a55b6428629d3bb817edb136929d9144a8d9d4cbcd2d52edd6d179ae966ec"}
Jul 02 21:21:28.837171 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.836962    2564 scope.go:117] "RemoveContainer" containerID="d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f"
Jul 02 21:21:28.843970 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.843951    2564 scope.go:117] "RemoveContainer" containerID="359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491"
Jul 02 21:21:28.851332 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.851239    2564 scope.go:117] "RemoveContainer" containerID="34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291"
Jul 02 21:21:28.854035 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.854007    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 21:21:28.859031 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.859003    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 21:21:28.859429 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.859409    2564 scope.go:117] "RemoveContainer" containerID="9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b"
Jul 02 21:21:28.866501 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.866483    2564 scope.go:117] "RemoveContainer" containerID="d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865"
Jul 02 21:21:28.873535 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.873516    2564 scope.go:117] "RemoveContainer" containerID="7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b"
Jul 02 21:21:28.882133 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.882100    2564 scope.go:117] "RemoveContainer" containerID="47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec"
Jul 02 21:21:28.882644 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:21:28.882619    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec\": container with ID starting with 47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec not found: ID does not exist" containerID="47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec"
Jul 02 21:21:28.882734 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.882655    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec"} err="failed to get container status \"47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec\": rpc error: code = NotFound desc = could not find container \"47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec\": container with ID starting with 47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec not found: ID does not exist"
Jul 02 21:21:28.882734 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.882676    2564 scope.go:117] "RemoveContainer" containerID="d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f"
Jul 02 21:21:28.883002 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:21:28.882984    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f\": container with ID starting with d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f not found: ID does not exist" containerID="d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f"
Jul 02 21:21:28.883059 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.883013    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f"} err="failed to get container status \"d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f\": rpc error: code = NotFound desc = could not find container \"d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f\": container with ID starting with d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f not found: ID does not exist"
Jul 02 21:21:28.883059 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.883029    2564 scope.go:117] "RemoveContainer" containerID="359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491"
Jul 02 21:21:28.883387 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:21:28.883369    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491\": container with ID starting with 359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491 not found: ID does not exist" containerID="359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491"
Jul 02 21:21:28.883433 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.883390    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491"} err="failed to get container status \"359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491\": rpc error: code = NotFound desc = could not find container \"359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491\": container with ID starting with 359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491 not found: ID does not exist"
Jul 02 21:21:28.883433 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.883403    2564 scope.go:117] "RemoveContainer" containerID="34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291"
Jul 02 21:21:28.883635 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:21:28.883615    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291\": container with ID starting with 34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291 not found: ID does not exist" containerID="34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291"
Jul 02 21:21:28.883675 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.883644    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291"} err="failed to get container status \"34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291\": rpc error: code = NotFound desc = could not find container \"34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291\": container with ID starting with 34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291 not found: ID does not exist"
Jul 02 21:21:28.883675 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.883664    2564 scope.go:117] "RemoveContainer" containerID="9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b"
Jul 02 21:21:28.883902 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:21:28.883885    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b\": container with ID starting with 9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b not found: ID does not exist" containerID="9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b"
Jul 02 21:21:28.883941 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.883907    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b"} err="failed to get container status \"9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b\": rpc error: code = NotFound desc = could not find container \"9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b\": container with ID starting with 9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b not found: ID does not exist"
Jul 02 21:21:28.883941 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.883924    2564 scope.go:117] "RemoveContainer" containerID="d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865"
Jul 02 21:21:28.884154 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:21:28.884131    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865\": container with ID starting with d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865 not found: ID does not exist" containerID="d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865"
Jul 02 21:21:28.884262 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.884157    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865"} err="failed to get container status \"d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865\": rpc error: code = NotFound desc = could not find container \"d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865\": container with ID starting with d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865 not found: ID does not exist"
Jul 02 21:21:28.884262 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.884175    2564 scope.go:117] "RemoveContainer" containerID="7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b"
Jul 02 21:21:28.884532 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:21:28.884489    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b\": container with ID starting with 7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b not found: ID does not exist" containerID="7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b"
Jul 02 21:21:28.884532 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.884524    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b"} err="failed to get container status \"7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b\": rpc error: code = NotFound desc = could not find container \"7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b\": container with ID starting with 7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b not found: ID does not exist"
Jul 02 21:21:28.884680 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.884544    2564 scope.go:117] "RemoveContainer" containerID="47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec"
Jul 02 21:21:28.884953 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.884927    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec"} err="failed to get container status \"47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec\": rpc error: code = NotFound desc = could not find container \"47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec\": container with ID starting with 47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec not found: ID does not exist"
Jul 02 21:21:28.884953 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.884952    2564 scope.go:117] "RemoveContainer" containerID="d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f"
Jul 02 21:21:28.885347 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.885274    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f"} err="failed to get container status \"d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f\": rpc error: code = NotFound desc = could not find container \"d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f\": container with ID starting with d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f not found: ID does not exist"
Jul 02 21:21:28.885347 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.885301    2564 scope.go:117] "RemoveContainer" containerID="359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491"
Jul 02 21:21:28.885662 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.885631    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491"} err="failed to get container status \"359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491\": rpc error: code = NotFound desc = could not find container \"359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491\": container with ID starting with 359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491 not found: ID does not exist"
Jul 02 21:21:28.885662 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.885656    2564 scope.go:117] "RemoveContainer" containerID="34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291"
Jul 02 21:21:28.885906 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.885890    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291"} err="failed to get container status \"34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291\": rpc error: code = NotFound desc = could not find container \"34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291\": container with ID starting with 34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291 not found: ID does not exist"
Jul 02 21:21:28.885958 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.885906    2564 scope.go:117] "RemoveContainer" containerID="9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b"
Jul 02 21:21:28.886104 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.886075    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b"} err="failed to get container status \"9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b\": rpc error: code = NotFound desc = could not find container \"9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b\": container with ID starting with 9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b not found: ID does not exist"
Jul 02 21:21:28.886104 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.886096    2564 scope.go:117] "RemoveContainer" containerID="d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865"
Jul 02 21:21:28.886354 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.886337    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865"} err="failed to get container status \"d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865\": rpc error: code = NotFound desc = could not find container \"d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865\": container with ID starting with d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865 not found: ID does not exist"
Jul 02 21:21:28.886354 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.886354    2564 scope.go:117] "RemoveContainer" containerID="7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b"
Jul 02 21:21:28.886564 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.886545    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b"} err="failed to get container status \"7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b\": rpc error: code = NotFound desc = could not find container \"7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b\": container with ID starting with 7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b not found: ID does not exist"
Jul 02 21:21:28.886564 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.886564    2564 scope.go:117] "RemoveContainer" containerID="47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec"
Jul 02 21:21:28.886712 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.886696    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 21:21:28.886752 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.886723    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec"} err="failed to get container status \"47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec\": rpc error: code = NotFound desc = could not find container \"47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec\": container with ID starting with 47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec not found: ID does not exist"
Jul 02 21:21:28.886752 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.886738    2564 scope.go:117] "RemoveContainer" containerID="d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f"
Jul 02 21:21:28.886934 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.886917    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f"} err="failed to get container status \"d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f\": rpc error: code = NotFound desc = could not find container \"d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f\": container with ID starting with d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f not found: ID does not exist"
Jul 02 21:21:28.886979 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.886934    2564 scope.go:117] "RemoveContainer" containerID="359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491"
Jul 02 21:21:28.887023 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.886990    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerName="kube-rbac-proxy"
Jul 02 21:21:28.887023 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.887001    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerName="kube-rbac-proxy"
Jul 02 21:21:28.887023 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.887021    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerName="config-reloader"
Jul 02 21:21:28.887167 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.887030    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerName="config-reloader"
Jul 02 21:21:28.887167 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.887039    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerName="kube-rbac-proxy-thanos"
Jul 02 21:21:28.887167 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.887048    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerName="kube-rbac-proxy-thanos"
Jul 02 21:21:28.887167 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.887066    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerName="prometheus"
Jul 02 21:21:28.887167 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.887074    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerName="prometheus"
Jul 02 21:21:28.887167 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.887083    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerName="thanos-sidecar"
Jul 02 21:21:28.887167 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.887091    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerName="thanos-sidecar"
Jul 02 21:21:28.887167 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.887099    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerName="init-config-reloader"
Jul 02 21:21:28.887167 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.887105    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerName="init-config-reloader"
Jul 02 21:21:28.887167 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.887129    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerName="kube-rbac-proxy-web"
Jul 02 21:21:28.887167 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.887138    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerName="kube-rbac-proxy-web"
Jul 02 21:21:28.887527 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.887167    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491"} err="failed to get container status \"359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491\": rpc error: code = NotFound desc = could not find container \"359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491\": container with ID starting with 359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491 not found: ID does not exist"
Jul 02 21:21:28.887527 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.887187    2564 scope.go:117] "RemoveContainer" containerID="34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291"
Jul 02 21:21:28.887527 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.887191    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerName="kube-rbac-proxy"
Jul 02 21:21:28.887527 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.887202    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerName="thanos-sidecar"
Jul 02 21:21:28.887527 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.887208    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerName="kube-rbac-proxy-thanos"
Jul 02 21:21:28.887527 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.887214    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerName="kube-rbac-proxy-web"
Jul 02 21:21:28.887527 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.887221    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerName="prometheus"
Jul 02 21:21:28.887527 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.887226    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" containerName="config-reloader"
Jul 02 21:21:28.887527 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.887400    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291"} err="failed to get container status \"34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291\": rpc error: code = NotFound desc = could not find container \"34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291\": container with ID starting with 34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291 not found: ID does not exist"
Jul 02 21:21:28.887527 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.887413    2564 scope.go:117] "RemoveContainer" containerID="9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b"
Jul 02 21:21:28.887873 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.887636    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b"} err="failed to get container status \"9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b\": rpc error: code = NotFound desc = could not find container \"9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b\": container with ID starting with 9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b not found: ID does not exist"
Jul 02 21:21:28.887873 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.887651    2564 scope.go:117] "RemoveContainer" containerID="d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865"
Jul 02 21:21:28.887873 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.887834    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865"} err="failed to get container status \"d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865\": rpc error: code = NotFound desc = could not find container \"d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865\": container with ID starting with d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865 not found: ID does not exist"
Jul 02 21:21:28.887873 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.887848    2564 scope.go:117] "RemoveContainer" containerID="7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b"
Jul 02 21:21:28.888052 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.888036    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b"} err="failed to get container status \"7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b\": rpc error: code = NotFound desc = could not find container \"7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b\": container with ID starting with 7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b not found: ID does not exist"
Jul 02 21:21:28.888092 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.888052    2564 scope.go:117] "RemoveContainer" containerID="47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec"
Jul 02 21:21:28.888302 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.888285    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec"} err="failed to get container status \"47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec\": rpc error: code = NotFound desc = could not find container \"47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec\": container with ID starting with 47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec not found: ID does not exist"
Jul 02 21:21:28.888343 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.888303    2564 scope.go:117] "RemoveContainer" containerID="d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f"
Jul 02 21:21:28.888527 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.888510    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f"} err="failed to get container status \"d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f\": rpc error: code = NotFound desc = could not find container \"d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f\": container with ID starting with d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f not found: ID does not exist"
Jul 02 21:21:28.888578 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.888526    2564 scope.go:117] "RemoveContainer" containerID="359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491"
Jul 02 21:21:28.888736 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.888720    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491"} err="failed to get container status \"359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491\": rpc error: code = NotFound desc = could not find container \"359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491\": container with ID starting with 359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491 not found: ID does not exist"
Jul 02 21:21:28.888791 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.888738    2564 scope.go:117] "RemoveContainer" containerID="34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291"
Jul 02 21:21:28.888903 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.888886    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291"} err="failed to get container status \"34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291\": rpc error: code = NotFound desc = could not find container \"34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291\": container with ID starting with 34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291 not found: ID does not exist"
Jul 02 21:21:28.888939 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.888904    2564 scope.go:117] "RemoveContainer" containerID="9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b"
Jul 02 21:21:28.889104 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.889081    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b"} err="failed to get container status \"9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b\": rpc error: code = NotFound desc = could not find container \"9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b\": container with ID starting with 9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b not found: ID does not exist"
Jul 02 21:21:28.889104 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.889103    2564 scope.go:117] "RemoveContainer" containerID="d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865"
Jul 02 21:21:28.889386 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.889359    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865"} err="failed to get container status \"d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865\": rpc error: code = NotFound desc = could not find container \"d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865\": container with ID starting with d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865 not found: ID does not exist"
Jul 02 21:21:28.889488 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.889388    2564 scope.go:117] "RemoveContainer" containerID="7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b"
Jul 02 21:21:28.889688 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.889665    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b"} err="failed to get container status \"7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b\": rpc error: code = NotFound desc = could not find container \"7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b\": container with ID starting with 7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b not found: ID does not exist"
Jul 02 21:21:28.889781 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.889690    2564 scope.go:117] "RemoveContainer" containerID="47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec"
Jul 02 21:21:28.889981 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.889958    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec"} err="failed to get container status \"47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec\": rpc error: code = NotFound desc = could not find container \"47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec\": container with ID starting with 47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec not found: ID does not exist"
Jul 02 21:21:28.890064 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.889982    2564 scope.go:117] "RemoveContainer" containerID="d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f"
Jul 02 21:21:28.890276 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.890250    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f"} err="failed to get container status \"d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f\": rpc error: code = NotFound desc = could not find container \"d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f\": container with ID starting with d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f not found: ID does not exist"
Jul 02 21:21:28.890276 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.890276    2564 scope.go:117] "RemoveContainer" containerID="359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491"
Jul 02 21:21:28.892273 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.890525    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491"} err="failed to get container status \"359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491\": rpc error: code = NotFound desc = could not find container \"359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491\": container with ID starting with 359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491 not found: ID does not exist"
Jul 02 21:21:28.892273 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.890546    2564 scope.go:117] "RemoveContainer" containerID="34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291"
Jul 02 21:21:28.892273 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.890766    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291"} err="failed to get container status \"34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291\": rpc error: code = NotFound desc = could not find container \"34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291\": container with ID starting with 34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291 not found: ID does not exist"
Jul 02 21:21:28.892273 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.890785    2564 scope.go:117] "RemoveContainer" containerID="9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b"
Jul 02 21:21:28.892273 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.890965    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b"} err="failed to get container status \"9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b\": rpc error: code = NotFound desc = could not find container \"9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b\": container with ID starting with 9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b not found: ID does not exist"
Jul 02 21:21:28.892273 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.890983    2564 scope.go:117] "RemoveContainer" containerID="d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865"
Jul 02 21:21:28.892273 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.891198    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865"} err="failed to get container status \"d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865\": rpc error: code = NotFound desc = could not find container \"d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865\": container with ID starting with d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865 not found: ID does not exist"
Jul 02 21:21:28.892273 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.891217    2564 scope.go:117] "RemoveContainer" containerID="7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b"
Jul 02 21:21:28.892273 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.891396    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b"} err="failed to get container status \"7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b\": rpc error: code = NotFound desc = could not find container \"7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b\": container with ID starting with 7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b not found: ID does not exist"
Jul 02 21:21:28.892273 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.891415    2564 scope.go:117] "RemoveContainer" containerID="47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec"
Jul 02 21:21:28.892273 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.891585    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec"} err="failed to get container status \"47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec\": rpc error: code = NotFound desc = could not find container \"47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec\": container with ID starting with 47c9bd13bcb6370b06f31732c89fb4300e1816d2aa9c4317b5389f26357b59ec not found: ID does not exist"
Jul 02 21:21:28.892273 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.891605    2564 scope.go:117] "RemoveContainer" containerID="d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f"
Jul 02 21:21:28.892273 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.891907    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f"} err="failed to get container status \"d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f\": rpc error: code = NotFound desc = could not find container \"d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f\": container with ID starting with d831ddc05fe83984a4eb17d6f31a9d9a65db9571ac486c586319b3dd6b68239f not found: ID does not exist"
Jul 02 21:21:28.892273 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.891926    2564 scope.go:117] "RemoveContainer" containerID="359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491"
Jul 02 21:21:28.892273 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.892159    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491"} err="failed to get container status \"359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491\": rpc error: code = NotFound desc = could not find container \"359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491\": container with ID starting with 359938ffc541aad07c4ca57bf578374b544a2a9023664b8b6b06643039a67491 not found: ID does not exist"
Jul 02 21:21:28.892273 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.892179    2564 scope.go:117] "RemoveContainer" containerID="34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291"
Jul 02 21:21:28.892809 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.892423    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291"} err="failed to get container status \"34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291\": rpc error: code = NotFound desc = could not find container \"34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291\": container with ID starting with 34415b1124c69c8dbc7822cdb488e1a1b4b9303d9699f1589b434eaccada9291 not found: ID does not exist"
Jul 02 21:21:28.892809 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.892439    2564 scope.go:117] "RemoveContainer" containerID="9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b"
Jul 02 21:21:28.892809 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.892613    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b"} err="failed to get container status \"9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b\": rpc error: code = NotFound desc = could not find container \"9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b\": container with ID starting with 9f5159b8e9ed98c216d92f584b8d966a67094e66afa99135b94bb6259bc7ea9b not found: ID does not exist"
Jul 02 21:21:28.892809 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.892629    2564 scope.go:117] "RemoveContainer" containerID="d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865"
Jul 02 21:21:28.892922 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.892848    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865"} err="failed to get container status \"d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865\": rpc error: code = NotFound desc = could not find container \"d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865\": container with ID starting with d445ef6eb08ac10437007e137539d4e88f29a04796682bae9bf7e3d20b0ec865 not found: ID does not exist"
Jul 02 21:21:28.892922 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.892866    2564 scope.go:117] "RemoveContainer" containerID="7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b"
Jul 02 21:21:28.893067 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.893050    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b"} err="failed to get container status \"7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b\": rpc error: code = NotFound desc = could not find container \"7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b\": container with ID starting with 7a206e2015581d87bd70fa7c0163cd01b585446695e250afed21378b53206c2b not found: ID does not exist"
Jul 02 21:21:28.893469 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.893454    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:28.896086 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.896066    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 02 21:21:28.896086 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.896081    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 02 21:21:28.896252 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.896081    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 02 21:21:28.896252 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.896141    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 02 21:21:28.896348 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.896272    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 02 21:21:28.896409 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.896393    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 02 21:21:28.896486 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.896470    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 02 21:21:28.896601 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.896585    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 02 21:21:28.896660 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.896596    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 02 21:21:28.896660 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.896593    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 02 21:21:28.896660 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.896652    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 02 21:21:28.897249 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.897231    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 02 21:21:28.897350 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.897336    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cv89tloccb7c6\""
Jul 02 21:21:28.897599 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.897583    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 02 21:21:28.897599 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.897596    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8vhlb\""
Jul 02 21:21:28.899608 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.899587    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 02 21:21:28.901764 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.901740    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 02 21:21:28.909530 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.909507    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 21:21:28.997009 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.996966    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:28.997009 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.997008    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:28.997252 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.997030    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq6gx\" (UniqueName: \"kubernetes.io/projected/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-kube-api-access-xq6gx\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:28.997252 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.997108    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:28.997252 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.997156    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:28.997252 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.997178    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:28.997252 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.997196    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:28.997400 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.997256    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:28.997400 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.997311    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-config-out\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:28.997400 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.997344    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:28.997482 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.997391    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-config\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:28.997482 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.997452    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:28.997571 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.997550    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:28.997603 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.997591    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:28.997646 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.997632    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-web-config\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:28.997725 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.997711    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:28.997761 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.997747    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:28.998020 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.998003    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:28.998060 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.998043    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:28.998091 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:28.998076    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.099206 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.099106    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.099206 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.099174    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.099206 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.099202    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.099395 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.099219    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.099395 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.099236    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.099395 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.099256    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.099545 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.099425    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.099545 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.099468    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq6gx\" (UniqueName: \"kubernetes.io/projected/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-kube-api-access-xq6gx\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.099545 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.099506    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.099545 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.099533    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.099746 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.099562    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.099746 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.099585    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.099746 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.099629    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.099746 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.099657    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-config-out\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.099746 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.099694    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.099746 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.099719    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-config\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.099746 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.099743    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.100066 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.099781    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.100066 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.099813    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.100066 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.099843    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-web-config\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.100250 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.100161    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.100690 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.100662    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.102374 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.102138    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.102374 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.102269    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.102541 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.102393    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.102650 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.102627    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-web-config\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.102849 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.102823    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.103011 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.102920    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.103864 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.103436    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.103864 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.103512    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.103864 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.103531    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.103864 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.103557    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.103864 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.103810    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.104188 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.104062    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.104188 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.104152    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.104916 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.104894    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.105022 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.105004    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-config\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.105073 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.105055    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-config-out\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.105162 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.105145    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.110232 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.110215    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq6gx\" (UniqueName: \"kubernetes.io/projected/cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f-kube-api-access-xq6gx\") pod \"prometheus-k8s-0\" (UID: \"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.204300 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.204266    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:21:29.345617 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.345554    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 21:21:29.348022 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:21:29.347995    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc8aee9_2e8e_497c_8bf0_e3c5b3bcf85f.slice/crio-7d2d2649210a635b2933914883aa0fceec59238b7540a06132d9a83d97f81c69 WatchSource:0}: Error finding container 7d2d2649210a635b2933914883aa0fceec59238b7540a06132d9a83d97f81c69: Status 404 returned error can't find the container with id 7d2d2649210a635b2933914883aa0fceec59238b7540a06132d9a83d97f81c69
Jul 02 21:21:29.835053 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.835017    2564 generic.go:356] "Generic (PLEG): container finished" podID="cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f" containerID="3f067a96dd6b1fff3af9bf9d03172511a2fea3c723653e429ef8f75f3c088b3c" exitCode=0
Jul 02 21:21:29.835524 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.835122    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f","Type":"ContainerDied","Data":"3f067a96dd6b1fff3af9bf9d03172511a2fea3c723653e429ef8f75f3c088b3c"}
Jul 02 21:21:29.835524 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:29.835192    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f","Type":"ContainerStarted","Data":"7d2d2649210a635b2933914883aa0fceec59238b7540a06132d9a83d97f81c69"}
Jul 02 21:21:30.074886 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:30.074847    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18" path="/var/lib/kubelet/pods/4a2ed5a9-3ec9-4f8f-a3ba-3858705e3f18/volumes"
Jul 02 21:21:30.843953 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:30.843916    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f","Type":"ContainerStarted","Data":"66b393ad5f74e044e0af31e5f39db8e968438d25890d4634469c79044adb94bb"}
Jul 02 21:21:30.844343 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:30.844012    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f","Type":"ContainerStarted","Data":"3573656aa3e18e16077e2e1bd2b1191017d8fa81b4299ce48dc378479a93d89a"}
Jul 02 21:21:30.844343 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:30.844040    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f","Type":"ContainerStarted","Data":"9658b0cefbe39eda35d4ff4f4ec0910a57ae59b2a9397424069e0aca0853acad"}
Jul 02 21:21:30.844343 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:30.844066    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f","Type":"ContainerStarted","Data":"778875ec46ca97930070eacd65e816c71006bf80890e3124d1920877973f3aa1"}
Jul 02 21:21:30.844343 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:30.844089    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f","Type":"ContainerStarted","Data":"34ba63487d6504b64a643e76c4355ac1be5abddb50f2a64a3b40413cf01e40a1"}
Jul 02 21:21:30.844343 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:30.844132    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f","Type":"ContainerStarted","Data":"52236880e5954a7458a0a7305524b0a8197c623173cf2f4a4cea98cfa4ee411b"}
Jul 02 21:21:30.873876 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:30.873808    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.8549381780000003 podStartE2EDuration="2.873784469s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.846291ms" isStatefulPod=false podCreationTimestamp="2026-07-02 21:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:21:30.872332116 +0000 UTC m=+151.709572164" watchObservedRunningTime="2026-07-02 21:21:30.873784469 +0000 UTC m=+151.711024519"
Jul 02 21:21:34.204505 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:21:34.204459    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:22:29.204883 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:22:29.204839    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:22:29.220108 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:22:29.220084    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:22:30.096756 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:22:30.096725    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 21:23:59.882107 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:23:59.882079    2564 kubelet.go:1648] "Image garbage collection succeeded"
Jul 02 21:26:26.992168 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:26:26.992077    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-dv69s"]
Jul 02 21:26:26.995352 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:26:26.995334    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-dv69s"
Jul 02 21:26:27.007840 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:26:27.007813    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-dv69s"]
Jul 02 21:26:27.024837 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:26:27.024815    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 02 21:26:27.025179 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:26:27.025164    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 02 21:26:27.025250 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:26:27.025229    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-4l2jm\""
Jul 02 21:26:27.025289 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:26:27.025254    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 02 21:26:27.033197 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:26:27.033175    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35aebdb0-c85d-4cb4-ae73-0451f2992d50-cert\") pod \"llmisvc-controller-manager-67d95fdc48-dv69s\" (UID: \"35aebdb0-c85d-4cb4-ae73-0451f2992d50\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-dv69s"
Jul 02 21:26:27.033300 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:26:27.033222    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzz6n\" (UniqueName: \"kubernetes.io/projected/35aebdb0-c85d-4cb4-ae73-0451f2992d50-kube-api-access-bzz6n\") pod \"llmisvc-controller-manager-67d95fdc48-dv69s\" (UID: \"35aebdb0-c85d-4cb4-ae73-0451f2992d50\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-dv69s"
Jul 02 21:26:27.134285 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:26:27.134255    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35aebdb0-c85d-4cb4-ae73-0451f2992d50-cert\") pod \"llmisvc-controller-manager-67d95fdc48-dv69s\" (UID: \"35aebdb0-c85d-4cb4-ae73-0451f2992d50\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-dv69s"
Jul 02 21:26:27.134480 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:26:27.134319    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzz6n\" (UniqueName: \"kubernetes.io/projected/35aebdb0-c85d-4cb4-ae73-0451f2992d50-kube-api-access-bzz6n\") pod \"llmisvc-controller-manager-67d95fdc48-dv69s\" (UID: \"35aebdb0-c85d-4cb4-ae73-0451f2992d50\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-dv69s"
Jul 02 21:26:27.134480 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:26:27.134347    2564 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 02 21:26:27.134480 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:26:27.134435    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35aebdb0-c85d-4cb4-ae73-0451f2992d50-cert podName:35aebdb0-c85d-4cb4-ae73-0451f2992d50 nodeName:}" failed. No retries permitted until 2026-07-02 21:26:27.634410259 +0000 UTC m=+448.471650288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/35aebdb0-c85d-4cb4-ae73-0451f2992d50-cert") pod "llmisvc-controller-manager-67d95fdc48-dv69s" (UID: "35aebdb0-c85d-4cb4-ae73-0451f2992d50") : secret "llmisvc-webhook-server-cert" not found
Jul 02 21:26:27.144669 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:26:27.144645    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzz6n\" (UniqueName: \"kubernetes.io/projected/35aebdb0-c85d-4cb4-ae73-0451f2992d50-kube-api-access-bzz6n\") pod \"llmisvc-controller-manager-67d95fdc48-dv69s\" (UID: \"35aebdb0-c85d-4cb4-ae73-0451f2992d50\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-dv69s"
Jul 02 21:26:27.639602 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:26:27.639564    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35aebdb0-c85d-4cb4-ae73-0451f2992d50-cert\") pod \"llmisvc-controller-manager-67d95fdc48-dv69s\" (UID: \"35aebdb0-c85d-4cb4-ae73-0451f2992d50\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-dv69s"
Jul 02 21:26:27.641926 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:26:27.641895    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35aebdb0-c85d-4cb4-ae73-0451f2992d50-cert\") pod \"llmisvc-controller-manager-67d95fdc48-dv69s\" (UID: \"35aebdb0-c85d-4cb4-ae73-0451f2992d50\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-dv69s"
Jul 02 21:26:27.904897 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:26:27.904792    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-dv69s"
Jul 02 21:26:28.028863 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:26:28.028839    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-dv69s"]
Jul 02 21:26:28.030976 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:26:28.030951    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod35aebdb0_c85d_4cb4_ae73_0451f2992d50.slice/crio-0bde2baa6b54d1b78e2813501232d3f93350f7e48fbf745cdc637b1fecea55b9 WatchSource:0}: Error finding container 0bde2baa6b54d1b78e2813501232d3f93350f7e48fbf745cdc637b1fecea55b9: Status 404 returned error can't find the container with id 0bde2baa6b54d1b78e2813501232d3f93350f7e48fbf745cdc637b1fecea55b9
Jul 02 21:26:28.032261 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:26:28.032243    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 21:26:28.991448 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:26:28.991394    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-dv69s" event={"ID":"35aebdb0-c85d-4cb4-ae73-0451f2992d50","Type":"ContainerStarted","Data":"0bde2baa6b54d1b78e2813501232d3f93350f7e48fbf745cdc637b1fecea55b9"}
Jul 02 21:26:31.001924 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:26:31.001887    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-dv69s" event={"ID":"35aebdb0-c85d-4cb4-ae73-0451f2992d50","Type":"ContainerStarted","Data":"94dfc91f98eb251e1885e8adfc67810725eaf64020316cddd6739a4539180415"}
Jul 02 21:26:31.002310 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:26:31.002024    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-dv69s"
Jul 02 21:26:31.020952 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:26:31.020901    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-67d95fdc48-dv69s" podStartSLOduration=3.144214962 podStartE2EDuration="5.020885579s" totalImagesPullingTime="1.876670617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:26:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:26:31.019768523 +0000 UTC m=+451.857008586" watchObservedRunningTime="2026-07-02 21:26:31.020885579 +0000 UTC m=+451.858125627"
Jul 02 21:27:02.008277 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:27:02.008246    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-dv69s"
Jul 02 21:27:52.912435 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:27:52.912359    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-2lrcz"]
Jul 02 21:27:52.914448 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:27:52.914431    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2lrcz"
Jul 02 21:27:52.917782 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:27:52.917758    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-mc9tx\""
Jul 02 21:27:52.917926 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:27:52.917843    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 02 21:27:52.926709 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:27:52.926686    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-2lrcz"]
Jul 02 21:27:52.975552 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:27:52.975526    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djwsh\" (UniqueName: \"kubernetes.io/projected/cf41621d-1fee-44e5-bd4f-0a077bf581fb-kube-api-access-djwsh\") pod \"s3-init-2lrcz\" (UID: \"cf41621d-1fee-44e5-bd4f-0a077bf581fb\") " pod="kserve/s3-init-2lrcz"
Jul 02 21:27:52.975689 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:27:52.975572    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf41621d-1fee-44e5-bd4f-0a077bf581fb-model-cache\") pod \"s3-init-2lrcz\" (UID: \"cf41621d-1fee-44e5-bd4f-0a077bf581fb\") " pod="kserve/s3-init-2lrcz"
Jul 02 21:27:53.076622 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:27:53.076595    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djwsh\" (UniqueName: \"kubernetes.io/projected/cf41621d-1fee-44e5-bd4f-0a077bf581fb-kube-api-access-djwsh\") pod \"s3-init-2lrcz\" (UID: \"cf41621d-1fee-44e5-bd4f-0a077bf581fb\") " pod="kserve/s3-init-2lrcz"
Jul 02 21:27:53.076770 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:27:53.076641    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf41621d-1fee-44e5-bd4f-0a077bf581fb-model-cache\") pod \"s3-init-2lrcz\" (UID: \"cf41621d-1fee-44e5-bd4f-0a077bf581fb\") " pod="kserve/s3-init-2lrcz"
Jul 02 21:27:53.077052 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:27:53.077034    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf41621d-1fee-44e5-bd4f-0a077bf581fb-model-cache\") pod \"s3-init-2lrcz\" (UID: \"cf41621d-1fee-44e5-bd4f-0a077bf581fb\") " pod="kserve/s3-init-2lrcz"
Jul 02 21:27:53.085833 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:27:53.085809    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djwsh\" (UniqueName: \"kubernetes.io/projected/cf41621d-1fee-44e5-bd4f-0a077bf581fb-kube-api-access-djwsh\") pod \"s3-init-2lrcz\" (UID: \"cf41621d-1fee-44e5-bd4f-0a077bf581fb\") " pod="kserve/s3-init-2lrcz"
Jul 02 21:27:53.234736 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:27:53.234711    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2lrcz"
Jul 02 21:27:53.364959 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:27:53.364936    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-2lrcz"]
Jul 02 21:27:53.366865 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:27:53.366835    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf41621d_1fee_44e5_bd4f_0a077bf581fb.slice/crio-562e03c9ce48f8701ad2790af5cdc69d7994c5f1b922a09cd0b642cca68345d8 WatchSource:0}: Error finding container 562e03c9ce48f8701ad2790af5cdc69d7994c5f1b922a09cd0b642cca68345d8: Status 404 returned error can't find the container with id 562e03c9ce48f8701ad2790af5cdc69d7994c5f1b922a09cd0b642cca68345d8
Jul 02 21:27:54.313575 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:27:54.313533    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2lrcz" event={"ID":"cf41621d-1fee-44e5-bd4f-0a077bf581fb","Type":"ContainerStarted","Data":"562e03c9ce48f8701ad2790af5cdc69d7994c5f1b922a09cd0b642cca68345d8"}
Jul 02 21:27:57.327400 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:27:57.327295    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2lrcz" event={"ID":"cf41621d-1fee-44e5-bd4f-0a077bf581fb","Type":"ContainerStarted","Data":"3d4a12f94996c5842b9fa266cd55d49ab38febd2b7e56dcbf11be062f16e0669"}
Jul 02 21:27:59.338357 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:27:59.338323    2564 generic.go:356] "Generic (PLEG): container finished" podID="cf41621d-1fee-44e5-bd4f-0a077bf581fb" containerID="3d4a12f94996c5842b9fa266cd55d49ab38febd2b7e56dcbf11be062f16e0669" exitCode=0
Jul 02 21:27:59.338720 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:27:59.338369    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2lrcz" event={"ID":"cf41621d-1fee-44e5-bd4f-0a077bf581fb","Type":"ContainerDied","Data":"3d4a12f94996c5842b9fa266cd55d49ab38febd2b7e56dcbf11be062f16e0669"}
Jul 02 21:28:03.782601 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:28:03.782581    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 02 21:28:04.363670 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:28:04.363636    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2lrcz" event={"ID":"cf41621d-1fee-44e5-bd4f-0a077bf581fb","Type":"ContainerStarted","Data":"d0164f7730e19f1e6f58938f6bc737f84de2d1c2b59606616b929e049f69553a"}
Jul 02 21:28:04.381957 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:28:04.381906    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-2lrcz" podStartSLOduration=2.648187971 podStartE2EDuration="12.381887842s" totalImagesPullingTime="8.090695264s" totalInitContainerRuntime="1.643004607s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:27:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:28:04.381078527 +0000 UTC m=+545.218318575" watchObservedRunningTime="2026-07-02 21:28:04.381887842 +0000 UTC m=+545.219127888"
Jul 02 21:29:09.624315 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:09.624267    2564 generic.go:356] "Generic (PLEG): container finished" podID="cf41621d-1fee-44e5-bd4f-0a077bf581fb" containerID="d0164f7730e19f1e6f58938f6bc737f84de2d1c2b59606616b929e049f69553a" exitCode=0
Jul 02 21:29:09.624737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:09.624335    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2lrcz" event={"ID":"cf41621d-1fee-44e5-bd4f-0a077bf581fb","Type":"ContainerDied","Data":"d0164f7730e19f1e6f58938f6bc737f84de2d1c2b59606616b929e049f69553a"}
Jul 02 21:29:10.746887 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:10.746864    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2lrcz"
Jul 02 21:29:10.826465 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:10.826435    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf41621d-1fee-44e5-bd4f-0a077bf581fb-model-cache\") pod \"cf41621d-1fee-44e5-bd4f-0a077bf581fb\" (UID: \"cf41621d-1fee-44e5-bd4f-0a077bf581fb\") "
Jul 02 21:29:10.826603 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:10.826536    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djwsh\" (UniqueName: \"kubernetes.io/projected/cf41621d-1fee-44e5-bd4f-0a077bf581fb-kube-api-access-djwsh\") pod \"cf41621d-1fee-44e5-bd4f-0a077bf581fb\" (UID: \"cf41621d-1fee-44e5-bd4f-0a077bf581fb\") "
Jul 02 21:29:10.828724 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:10.828702    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf41621d-1fee-44e5-bd4f-0a077bf581fb-kube-api-access-djwsh" (OuterVolumeSpecName: "kube-api-access-djwsh") pod "cf41621d-1fee-44e5-bd4f-0a077bf581fb" (UID: "cf41621d-1fee-44e5-bd4f-0a077bf581fb"). InnerVolumeSpecName "kube-api-access-djwsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:29:10.844748 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:10.844722    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf41621d-1fee-44e5-bd4f-0a077bf581fb-model-cache" (OuterVolumeSpecName: "model-cache") pod "cf41621d-1fee-44e5-bd4f-0a077bf581fb" (UID: "cf41621d-1fee-44e5-bd4f-0a077bf581fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:29:10.927467 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:10.927397    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djwsh\" (UniqueName: \"kubernetes.io/projected/cf41621d-1fee-44e5-bd4f-0a077bf581fb-kube-api-access-djwsh\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:29:10.927467 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:10.927425    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf41621d-1fee-44e5-bd4f-0a077bf581fb-model-cache\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:29:11.633857 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:11.633827    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2lrcz"
Jul 02 21:29:11.633857 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:11.633839    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2lrcz" event={"ID":"cf41621d-1fee-44e5-bd4f-0a077bf581fb","Type":"ContainerDied","Data":"562e03c9ce48f8701ad2790af5cdc69d7994c5f1b922a09cd0b642cca68345d8"}
Jul 02 21:29:11.634097 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:11.633869    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="562e03c9ce48f8701ad2790af5cdc69d7994c5f1b922a09cd0b642cca68345d8"
Jul 02 21:29:25.512683 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.512603    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs"]
Jul 02 21:29:25.513028 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.512989    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf41621d-1fee-44e5-bd4f-0a077bf581fb" containerName="download-hf-model"
Jul 02 21:29:25.513028 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.513002    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf41621d-1fee-44e5-bd4f-0a077bf581fb" containerName="download-hf-model"
Jul 02 21:29:25.513028 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.513017    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf41621d-1fee-44e5-bd4f-0a077bf581fb" containerName="s3-init"
Jul 02 21:29:25.513028 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.513022    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf41621d-1fee-44e5-bd4f-0a077bf581fb" containerName="s3-init"
Jul 02 21:29:25.513174 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.513072    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf41621d-1fee-44e5-bd4f-0a077bf581fb" containerName="s3-init"
Jul 02 21:29:25.515921 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.515905    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs"
Jul 02 21:29:25.518286 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.518260    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e36cd-kube-rbac-proxy-sar-config\""
Jul 02 21:29:25.518404 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.518371    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 21:29:25.518487 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.518468    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e36cd-predictor-serving-cert\""
Jul 02 21:29:25.519205 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.519184    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-q64dw\""
Jul 02 21:29:25.519300 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.519184    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 21:29:25.524451 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.524430    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs"]
Jul 02 21:29:25.652078 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.652045    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49gph\" (UniqueName: \"kubernetes.io/projected/2f5229d4-257f-4fd1-9593-85e274c1be8a-kube-api-access-49gph\") pod \"success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs\" (UID: \"2f5229d4-257f-4fd1-9593-85e274c1be8a\") " pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs"
Jul 02 21:29:25.652078 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.652083    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f5229d4-257f-4fd1-9593-85e274c1be8a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs\" (UID: \"2f5229d4-257f-4fd1-9593-85e274c1be8a\") " pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs"
Jul 02 21:29:25.652292 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.652164    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f5229d4-257f-4fd1-9593-85e274c1be8a-proxy-tls\") pod \"success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs\" (UID: \"2f5229d4-257f-4fd1-9593-85e274c1be8a\") " pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs"
Jul 02 21:29:25.753268 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.753235    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49gph\" (UniqueName: \"kubernetes.io/projected/2f5229d4-257f-4fd1-9593-85e274c1be8a-kube-api-access-49gph\") pod \"success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs\" (UID: \"2f5229d4-257f-4fd1-9593-85e274c1be8a\") " pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs"
Jul 02 21:29:25.753461 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.753282    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f5229d4-257f-4fd1-9593-85e274c1be8a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs\" (UID: \"2f5229d4-257f-4fd1-9593-85e274c1be8a\") " pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs"
Jul 02 21:29:25.753461 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.753331    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f5229d4-257f-4fd1-9593-85e274c1be8a-proxy-tls\") pod \"success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs\" (UID: \"2f5229d4-257f-4fd1-9593-85e274c1be8a\") " pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs"
Jul 02 21:29:25.754304 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.754273    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f5229d4-257f-4fd1-9593-85e274c1be8a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs\" (UID: \"2f5229d4-257f-4fd1-9593-85e274c1be8a\") " pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs"
Jul 02 21:29:25.755845 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.755819    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f5229d4-257f-4fd1-9593-85e274c1be8a-proxy-tls\") pod \"success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs\" (UID: \"2f5229d4-257f-4fd1-9593-85e274c1be8a\") " pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs"
Jul 02 21:29:25.761804 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.761769    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49gph\" (UniqueName: \"kubernetes.io/projected/2f5229d4-257f-4fd1-9593-85e274c1be8a-kube-api-access-49gph\") pod \"success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs\" (UID: \"2f5229d4-257f-4fd1-9593-85e274c1be8a\") " pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs"
Jul 02 21:29:25.814174 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.814096    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"]
Jul 02 21:29:25.817885 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.817865    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"
Jul 02 21:29:25.820258 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.820235    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 21:29:25.820543 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.820527    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 21:29:25.820701 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.820685    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
Jul 02 21:29:25.820765 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.820749    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
Jul 02 21:29:25.827247 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.827225    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs"
Jul 02 21:29:25.827827 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.827806    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"]
Jul 02 21:29:25.955418 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.955389    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2n4r\" (UniqueName: \"kubernetes.io/projected/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-kube-api-access-k2n4r\") pod \"isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4\" (UID: \"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"
Jul 02 21:29:25.955574 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.955426    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4\" (UID: \"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"
Jul 02 21:29:25.955574 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.955459    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4\" (UID: \"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"
Jul 02 21:29:25.955648 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.955566    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4\" (UID: \"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"
Jul 02 21:29:25.955648 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.955614    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4\" (UID: \"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"
Jul 02 21:29:25.958771 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:25.958752    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs"]
Jul 02 21:29:25.961445 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:29:25.961417    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f5229d4_257f_4fd1_9593_85e274c1be8a.slice/crio-fdc89792730550cd6816041d5c6d3f2e6bb1555b1aa9fe7e4ad261d9c0e8d611 WatchSource:0}: Error finding container fdc89792730550cd6816041d5c6d3f2e6bb1555b1aa9fe7e4ad261d9c0e8d611: Status 404 returned error can't find the container with id fdc89792730550cd6816041d5c6d3f2e6bb1555b1aa9fe7e4ad261d9c0e8d611
Jul 02 21:29:26.013577 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.013550    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"]
Jul 02 21:29:26.018367 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.018349    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"
Jul 02 21:29:26.020553 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.020533    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
Jul 02 21:29:26.020687 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.020667    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
Jul 02 21:29:26.031202 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.031172    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"]
Jul 02 21:29:26.056181 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.056148    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4\" (UID: \"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"
Jul 02 21:29:26.056318 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.056294    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4\" (UID: \"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"
Jul 02 21:29:26.056359 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.056336    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4\" (UID: \"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"
Jul 02 21:29:26.056411 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.056400    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2n4r\" (UniqueName: \"kubernetes.io/projected/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-kube-api-access-k2n4r\") pod \"isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4\" (UID: \"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"
Jul 02 21:29:26.056461 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.056436    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4\" (UID: \"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"
Jul 02 21:29:26.056693 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.056668    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4\" (UID: \"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"
Jul 02 21:29:26.056833 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.056816    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4\" (UID: \"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"
Jul 02 21:29:26.057014 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.056993    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4\" (UID: \"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"
Jul 02 21:29:26.058835 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.058817    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4\" (UID: \"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"
Jul 02 21:29:26.067710 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.067661    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2n4r\" (UniqueName: \"kubernetes.io/projected/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-kube-api-access-k2n4r\") pod \"isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4\" (UID: \"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"
Jul 02 21:29:26.114510 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.114488    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"]
Jul 02 21:29:26.118261 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.118246    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"
Jul 02 21:29:26.120542 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.120524    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Jul 02 21:29:26.120542 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.120538    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Jul 02 21:29:26.128370 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.128351    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"
Jul 02 21:29:26.131104 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.131027    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"]
Jul 02 21:29:26.157299 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.157269    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d294529b-b6a5-4b4a-ac86-8db2ab88adae-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-445c7\" (UID: \"d294529b-b6a5-4b4a-ac86-8db2ab88adae\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"
Jul 02 21:29:26.157421 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.157324    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d294529b-b6a5-4b4a-ac86-8db2ab88adae-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-445c7\" (UID: \"d294529b-b6a5-4b4a-ac86-8db2ab88adae\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"
Jul 02 21:29:26.157421 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.157392    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xtvm\" (UniqueName: \"kubernetes.io/projected/d294529b-b6a5-4b4a-ac86-8db2ab88adae-kube-api-access-6xtvm\") pod \"isvc-xgboost-graph-predictor-555fc69d56-445c7\" (UID: \"d294529b-b6a5-4b4a-ac86-8db2ab88adae\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"
Jul 02 21:29:26.157514 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.157444    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d294529b-b6a5-4b4a-ac86-8db2ab88adae-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-445c7\" (UID: \"d294529b-b6a5-4b4a-ac86-8db2ab88adae\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"
Jul 02 21:29:26.157514 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.157492    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d294529b-b6a5-4b4a-ac86-8db2ab88adae-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-445c7\" (UID: \"d294529b-b6a5-4b4a-ac86-8db2ab88adae\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"
Jul 02 21:29:26.258690 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.258654    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d294529b-b6a5-4b4a-ac86-8db2ab88adae-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-445c7\" (UID: \"d294529b-b6a5-4b4a-ac86-8db2ab88adae\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"
Jul 02 21:29:26.258837 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.258699    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d294529b-b6a5-4b4a-ac86-8db2ab88adae-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-445c7\" (UID: \"d294529b-b6a5-4b4a-ac86-8db2ab88adae\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"
Jul 02 21:29:26.258837 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.258725    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2\" (UID: \"07c2fae2-0515-4e71-82a5-e8da54ad9ea7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"
Jul 02 21:29:26.258837 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.258746    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xtvm\" (UniqueName: \"kubernetes.io/projected/d294529b-b6a5-4b4a-ac86-8db2ab88adae-kube-api-access-6xtvm\") pod \"isvc-xgboost-graph-predictor-555fc69d56-445c7\" (UID: \"d294529b-b6a5-4b4a-ac86-8db2ab88adae\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"
Jul 02 21:29:26.258837 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.258793    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwjgn\" (UniqueName: \"kubernetes.io/projected/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-kube-api-access-mwjgn\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2\" (UID: \"07c2fae2-0515-4e71-82a5-e8da54ad9ea7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"
Jul 02 21:29:26.259038 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.258844    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2\" (UID: \"07c2fae2-0515-4e71-82a5-e8da54ad9ea7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"
Jul 02 21:29:26.259038 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.258891    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d294529b-b6a5-4b4a-ac86-8db2ab88adae-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-445c7\" (UID: \"d294529b-b6a5-4b4a-ac86-8db2ab88adae\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"
Jul 02 21:29:26.259038 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.258933    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2\" (UID: \"07c2fae2-0515-4e71-82a5-e8da54ad9ea7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"
Jul 02 21:29:26.259038 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:29:26.258976    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-predictor-serving-cert: secret "isvc-xgboost-graph-predictor-serving-cert" not found
Jul 02 21:29:26.259038 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.258981    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d294529b-b6a5-4b4a-ac86-8db2ab88adae-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-445c7\" (UID: \"d294529b-b6a5-4b4a-ac86-8db2ab88adae\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"
Jul 02 21:29:26.259038 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.259015    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2\" (UID: \"07c2fae2-0515-4e71-82a5-e8da54ad9ea7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"
Jul 02 21:29:26.259356 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:29:26.259052    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d294529b-b6a5-4b4a-ac86-8db2ab88adae-proxy-tls podName:d294529b-b6a5-4b4a-ac86-8db2ab88adae nodeName:}" failed. No retries permitted until 2026-07-02 21:29:26.759029247 +0000 UTC m=+627.596269274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d294529b-b6a5-4b4a-ac86-8db2ab88adae-proxy-tls") pod "isvc-xgboost-graph-predictor-555fc69d56-445c7" (UID: "d294529b-b6a5-4b4a-ac86-8db2ab88adae") : secret "isvc-xgboost-graph-predictor-serving-cert" not found
Jul 02 21:29:26.259356 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.259162    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d294529b-b6a5-4b4a-ac86-8db2ab88adae-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-445c7\" (UID: \"d294529b-b6a5-4b4a-ac86-8db2ab88adae\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"
Jul 02 21:29:26.259471 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.259457    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d294529b-b6a5-4b4a-ac86-8db2ab88adae-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-445c7\" (UID: \"d294529b-b6a5-4b4a-ac86-8db2ab88adae\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"
Jul 02 21:29:26.259631 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.259609    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d294529b-b6a5-4b4a-ac86-8db2ab88adae-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-445c7\" (UID: \"d294529b-b6a5-4b4a-ac86-8db2ab88adae\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"
Jul 02 21:29:26.270729 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:29:26.270709    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30a040f_b5b2_4ef4_9cc9_a389f7c7c83b.slice/crio-9b419c5c85f73b08cf4be2f385adf2719116ad895d64d16028b89d2ce50e945b WatchSource:0}: Error finding container 9b419c5c85f73b08cf4be2f385adf2719116ad895d64d16028b89d2ce50e945b: Status 404 returned error can't find the container with id 9b419c5c85f73b08cf4be2f385adf2719116ad895d64d16028b89d2ce50e945b
Jul 02 21:29:26.275067 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.275047    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"]
Jul 02 21:29:26.276098 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.276076    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xtvm\" (UniqueName: \"kubernetes.io/projected/d294529b-b6a5-4b4a-ac86-8db2ab88adae-kube-api-access-6xtvm\") pod \"isvc-xgboost-graph-predictor-555fc69d56-445c7\" (UID: \"d294529b-b6a5-4b4a-ac86-8db2ab88adae\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"
Jul 02 21:29:26.359706 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.359678    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2\" (UID: \"07c2fae2-0515-4e71-82a5-e8da54ad9ea7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"
Jul 02 21:29:26.359820 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.359718    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwjgn\" (UniqueName: \"kubernetes.io/projected/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-kube-api-access-mwjgn\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2\" (UID: \"07c2fae2-0515-4e71-82a5-e8da54ad9ea7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"
Jul 02 21:29:26.359820 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.359739    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2\" (UID: \"07c2fae2-0515-4e71-82a5-e8da54ad9ea7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"
Jul 02 21:29:26.359820 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.359781    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2\" (UID: \"07c2fae2-0515-4e71-82a5-e8da54ad9ea7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"
Jul 02 21:29:26.359820 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.359811    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2\" (UID: \"07c2fae2-0515-4e71-82a5-e8da54ad9ea7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"
Jul 02 21:29:26.360247 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.360229    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2\" (UID: \"07c2fae2-0515-4e71-82a5-e8da54ad9ea7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"
Jul 02 21:29:26.360592 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.360569    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2\" (UID: \"07c2fae2-0515-4e71-82a5-e8da54ad9ea7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"
Jul 02 21:29:26.360687 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.360656    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2\" (UID: \"07c2fae2-0515-4e71-82a5-e8da54ad9ea7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"
Jul 02 21:29:26.362180 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.362162    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2\" (UID: \"07c2fae2-0515-4e71-82a5-e8da54ad9ea7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"
Jul 02 21:29:26.374789 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.374767    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwjgn\" (UniqueName: \"kubernetes.io/projected/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-kube-api-access-mwjgn\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2\" (UID: \"07c2fae2-0515-4e71-82a5-e8da54ad9ea7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"
Jul 02 21:29:26.428315 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.428286    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"
Jul 02 21:29:26.594409 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.594280    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"]
Jul 02 21:29:26.598297 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:29:26.598263    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c2fae2_0515_4e71_82a5_e8da54ad9ea7.slice/crio-7c6f5a7a15d050fdaaaa5986054a0c8004eecd9b1bf8a5abe65d864406b8c4f4 WatchSource:0}: Error finding container 7c6f5a7a15d050fdaaaa5986054a0c8004eecd9b1bf8a5abe65d864406b8c4f4: Status 404 returned error can't find the container with id 7c6f5a7a15d050fdaaaa5986054a0c8004eecd9b1bf8a5abe65d864406b8c4f4
Jul 02 21:29:26.715998 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.715891    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs" event={"ID":"2f5229d4-257f-4fd1-9593-85e274c1be8a","Type":"ContainerStarted","Data":"fdc89792730550cd6816041d5c6d3f2e6bb1555b1aa9fe7e4ad261d9c0e8d611"}
Jul 02 21:29:26.718790 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.718693    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2" event={"ID":"07c2fae2-0515-4e71-82a5-e8da54ad9ea7","Type":"ContainerStarted","Data":"7c6f5a7a15d050fdaaaa5986054a0c8004eecd9b1bf8a5abe65d864406b8c4f4"}
Jul 02 21:29:26.722414 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.721914    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4" event={"ID":"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b","Type":"ContainerStarted","Data":"1d350d04d3582134387b92edb51eff2cc8c386472b27a059c120030fd71022b6"}
Jul 02 21:29:26.722414 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.721962    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4" event={"ID":"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b","Type":"ContainerStarted","Data":"9b419c5c85f73b08cf4be2f385adf2719116ad895d64d16028b89d2ce50e945b"}
Jul 02 21:29:26.764083 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.763523    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d294529b-b6a5-4b4a-ac86-8db2ab88adae-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-445c7\" (UID: \"d294529b-b6a5-4b4a-ac86-8db2ab88adae\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"
Jul 02 21:29:26.775993 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.775939    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d294529b-b6a5-4b4a-ac86-8db2ab88adae-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-445c7\" (UID: \"d294529b-b6a5-4b4a-ac86-8db2ab88adae\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"
Jul 02 21:29:26.931017 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:26.930935    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"
Jul 02 21:29:27.166754 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:27.166723    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"]
Jul 02 21:29:27.169286 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:29:27.169256    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd294529b_b6a5_4b4a_ac86_8db2ab88adae.slice/crio-e84130da34e9fdacbf280ed801b709e85e03800fb447d01e93f9b68b64ddcff1 WatchSource:0}: Error finding container e84130da34e9fdacbf280ed801b709e85e03800fb447d01e93f9b68b64ddcff1: Status 404 returned error can't find the container with id e84130da34e9fdacbf280ed801b709e85e03800fb447d01e93f9b68b64ddcff1
Jul 02 21:29:27.733627 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:27.733590    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7" event={"ID":"d294529b-b6a5-4b4a-ac86-8db2ab88adae","Type":"ContainerStarted","Data":"890834ae0f66a4bc1bad95c59aa577eb697d9a5f2c6201b703e79abacbf86a33"}
Jul 02 21:29:27.734078 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:27.733668    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7" event={"ID":"d294529b-b6a5-4b4a-ac86-8db2ab88adae","Type":"ContainerStarted","Data":"e84130da34e9fdacbf280ed801b709e85e03800fb447d01e93f9b68b64ddcff1"}
Jul 02 21:29:27.739056 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:27.737997    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2" event={"ID":"07c2fae2-0515-4e71-82a5-e8da54ad9ea7","Type":"ContainerStarted","Data":"fa062ad2e6156a34636710e10d094e493b3942a240b3fbe188ac69a0823d77cf"}
Jul 02 21:29:31.763620 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:31.763578    2564 generic.go:356] "Generic (PLEG): container finished" podID="d294529b-b6a5-4b4a-ac86-8db2ab88adae" containerID="890834ae0f66a4bc1bad95c59aa577eb697d9a5f2c6201b703e79abacbf86a33" exitCode=0
Jul 02 21:29:31.764067 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:31.763662    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7" event={"ID":"d294529b-b6a5-4b4a-ac86-8db2ab88adae","Type":"ContainerDied","Data":"890834ae0f66a4bc1bad95c59aa577eb697d9a5f2c6201b703e79abacbf86a33"}
Jul 02 21:29:31.765440 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:31.765416    2564 generic.go:356] "Generic (PLEG): container finished" podID="07c2fae2-0515-4e71-82a5-e8da54ad9ea7" containerID="fa062ad2e6156a34636710e10d094e493b3942a240b3fbe188ac69a0823d77cf" exitCode=0
Jul 02 21:29:31.765535 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:31.765481    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2" event={"ID":"07c2fae2-0515-4e71-82a5-e8da54ad9ea7","Type":"ContainerDied","Data":"fa062ad2e6156a34636710e10d094e493b3942a240b3fbe188ac69a0823d77cf"}
Jul 02 21:29:31.767059 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:31.767035    2564 generic.go:356] "Generic (PLEG): container finished" podID="d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" containerID="1d350d04d3582134387b92edb51eff2cc8c386472b27a059c120030fd71022b6" exitCode=0
Jul 02 21:29:31.767177 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:31.767067    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4" event={"ID":"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b","Type":"ContainerDied","Data":"1d350d04d3582134387b92edb51eff2cc8c386472b27a059c120030fd71022b6"}
Jul 02 21:29:42.843307 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:42.843262    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs" event={"ID":"2f5229d4-257f-4fd1-9593-85e274c1be8a","Type":"ContainerStarted","Data":"275c008b959412d236ec20c46df7bd273186cfaf99b273ec2a227aad15658ac2"}
Jul 02 21:29:42.853900 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:42.853819    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2" event={"ID":"07c2fae2-0515-4e71-82a5-e8da54ad9ea7","Type":"ContainerStarted","Data":"5e60ea43f275ce6a6fbc3dd897b476ae2d8635918e2584027fdf9f560f5020cd"}
Jul 02 21:29:42.861817 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:42.861728    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4" event={"ID":"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b","Type":"ContainerStarted","Data":"0c0f5f408c80a5643c5880090c6c5798958a9768f982ecbab90903cf1de76818"}
Jul 02 21:29:47.902569 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:47.902527    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs" event={"ID":"2f5229d4-257f-4fd1-9593-85e274c1be8a","Type":"ContainerStarted","Data":"c0e16b4d68ba1b4436e55c7914f6ddbef33cc9e21230f1c0b60081159be04cfb"}
Jul 02 21:29:47.903158 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:47.902766    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs"
Jul 02 21:29:47.903158 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:47.902821    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs"
Jul 02 21:29:47.904240 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:47.904199    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs" podUID="2f5229d4-257f-4fd1-9593-85e274c1be8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 02 21:29:47.905164 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:47.905137    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2" event={"ID":"07c2fae2-0515-4e71-82a5-e8da54ad9ea7","Type":"ContainerStarted","Data":"83136ea716789f2e97e63715c2068283dc7d036d6deceae762c463d92b1611df"}
Jul 02 21:29:47.905268 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:47.905253    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"
Jul 02 21:29:47.907471 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:47.907448    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4" event={"ID":"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b","Type":"ContainerStarted","Data":"984fbccad6aea1639bb87075e39b1e13e7de433cbab583ccb3d828f67b55e13e"}
Jul 02 21:29:47.907649 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:47.907632    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"
Jul 02 21:29:47.907753 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:47.907671    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"
Jul 02 21:29:47.908863 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:47.908838    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4" podUID="d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 21:29:47.919669 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:47.919600    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs" podStartSLOduration=2.097347384 podStartE2EDuration="22.919583302s" totalImagesPullingTime="20.822235918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:29:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:29:47.918927813 +0000 UTC m=+648.756167861" watchObservedRunningTime="2026-07-02 21:29:47.919583302 +0000 UTC m=+648.756823353"
Jul 02 21:29:47.939850 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:47.939789    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2" podStartSLOduration=3.056855559 podStartE2EDuration="21.939769821s" totalImagesPullingTime="15.023592323s" totalInitContainerRuntime="3.859321939s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:29:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:29:47.936887557 +0000 UTC m=+648.774127634" watchObservedRunningTime="2026-07-02 21:29:47.939769821 +0000 UTC m=+648.777009872"
Jul 02 21:29:47.954829 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:47.954783    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4" podStartSLOduration=4.436704564 podStartE2EDuration="22.954769911s" totalImagesPullingTime="14.77775698s" totalInitContainerRuntime="3.740308367s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:29:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:29:47.952933738 +0000 UTC m=+648.790173784" watchObservedRunningTime="2026-07-02 21:29:47.954769911 +0000 UTC m=+648.792010008"
Jul 02 21:29:48.913330 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:48.913255    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"
Jul 02 21:29:48.913745 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:48.913440    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs" podUID="2f5229d4-257f-4fd1-9593-85e274c1be8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 02 21:29:48.913745 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:48.913451    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4" podUID="d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 21:29:48.914773 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:48.914747    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2" podUID="07c2fae2-0515-4e71-82a5-e8da54ad9ea7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 02 21:29:49.918794 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:49.918753    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2" podUID="07c2fae2-0515-4e71-82a5-e8da54ad9ea7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 02 21:29:53.918711 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:53.918676    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"
Jul 02 21:29:53.919561 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:53.919159    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs"
Jul 02 21:29:53.919561 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:53.919368    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4" podUID="d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 21:29:53.919822 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:53.919794    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs" podUID="2f5229d4-257f-4fd1-9593-85e274c1be8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 02 21:29:54.924129 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:54.924088    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"
Jul 02 21:29:54.924697 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:29:54.924668    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2" podUID="07c2fae2-0515-4e71-82a5-e8da54ad9ea7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 02 21:30:01.981335 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:01.981298    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7" event={"ID":"d294529b-b6a5-4b4a-ac86-8db2ab88adae","Type":"ContainerStarted","Data":"084d133613437b823da7deef6f4344a0dc6b433a97b3ea6b58161e47f656f4d5"}
Jul 02 21:30:01.981871 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:01.981373    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7" event={"ID":"d294529b-b6a5-4b4a-ac86-8db2ab88adae","Type":"ContainerStarted","Data":"4e750fdb6f43be08966873206a9390bc6bbd7134a3ebfac886e0ee4a480a2ce1"}
Jul 02 21:30:01.981871 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:01.981586    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"
Jul 02 21:30:02.008368 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:02.008323    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7" podStartSLOduration=3.692500023 podStartE2EDuration="37.008309299s" totalImagesPullingTime="29.517531136s" totalInitContainerRuntime="3.79827814s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:30:02.007667284 +0000 UTC m=+662.844907342" watchObservedRunningTime="2026-07-02 21:30:02.008309299 +0000 UTC m=+662.845549347"
Jul 02 21:30:02.987887 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:02.987854    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"
Jul 02 21:30:02.989977 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:02.989935    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7" podUID="d294529b-b6a5-4b4a-ac86-8db2ab88adae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 21:30:03.920250 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:03.920217    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs" podUID="2f5229d4-257f-4fd1-9593-85e274c1be8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 02 21:30:03.920436 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:03.920216    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4" podUID="d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 21:30:03.990896 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:03.990853    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7" podUID="d294529b-b6a5-4b4a-ac86-8db2ab88adae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 21:30:04.925075 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:04.925031    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2" podUID="07c2fae2-0515-4e71-82a5-e8da54ad9ea7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 02 21:30:08.996424 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:08.996396    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"
Jul 02 21:30:08.996905 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:08.996881    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7" podUID="d294529b-b6a5-4b4a-ac86-8db2ab88adae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 21:30:13.920140 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:13.920084    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4" podUID="d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 21:30:13.920542 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:13.920084    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs" podUID="2f5229d4-257f-4fd1-9593-85e274c1be8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 02 21:30:14.925333 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:14.925296    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2" podUID="07c2fae2-0515-4e71-82a5-e8da54ad9ea7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 02 21:30:18.997666 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:18.997623    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7" podUID="d294529b-b6a5-4b4a-ac86-8db2ab88adae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 21:30:23.920216 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:23.920175    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs" podUID="2f5229d4-257f-4fd1-9593-85e274c1be8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 02 21:30:23.920701 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:23.920185    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4" podUID="d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 21:30:24.925032 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:24.924988    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2" podUID="07c2fae2-0515-4e71-82a5-e8da54ad9ea7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 02 21:30:28.997262 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:28.997228    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7" podUID="d294529b-b6a5-4b4a-ac86-8db2ab88adae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 21:30:33.920253 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:33.920213    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4" podUID="d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 21:30:33.920691 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:33.920226    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs" podUID="2f5229d4-257f-4fd1-9593-85e274c1be8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 02 21:30:34.925488 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:34.925451    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2" podUID="07c2fae2-0515-4e71-82a5-e8da54ad9ea7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 02 21:30:38.997089 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:38.997047    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7" podUID="d294529b-b6a5-4b4a-ac86-8db2ab88adae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 21:30:43.919631 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:43.919589    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4" podUID="d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 21:30:43.920261 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:43.920244    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs"
Jul 02 21:30:44.924770 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:44.924731    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2" podUID="07c2fae2-0515-4e71-82a5-e8da54ad9ea7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 02 21:30:48.997782 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:48.997740    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7" podUID="d294529b-b6a5-4b4a-ac86-8db2ab88adae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 21:30:53.919663 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:53.919579    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4" podUID="d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 21:30:54.925340 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:54.925296    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2" podUID="07c2fae2-0515-4e71-82a5-e8da54ad9ea7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 02 21:30:58.997504 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:30:58.997458    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7" podUID="d294529b-b6a5-4b4a-ac86-8db2ab88adae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 21:31:00.069023 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:00.068983    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2" podUID="07c2fae2-0515-4e71-82a5-e8da54ad9ea7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 02 21:31:03.920707 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:03.920676    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"
Jul 02 21:31:08.997868 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:08.997838    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"
Jul 02 21:31:10.055398 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:10.055367    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn"]
Jul 02 21:31:10.058614 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:10.058584    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn"
Jul 02 21:31:10.060957 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:10.060934    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7615e-predictor-serving-cert\""
Jul 02 21:31:10.061105 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:10.060968    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7615e-kube-rbac-proxy-sar-config\""
Jul 02 21:31:10.074026 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:10.074000    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"
Jul 02 21:31:10.074168 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:10.074073    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn"]
Jul 02 21:31:10.084419 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:10.084392    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs"]
Jul 02 21:31:10.084884 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:10.084635    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs" podUID="2f5229d4-257f-4fd1-9593-85e274c1be8a" containerName="kserve-container" containerID="cri-o://275c008b959412d236ec20c46df7bd273186cfaf99b273ec2a227aad15658ac2" gracePeriod=30
Jul 02 21:31:10.084884 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:10.084682    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs" podUID="2f5229d4-257f-4fd1-9593-85e274c1be8a" containerName="kube-rbac-proxy" containerID="cri-o://c0e16b4d68ba1b4436e55c7914f6ddbef33cc9e21230f1c0b60081159be04cfb" gracePeriod=30
Jul 02 21:31:10.123994 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:10.123966    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13a08355-135a-43a9-859d-b69891f40231-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7615e-predictor-6fc7996664-4pfhn\" (UID: \"13a08355-135a-43a9-859d-b69891f40231\") " pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn"
Jul 02 21:31:10.124144 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:10.124029    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13a08355-135a-43a9-859d-b69891f40231-proxy-tls\") pod \"success-200-isvc-7615e-predictor-6fc7996664-4pfhn\" (UID: \"13a08355-135a-43a9-859d-b69891f40231\") " pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn"
Jul 02 21:31:10.124144 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:10.124076    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9w6q\" (UniqueName: \"kubernetes.io/projected/13a08355-135a-43a9-859d-b69891f40231-kube-api-access-f9w6q\") pod \"success-200-isvc-7615e-predictor-6fc7996664-4pfhn\" (UID: \"13a08355-135a-43a9-859d-b69891f40231\") " pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn"
Jul 02 21:31:10.224678 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:10.224653    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13a08355-135a-43a9-859d-b69891f40231-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7615e-predictor-6fc7996664-4pfhn\" (UID: \"13a08355-135a-43a9-859d-b69891f40231\") " pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn"
Jul 02 21:31:10.224842 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:10.224698    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13a08355-135a-43a9-859d-b69891f40231-proxy-tls\") pod \"success-200-isvc-7615e-predictor-6fc7996664-4pfhn\" (UID: \"13a08355-135a-43a9-859d-b69891f40231\") " pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn"
Jul 02 21:31:10.224900 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:10.224850    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9w6q\" (UniqueName: \"kubernetes.io/projected/13a08355-135a-43a9-859d-b69891f40231-kube-api-access-f9w6q\") pod \"success-200-isvc-7615e-predictor-6fc7996664-4pfhn\" (UID: \"13a08355-135a-43a9-859d-b69891f40231\") " pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn"
Jul 02 21:31:10.225403 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:10.225377    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13a08355-135a-43a9-859d-b69891f40231-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7615e-predictor-6fc7996664-4pfhn\" (UID: \"13a08355-135a-43a9-859d-b69891f40231\") " pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn"
Jul 02 21:31:10.226873 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:10.226854    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13a08355-135a-43a9-859d-b69891f40231-proxy-tls\") pod \"success-200-isvc-7615e-predictor-6fc7996664-4pfhn\" (UID: \"13a08355-135a-43a9-859d-b69891f40231\") " pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn"
Jul 02 21:31:10.234009 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:10.233991    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9w6q\" (UniqueName: \"kubernetes.io/projected/13a08355-135a-43a9-859d-b69891f40231-kube-api-access-f9w6q\") pod \"success-200-isvc-7615e-predictor-6fc7996664-4pfhn\" (UID: \"13a08355-135a-43a9-859d-b69891f40231\") " pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn"
Jul 02 21:31:10.281137 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:10.281084    2564 generic.go:356] "Generic (PLEG): container finished" podID="2f5229d4-257f-4fd1-9593-85e274c1be8a" containerID="c0e16b4d68ba1b4436e55c7914f6ddbef33cc9e21230f1c0b60081159be04cfb" exitCode=2
Jul 02 21:31:10.281300 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:10.281164    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs" event={"ID":"2f5229d4-257f-4fd1-9593-85e274c1be8a","Type":"ContainerDied","Data":"c0e16b4d68ba1b4436e55c7914f6ddbef33cc9e21230f1c0b60081159be04cfb"}
Jul 02 21:31:10.372862 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:10.372766    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn"
Jul 02 21:31:10.498372 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:10.498350    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn"]
Jul 02 21:31:10.500638 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:31:10.500608    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a08355_135a_43a9_859d_b69891f40231.slice/crio-7af616d316b89159ef61dd65d0834a770293bb0fdd3dc6fbbdd7a893162305c2 WatchSource:0}: Error finding container 7af616d316b89159ef61dd65d0834a770293bb0fdd3dc6fbbdd7a893162305c2: Status 404 returned error can't find the container with id 7af616d316b89159ef61dd65d0834a770293bb0fdd3dc6fbbdd7a893162305c2
Jul 02 21:31:11.287458 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:11.287421    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn" event={"ID":"13a08355-135a-43a9-859d-b69891f40231","Type":"ContainerStarted","Data":"63d27ba02ad4918da9c194f26fbb6f2ab0ccfd172e7b258af5a4881ad11316d2"}
Jul 02 21:31:11.287893 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:11.287576    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn"
Jul 02 21:31:11.287893 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:11.287624    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn" event={"ID":"13a08355-135a-43a9-859d-b69891f40231","Type":"ContainerStarted","Data":"bd6e088c024acb44a91c66d9aeff8c58efc03982b119b1246f141ddaa12c563a"}
Jul 02 21:31:11.287893 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:11.287650    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn" event={"ID":"13a08355-135a-43a9-859d-b69891f40231","Type":"ContainerStarted","Data":"7af616d316b89159ef61dd65d0834a770293bb0fdd3dc6fbbdd7a893162305c2"}
Jul 02 21:31:11.287893 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:11.287680    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn"
Jul 02 21:31:11.288940 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:11.288914    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn" podUID="13a08355-135a-43a9-859d-b69891f40231" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 02 21:31:11.305335 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:11.305288    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn" podStartSLOduration=1.305273399 podStartE2EDuration="1.305273399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:31:11.304492826 +0000 UTC m=+732.141732872" watchObservedRunningTime="2026-07-02 21:31:11.305273399 +0000 UTC m=+732.142513462"
Jul 02 21:31:12.292338 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:12.292296    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn" podUID="13a08355-135a-43a9-859d-b69891f40231" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 02 21:31:13.231270 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:13.231250    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs"
Jul 02 21:31:13.297941 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:13.297853    2564 generic.go:356] "Generic (PLEG): container finished" podID="2f5229d4-257f-4fd1-9593-85e274c1be8a" containerID="275c008b959412d236ec20c46df7bd273186cfaf99b273ec2a227aad15658ac2" exitCode=0
Jul 02 21:31:13.297941 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:13.297900    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs"
Jul 02 21:31:13.298372 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:13.297938    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs" event={"ID":"2f5229d4-257f-4fd1-9593-85e274c1be8a","Type":"ContainerDied","Data":"275c008b959412d236ec20c46df7bd273186cfaf99b273ec2a227aad15658ac2"}
Jul 02 21:31:13.298372 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:13.298002    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs" event={"ID":"2f5229d4-257f-4fd1-9593-85e274c1be8a","Type":"ContainerDied","Data":"fdc89792730550cd6816041d5c6d3f2e6bb1555b1aa9fe7e4ad261d9c0e8d611"}
Jul 02 21:31:13.298372 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:13.298029    2564 scope.go:117] "RemoveContainer" containerID="c0e16b4d68ba1b4436e55c7914f6ddbef33cc9e21230f1c0b60081159be04cfb"
Jul 02 21:31:13.306830 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:13.306805    2564 scope.go:117] "RemoveContainer" containerID="275c008b959412d236ec20c46df7bd273186cfaf99b273ec2a227aad15658ac2"
Jul 02 21:31:13.314847 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:13.314822    2564 scope.go:117] "RemoveContainer" containerID="c0e16b4d68ba1b4436e55c7914f6ddbef33cc9e21230f1c0b60081159be04cfb"
Jul 02 21:31:13.315091 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:31:13.315072    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0e16b4d68ba1b4436e55c7914f6ddbef33cc9e21230f1c0b60081159be04cfb\": container with ID starting with c0e16b4d68ba1b4436e55c7914f6ddbef33cc9e21230f1c0b60081159be04cfb not found: ID does not exist" containerID="c0e16b4d68ba1b4436e55c7914f6ddbef33cc9e21230f1c0b60081159be04cfb"
Jul 02 21:31:13.315184 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:13.315098    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0e16b4d68ba1b4436e55c7914f6ddbef33cc9e21230f1c0b60081159be04cfb"} err="failed to get container status \"c0e16b4d68ba1b4436e55c7914f6ddbef33cc9e21230f1c0b60081159be04cfb\": rpc error: code = NotFound desc = could not find container \"c0e16b4d68ba1b4436e55c7914f6ddbef33cc9e21230f1c0b60081159be04cfb\": container with ID starting with c0e16b4d68ba1b4436e55c7914f6ddbef33cc9e21230f1c0b60081159be04cfb not found: ID does not exist"
Jul 02 21:31:13.315184 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:13.315133    2564 scope.go:117] "RemoveContainer" containerID="275c008b959412d236ec20c46df7bd273186cfaf99b273ec2a227aad15658ac2"
Jul 02 21:31:13.315371 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:31:13.315354    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"275c008b959412d236ec20c46df7bd273186cfaf99b273ec2a227aad15658ac2\": container with ID starting with 275c008b959412d236ec20c46df7bd273186cfaf99b273ec2a227aad15658ac2 not found: ID does not exist" containerID="275c008b959412d236ec20c46df7bd273186cfaf99b273ec2a227aad15658ac2"
Jul 02 21:31:13.315425 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:13.315374    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"275c008b959412d236ec20c46df7bd273186cfaf99b273ec2a227aad15658ac2"} err="failed to get container status \"275c008b959412d236ec20c46df7bd273186cfaf99b273ec2a227aad15658ac2\": rpc error: code = NotFound desc = could not find container \"275c008b959412d236ec20c46df7bd273186cfaf99b273ec2a227aad15658ac2\": container with ID starting with 275c008b959412d236ec20c46df7bd273186cfaf99b273ec2a227aad15658ac2 not found: ID does not exist"
Jul 02 21:31:13.352271 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:13.352240    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f5229d4-257f-4fd1-9593-85e274c1be8a-kube-rbac-proxy-sar-config\") pod \"2f5229d4-257f-4fd1-9593-85e274c1be8a\" (UID: \"2f5229d4-257f-4fd1-9593-85e274c1be8a\") "
Jul 02 21:31:13.352446 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:13.352315    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49gph\" (UniqueName: \"kubernetes.io/projected/2f5229d4-257f-4fd1-9593-85e274c1be8a-kube-api-access-49gph\") pod \"2f5229d4-257f-4fd1-9593-85e274c1be8a\" (UID: \"2f5229d4-257f-4fd1-9593-85e274c1be8a\") "
Jul 02 21:31:13.352446 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:13.352362    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f5229d4-257f-4fd1-9593-85e274c1be8a-proxy-tls\") pod \"2f5229d4-257f-4fd1-9593-85e274c1be8a\" (UID: \"2f5229d4-257f-4fd1-9593-85e274c1be8a\") "
Jul 02 21:31:13.352649 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:13.352623    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f5229d4-257f-4fd1-9593-85e274c1be8a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2f5229d4-257f-4fd1-9593-85e274c1be8a" (UID: "2f5229d4-257f-4fd1-9593-85e274c1be8a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:31:13.354448 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:13.354418    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f5229d4-257f-4fd1-9593-85e274c1be8a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2f5229d4-257f-4fd1-9593-85e274c1be8a" (UID: "2f5229d4-257f-4fd1-9593-85e274c1be8a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:31:13.354448 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:13.354436    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f5229d4-257f-4fd1-9593-85e274c1be8a-kube-api-access-49gph" (OuterVolumeSpecName: "kube-api-access-49gph") pod "2f5229d4-257f-4fd1-9593-85e274c1be8a" (UID: "2f5229d4-257f-4fd1-9593-85e274c1be8a"). InnerVolumeSpecName "kube-api-access-49gph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:31:13.452975 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:13.452939    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f5229d4-257f-4fd1-9593-85e274c1be8a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:31:13.452975 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:13.452968    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49gph\" (UniqueName: \"kubernetes.io/projected/2f5229d4-257f-4fd1-9593-85e274c1be8a-kube-api-access-49gph\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:31:13.452975 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:13.452977    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f5229d4-257f-4fd1-9593-85e274c1be8a-proxy-tls\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:31:13.619991 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:13.619957    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs"]
Jul 02 21:31:13.625639 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:13.625615    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e36cd-predictor-fdf4cf55d-dfmzs"]
Jul 02 21:31:14.072008 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:14.071978    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f5229d4-257f-4fd1-9593-85e274c1be8a" path="/var/lib/kubelet/pods/2f5229d4-257f-4fd1-9593-85e274c1be8a/volumes"
Jul 02 21:31:17.297009 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:17.296979    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn"
Jul 02 21:31:17.297546 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:17.297520    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn" podUID="13a08355-135a-43a9-859d-b69891f40231" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 02 21:31:27.297946 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:27.297907    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn" podUID="13a08355-135a-43a9-859d-b69891f40231" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 02 21:31:35.820061 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:35.820029    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"]
Jul 02 21:31:35.820480 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:35.820352    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4" podUID="d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" containerName="kube-rbac-proxy" containerID="cri-o://984fbccad6aea1639bb87075e39b1e13e7de433cbab583ccb3d828f67b55e13e" gracePeriod=30
Jul 02 21:31:35.820480 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:35.820339    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4" podUID="d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" containerName="kserve-container" containerID="cri-o://0c0f5f408c80a5643c5880090c6c5798958a9768f982ecbab90903cf1de76818" gracePeriod=30
Jul 02 21:31:35.843666 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:35.843640    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn"]
Jul 02 21:31:35.844023 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:35.844011    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f5229d4-257f-4fd1-9593-85e274c1be8a" containerName="kube-rbac-proxy"
Jul 02 21:31:35.844066 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:35.844025    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f5229d4-257f-4fd1-9593-85e274c1be8a" containerName="kube-rbac-proxy"
Jul 02 21:31:35.844066 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:35.844036    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f5229d4-257f-4fd1-9593-85e274c1be8a" containerName="kserve-container"
Jul 02 21:31:35.844066 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:35.844041    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f5229d4-257f-4fd1-9593-85e274c1be8a" containerName="kserve-container"
Jul 02 21:31:35.844189 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:35.844096    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f5229d4-257f-4fd1-9593-85e274c1be8a" containerName="kserve-container"
Jul 02 21:31:35.844189 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:35.844124    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f5229d4-257f-4fd1-9593-85e274c1be8a" containerName="kube-rbac-proxy"
Jul 02 21:31:35.846326 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:35.846309    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn"
Jul 02 21:31:35.848893 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:35.848873    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a3b0c-predictor-serving-cert\""
Jul 02 21:31:35.848893 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:35.848886    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a3b0c-kube-rbac-proxy-sar-config\""
Jul 02 21:31:35.856231 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:35.856210    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn"]
Jul 02 21:31:35.937214 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:35.937130    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"]
Jul 02 21:31:35.937420 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:35.937393    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7" podUID="d294529b-b6a5-4b4a-ac86-8db2ab88adae" containerName="kserve-container" containerID="cri-o://4e750fdb6f43be08966873206a9390bc6bbd7134a3ebfac886e0ee4a480a2ce1" gracePeriod=30
Jul 02 21:31:35.937538 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:35.937425    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7" podUID="d294529b-b6a5-4b4a-ac86-8db2ab88adae" containerName="kube-rbac-proxy" containerID="cri-o://084d133613437b823da7deef6f4344a0dc6b433a97b3ea6b58161e47f656f4d5" gracePeriod=30
Jul 02 21:31:35.955412 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:35.955388    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83ed67a9-3f3d-4452-a21e-65b6eb3913ec-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn\" (UID: \"83ed67a9-3f3d-4452-a21e-65b6eb3913ec\") " pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn"
Jul 02 21:31:35.955509 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:35.955430    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83ed67a9-3f3d-4452-a21e-65b6eb3913ec-proxy-tls\") pod \"success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn\" (UID: \"83ed67a9-3f3d-4452-a21e-65b6eb3913ec\") " pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn"
Jul 02 21:31:35.955599 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:35.955579    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkft6\" (UniqueName: \"kubernetes.io/projected/83ed67a9-3f3d-4452-a21e-65b6eb3913ec-kube-api-access-mkft6\") pod \"success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn\" (UID: \"83ed67a9-3f3d-4452-a21e-65b6eb3913ec\") " pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn"
Jul 02 21:31:36.045497 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:36.045464    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"]
Jul 02 21:31:36.045727 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:36.045693    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2" podUID="07c2fae2-0515-4e71-82a5-e8da54ad9ea7" containerName="kserve-container" containerID="cri-o://5e60ea43f275ce6a6fbc3dd897b476ae2d8635918e2584027fdf9f560f5020cd" gracePeriod=30
Jul 02 21:31:36.045811 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:36.045741    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2" podUID="07c2fae2-0515-4e71-82a5-e8da54ad9ea7" containerName="kube-rbac-proxy" containerID="cri-o://83136ea716789f2e97e63715c2068283dc7d036d6deceae762c463d92b1611df" gracePeriod=30
Jul 02 21:31:36.056334 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:36.056313    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83ed67a9-3f3d-4452-a21e-65b6eb3913ec-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn\" (UID: \"83ed67a9-3f3d-4452-a21e-65b6eb3913ec\") " pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn"
Jul 02 21:31:36.056410 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:36.056354    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83ed67a9-3f3d-4452-a21e-65b6eb3913ec-proxy-tls\") pod \"success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn\" (UID: \"83ed67a9-3f3d-4452-a21e-65b6eb3913ec\") " pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn"
Jul 02 21:31:36.056452 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:36.056412    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkft6\" (UniqueName: \"kubernetes.io/projected/83ed67a9-3f3d-4452-a21e-65b6eb3913ec-kube-api-access-mkft6\") pod \"success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn\" (UID: \"83ed67a9-3f3d-4452-a21e-65b6eb3913ec\") " pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn"
Jul 02 21:31:36.056546 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:31:36.056529    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-serving-cert: secret "success-200-isvc-a3b0c-predictor-serving-cert" not found
Jul 02 21:31:36.056604 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:31:36.056594    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83ed67a9-3f3d-4452-a21e-65b6eb3913ec-proxy-tls podName:83ed67a9-3f3d-4452-a21e-65b6eb3913ec nodeName:}" failed. No retries permitted until 2026-07-02 21:31:36.556578198 +0000 UTC m=+757.393818223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/83ed67a9-3f3d-4452-a21e-65b6eb3913ec-proxy-tls") pod "success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn" (UID: "83ed67a9-3f3d-4452-a21e-65b6eb3913ec") : secret "success-200-isvc-a3b0c-predictor-serving-cert" not found
Jul 02 21:31:36.057143 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:36.057085    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83ed67a9-3f3d-4452-a21e-65b6eb3913ec-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn\" (UID: \"83ed67a9-3f3d-4452-a21e-65b6eb3913ec\") " pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn"
Jul 02 21:31:36.070586 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:36.070527    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkft6\" (UniqueName: \"kubernetes.io/projected/83ed67a9-3f3d-4452-a21e-65b6eb3913ec-kube-api-access-mkft6\") pod \"success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn\" (UID: \"83ed67a9-3f3d-4452-a21e-65b6eb3913ec\") " pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn"
Jul 02 21:31:36.402261 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:36.402181    2564 generic.go:356] "Generic (PLEG): container finished" podID="d294529b-b6a5-4b4a-ac86-8db2ab88adae" containerID="084d133613437b823da7deef6f4344a0dc6b433a97b3ea6b58161e47f656f4d5" exitCode=2
Jul 02 21:31:36.402436 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:36.402255    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7" event={"ID":"d294529b-b6a5-4b4a-ac86-8db2ab88adae","Type":"ContainerDied","Data":"084d133613437b823da7deef6f4344a0dc6b433a97b3ea6b58161e47f656f4d5"}
Jul 02 21:31:36.404285 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:36.404263    2564 generic.go:356] "Generic (PLEG): container finished" podID="07c2fae2-0515-4e71-82a5-e8da54ad9ea7" containerID="83136ea716789f2e97e63715c2068283dc7d036d6deceae762c463d92b1611df" exitCode=2
Jul 02 21:31:36.404400 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:36.404337    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2" event={"ID":"07c2fae2-0515-4e71-82a5-e8da54ad9ea7","Type":"ContainerDied","Data":"83136ea716789f2e97e63715c2068283dc7d036d6deceae762c463d92b1611df"}
Jul 02 21:31:36.406328 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:36.406305    2564 generic.go:356] "Generic (PLEG): container finished" podID="d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" containerID="984fbccad6aea1639bb87075e39b1e13e7de433cbab583ccb3d828f67b55e13e" exitCode=2
Jul 02 21:31:36.406409 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:36.406348    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4" event={"ID":"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b","Type":"ContainerDied","Data":"984fbccad6aea1639bb87075e39b1e13e7de433cbab583ccb3d828f67b55e13e"}
Jul 02 21:31:36.560552 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:36.560515    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83ed67a9-3f3d-4452-a21e-65b6eb3913ec-proxy-tls\") pod \"success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn\" (UID: \"83ed67a9-3f3d-4452-a21e-65b6eb3913ec\") " pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn"
Jul 02 21:31:36.562914 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:36.562892    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83ed67a9-3f3d-4452-a21e-65b6eb3913ec-proxy-tls\") pod \"success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn\" (UID: \"83ed67a9-3f3d-4452-a21e-65b6eb3913ec\") " pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn"
Jul 02 21:31:36.756687 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:36.756651    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn"
Jul 02 21:31:36.898236 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:36.898212    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn"]
Jul 02 21:31:36.900158 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:31:36.900128    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ed67a9_3f3d_4452_a21e_65b6eb3913ec.slice/crio-afb59cc280bd0fa1d2aa9100d10eb6b40d9c5d1268572916b4438023013040a8 WatchSource:0}: Error finding container afb59cc280bd0fa1d2aa9100d10eb6b40d9c5d1268572916b4438023013040a8: Status 404 returned error can't find the container with id afb59cc280bd0fa1d2aa9100d10eb6b40d9c5d1268572916b4438023013040a8
Jul 02 21:31:37.297789 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:37.297708    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn" podUID="13a08355-135a-43a9-859d-b69891f40231" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 02 21:31:37.413145 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:37.413091    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn" event={"ID":"83ed67a9-3f3d-4452-a21e-65b6eb3913ec","Type":"ContainerStarted","Data":"a22ae178fe4165b230ffe90d591d6700dcaf048fad2820be74748127b27ad1ed"}
Jul 02 21:31:37.413332 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:37.413176    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn" event={"ID":"83ed67a9-3f3d-4452-a21e-65b6eb3913ec","Type":"ContainerStarted","Data":"5557d6e3e8f78e26bfdab5592c3c32b5887d44d713b64d4c097bfa4277c39d7d"}
Jul 02 21:31:37.413332 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:37.413203    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn" event={"ID":"83ed67a9-3f3d-4452-a21e-65b6eb3913ec","Type":"ContainerStarted","Data":"afb59cc280bd0fa1d2aa9100d10eb6b40d9c5d1268572916b4438023013040a8"}
Jul 02 21:31:37.413332 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:37.413233    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn"
Jul 02 21:31:37.413332 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:37.413259    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn"
Jul 02 21:31:37.414435 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:37.414412    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn" podUID="83ed67a9-3f3d-4452-a21e-65b6eb3913ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 02 21:31:37.432901 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:37.432843    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn" podStartSLOduration=2.432823447 podStartE2EDuration="2.432823447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:31:37.430422443 +0000 UTC m=+758.267662503" watchObservedRunningTime="2026-07-02 21:31:37.432823447 +0000 UTC m=+758.270063494"
Jul 02 21:31:38.418250 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:38.418215    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn" podUID="83ed67a9-3f3d-4452-a21e-65b6eb3913ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 02 21:31:38.913505 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:38.913467    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4" podUID="d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 02 21:31:38.991965 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:38.991920    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7" podUID="d294529b-b6a5-4b4a-ac86-8db2ab88adae" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 02 21:31:38.997032 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:38.997005    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7" podUID="d294529b-b6a5-4b4a-ac86-8db2ab88adae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 21:31:39.686728 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:39.686695    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"
Jul 02 21:31:39.790411 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:39.790317    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xtvm\" (UniqueName: \"kubernetes.io/projected/d294529b-b6a5-4b4a-ac86-8db2ab88adae-kube-api-access-6xtvm\") pod \"d294529b-b6a5-4b4a-ac86-8db2ab88adae\" (UID: \"d294529b-b6a5-4b4a-ac86-8db2ab88adae\") "
Jul 02 21:31:39.790411 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:39.790366    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d294529b-b6a5-4b4a-ac86-8db2ab88adae-kube-rbac-proxy-sar-config\") pod \"d294529b-b6a5-4b4a-ac86-8db2ab88adae\" (UID: \"d294529b-b6a5-4b4a-ac86-8db2ab88adae\") "
Jul 02 21:31:39.790411 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:39.790393    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d294529b-b6a5-4b4a-ac86-8db2ab88adae-cabundle-cert\") pod \"d294529b-b6a5-4b4a-ac86-8db2ab88adae\" (UID: \"d294529b-b6a5-4b4a-ac86-8db2ab88adae\") "
Jul 02 21:31:39.790696 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:39.790478    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d294529b-b6a5-4b4a-ac86-8db2ab88adae-proxy-tls\") pod \"d294529b-b6a5-4b4a-ac86-8db2ab88adae\" (UID: \"d294529b-b6a5-4b4a-ac86-8db2ab88adae\") "
Jul 02 21:31:39.790696 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:39.790558    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d294529b-b6a5-4b4a-ac86-8db2ab88adae-kserve-provision-location\") pod \"d294529b-b6a5-4b4a-ac86-8db2ab88adae\" (UID: \"d294529b-b6a5-4b4a-ac86-8db2ab88adae\") "
Jul 02 21:31:39.790786 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:39.790746    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d294529b-b6a5-4b4a-ac86-8db2ab88adae-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d294529b-b6a5-4b4a-ac86-8db2ab88adae" (UID: "d294529b-b6a5-4b4a-ac86-8db2ab88adae"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:31:39.790786 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:39.790758    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d294529b-b6a5-4b4a-ac86-8db2ab88adae-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d294529b-b6a5-4b4a-ac86-8db2ab88adae" (UID: "d294529b-b6a5-4b4a-ac86-8db2ab88adae"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:31:39.790885 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:39.790833    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d294529b-b6a5-4b4a-ac86-8db2ab88adae-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:31:39.790885 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:39.790850    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d294529b-b6a5-4b4a-ac86-8db2ab88adae-cabundle-cert\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:31:39.791018 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:39.790996    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d294529b-b6a5-4b4a-ac86-8db2ab88adae-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d294529b-b6a5-4b4a-ac86-8db2ab88adae" (UID: "d294529b-b6a5-4b4a-ac86-8db2ab88adae"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:31:39.792486 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:39.792462    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d294529b-b6a5-4b4a-ac86-8db2ab88adae-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d294529b-b6a5-4b4a-ac86-8db2ab88adae" (UID: "d294529b-b6a5-4b4a-ac86-8db2ab88adae"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:31:39.792605 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:39.792537    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d294529b-b6a5-4b4a-ac86-8db2ab88adae-kube-api-access-6xtvm" (OuterVolumeSpecName: "kube-api-access-6xtvm") pod "d294529b-b6a5-4b4a-ac86-8db2ab88adae" (UID: "d294529b-b6a5-4b4a-ac86-8db2ab88adae"). InnerVolumeSpecName "kube-api-access-6xtvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:31:39.892239 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:39.892209    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d294529b-b6a5-4b4a-ac86-8db2ab88adae-proxy-tls\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:31:39.892239 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:39.892233    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d294529b-b6a5-4b4a-ac86-8db2ab88adae-kserve-provision-location\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:31:39.892239 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:39.892243    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xtvm\" (UniqueName: \"kubernetes.io/projected/d294529b-b6a5-4b4a-ac86-8db2ab88adae-kube-api-access-6xtvm\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:31:39.919472 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:39.919440    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2" podUID="07c2fae2-0515-4e71-82a5-e8da54ad9ea7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 02 21:31:40.069333 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:40.069241    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2" podUID="07c2fae2-0515-4e71-82a5-e8da54ad9ea7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 02 21:31:40.429499 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:40.429410    2564 generic.go:356] "Generic (PLEG): container finished" podID="d294529b-b6a5-4b4a-ac86-8db2ab88adae" containerID="4e750fdb6f43be08966873206a9390bc6bbd7134a3ebfac886e0ee4a480a2ce1" exitCode=0
Jul 02 21:31:40.429499 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:40.429466    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"
Jul 02 21:31:40.429499 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:40.429483    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7" event={"ID":"d294529b-b6a5-4b4a-ac86-8db2ab88adae","Type":"ContainerDied","Data":"4e750fdb6f43be08966873206a9390bc6bbd7134a3ebfac886e0ee4a480a2ce1"}
Jul 02 21:31:40.429778 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:40.429542    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7" event={"ID":"d294529b-b6a5-4b4a-ac86-8db2ab88adae","Type":"ContainerDied","Data":"e84130da34e9fdacbf280ed801b709e85e03800fb447d01e93f9b68b64ddcff1"}
Jul 02 21:31:40.429778 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:40.429568    2564 scope.go:117] "RemoveContainer" containerID="084d133613437b823da7deef6f4344a0dc6b433a97b3ea6b58161e47f656f4d5"
Jul 02 21:31:40.447837 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:40.447811    2564 scope.go:117] "RemoveContainer" containerID="4e750fdb6f43be08966873206a9390bc6bbd7134a3ebfac886e0ee4a480a2ce1"
Jul 02 21:31:40.449769 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:40.449745    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"]
Jul 02 21:31:40.453621 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:40.453601    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-445c7"]
Jul 02 21:31:40.455732 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:40.455720    2564 scope.go:117] "RemoveContainer" containerID="890834ae0f66a4bc1bad95c59aa577eb697d9a5f2c6201b703e79abacbf86a33"
Jul 02 21:31:40.463004 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:40.462986    2564 scope.go:117] "RemoveContainer" containerID="084d133613437b823da7deef6f4344a0dc6b433a97b3ea6b58161e47f656f4d5"
Jul 02 21:31:40.463263 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:31:40.463244    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"084d133613437b823da7deef6f4344a0dc6b433a97b3ea6b58161e47f656f4d5\": container with ID starting with 084d133613437b823da7deef6f4344a0dc6b433a97b3ea6b58161e47f656f4d5 not found: ID does not exist" containerID="084d133613437b823da7deef6f4344a0dc6b433a97b3ea6b58161e47f656f4d5"
Jul 02 21:31:40.463340 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:40.463276    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"084d133613437b823da7deef6f4344a0dc6b433a97b3ea6b58161e47f656f4d5"} err="failed to get container status \"084d133613437b823da7deef6f4344a0dc6b433a97b3ea6b58161e47f656f4d5\": rpc error: code = NotFound desc = could not find container \"084d133613437b823da7deef6f4344a0dc6b433a97b3ea6b58161e47f656f4d5\": container with ID starting with 084d133613437b823da7deef6f4344a0dc6b433a97b3ea6b58161e47f656f4d5 not found: ID does not exist"
Jul 02 21:31:40.463340 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:40.463302    2564 scope.go:117] "RemoveContainer" containerID="4e750fdb6f43be08966873206a9390bc6bbd7134a3ebfac886e0ee4a480a2ce1"
Jul 02 21:31:40.463521 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:31:40.463505    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e750fdb6f43be08966873206a9390bc6bbd7134a3ebfac886e0ee4a480a2ce1\": container with ID starting with 4e750fdb6f43be08966873206a9390bc6bbd7134a3ebfac886e0ee4a480a2ce1 not found: ID does not exist" containerID="4e750fdb6f43be08966873206a9390bc6bbd7134a3ebfac886e0ee4a480a2ce1"
Jul 02 21:31:40.463559 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:40.463527    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e750fdb6f43be08966873206a9390bc6bbd7134a3ebfac886e0ee4a480a2ce1"} err="failed to get container status \"4e750fdb6f43be08966873206a9390bc6bbd7134a3ebfac886e0ee4a480a2ce1\": rpc error: code = NotFound desc = could not find container \"4e750fdb6f43be08966873206a9390bc6bbd7134a3ebfac886e0ee4a480a2ce1\": container with ID starting with 4e750fdb6f43be08966873206a9390bc6bbd7134a3ebfac886e0ee4a480a2ce1 not found: ID does not exist"
Jul 02 21:31:40.463559 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:40.463542    2564 scope.go:117] "RemoveContainer" containerID="890834ae0f66a4bc1bad95c59aa577eb697d9a5f2c6201b703e79abacbf86a33"
Jul 02 21:31:40.463728 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:31:40.463713    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"890834ae0f66a4bc1bad95c59aa577eb697d9a5f2c6201b703e79abacbf86a33\": container with ID starting with 890834ae0f66a4bc1bad95c59aa577eb697d9a5f2c6201b703e79abacbf86a33 not found: ID does not exist" containerID="890834ae0f66a4bc1bad95c59aa577eb697d9a5f2c6201b703e79abacbf86a33"
Jul 02 21:31:40.463766 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:40.463732    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"890834ae0f66a4bc1bad95c59aa577eb697d9a5f2c6201b703e79abacbf86a33"} err="failed to get container status \"890834ae0f66a4bc1bad95c59aa577eb697d9a5f2c6201b703e79abacbf86a33\": rpc error: code = NotFound desc = could not find container \"890834ae0f66a4bc1bad95c59aa577eb697d9a5f2c6201b703e79abacbf86a33\": container with ID starting with 890834ae0f66a4bc1bad95c59aa577eb697d9a5f2c6201b703e79abacbf86a33 not found: ID does not exist"
Jul 02 21:31:40.988307 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:40.988285    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"
Jul 02 21:31:41.000188 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.000166    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-kube-rbac-proxy-sar-config\") pod \"07c2fae2-0515-4e71-82a5-e8da54ad9ea7\" (UID: \"07c2fae2-0515-4e71-82a5-e8da54ad9ea7\") "
Jul 02 21:31:41.000261 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.000211    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwjgn\" (UniqueName: \"kubernetes.io/projected/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-kube-api-access-mwjgn\") pod \"07c2fae2-0515-4e71-82a5-e8da54ad9ea7\" (UID: \"07c2fae2-0515-4e71-82a5-e8da54ad9ea7\") "
Jul 02 21:31:41.000261 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.000249    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-proxy-tls\") pod \"07c2fae2-0515-4e71-82a5-e8da54ad9ea7\" (UID: \"07c2fae2-0515-4e71-82a5-e8da54ad9ea7\") "
Jul 02 21:31:41.000324 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.000308    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-kserve-provision-location\") pod \"07c2fae2-0515-4e71-82a5-e8da54ad9ea7\" (UID: \"07c2fae2-0515-4e71-82a5-e8da54ad9ea7\") "
Jul 02 21:31:41.000357 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.000324    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-cabundle-cert\") pod \"07c2fae2-0515-4e71-82a5-e8da54ad9ea7\" (UID: \"07c2fae2-0515-4e71-82a5-e8da54ad9ea7\") "
Jul 02 21:31:41.000660 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.000637    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "07c2fae2-0515-4e71-82a5-e8da54ad9ea7" (UID: "07c2fae2-0515-4e71-82a5-e8da54ad9ea7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:31:41.000761 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.000695    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "07c2fae2-0515-4e71-82a5-e8da54ad9ea7" (UID: "07c2fae2-0515-4e71-82a5-e8da54ad9ea7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:31:41.000761 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.000716    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "07c2fae2-0515-4e71-82a5-e8da54ad9ea7" (UID: "07c2fae2-0515-4e71-82a5-e8da54ad9ea7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:31:41.002328 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.002306    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "07c2fae2-0515-4e71-82a5-e8da54ad9ea7" (UID: "07c2fae2-0515-4e71-82a5-e8da54ad9ea7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:31:41.002401 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.002338    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-kube-api-access-mwjgn" (OuterVolumeSpecName: "kube-api-access-mwjgn") pod "07c2fae2-0515-4e71-82a5-e8da54ad9ea7" (UID: "07c2fae2-0515-4e71-82a5-e8da54ad9ea7"). InnerVolumeSpecName "kube-api-access-mwjgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:31:41.101024 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.100945    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-kserve-provision-location\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:31:41.101024 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.100974    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-cabundle-cert\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:31:41.101024 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.100984    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:31:41.101024 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.100992    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwjgn\" (UniqueName: \"kubernetes.io/projected/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-kube-api-access-mwjgn\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:31:41.101024 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.101000    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07c2fae2-0515-4e71-82a5-e8da54ad9ea7-proxy-tls\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:31:41.437523 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.437496    2564 generic.go:356] "Generic (PLEG): container finished" podID="07c2fae2-0515-4e71-82a5-e8da54ad9ea7" containerID="5e60ea43f275ce6a6fbc3dd897b476ae2d8635918e2584027fdf9f560f5020cd" exitCode=0
Jul 02 21:31:41.437625 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.437586    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"
Jul 02 21:31:41.437625 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.437580    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2" event={"ID":"07c2fae2-0515-4e71-82a5-e8da54ad9ea7","Type":"ContainerDied","Data":"5e60ea43f275ce6a6fbc3dd897b476ae2d8635918e2584027fdf9f560f5020cd"}
Jul 02 21:31:41.437738 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.437678    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2" event={"ID":"07c2fae2-0515-4e71-82a5-e8da54ad9ea7","Type":"ContainerDied","Data":"7c6f5a7a15d050fdaaaa5986054a0c8004eecd9b1bf8a5abe65d864406b8c4f4"}
Jul 02 21:31:41.437800 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.437734    2564 scope.go:117] "RemoveContainer" containerID="83136ea716789f2e97e63715c2068283dc7d036d6deceae762c463d92b1611df"
Jul 02 21:31:41.440312 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.440287    2564 generic.go:356] "Generic (PLEG): container finished" podID="d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" containerID="0c0f5f408c80a5643c5880090c6c5798958a9768f982ecbab90903cf1de76818" exitCode=0
Jul 02 21:31:41.440438 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.440369    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4" event={"ID":"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b","Type":"ContainerDied","Data":"0c0f5f408c80a5643c5880090c6c5798958a9768f982ecbab90903cf1de76818"}
Jul 02 21:31:41.446095 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.446074    2564 scope.go:117] "RemoveContainer" containerID="5e60ea43f275ce6a6fbc3dd897b476ae2d8635918e2584027fdf9f560f5020cd"
Jul 02 21:31:41.454568 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.454546    2564 scope.go:117] "RemoveContainer" containerID="fa062ad2e6156a34636710e10d094e493b3942a240b3fbe188ac69a0823d77cf"
Jul 02 21:31:41.462501 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.462483    2564 scope.go:117] "RemoveContainer" containerID="83136ea716789f2e97e63715c2068283dc7d036d6deceae762c463d92b1611df"
Jul 02 21:31:41.462565 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.462545    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"]
Jul 02 21:31:41.462764 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:31:41.462745    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83136ea716789f2e97e63715c2068283dc7d036d6deceae762c463d92b1611df\": container with ID starting with 83136ea716789f2e97e63715c2068283dc7d036d6deceae762c463d92b1611df not found: ID does not exist" containerID="83136ea716789f2e97e63715c2068283dc7d036d6deceae762c463d92b1611df"
Jul 02 21:31:41.462817 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.462770    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83136ea716789f2e97e63715c2068283dc7d036d6deceae762c463d92b1611df"} err="failed to get container status \"83136ea716789f2e97e63715c2068283dc7d036d6deceae762c463d92b1611df\": rpc error: code = NotFound desc = could not find container \"83136ea716789f2e97e63715c2068283dc7d036d6deceae762c463d92b1611df\": container with ID starting with 83136ea716789f2e97e63715c2068283dc7d036d6deceae762c463d92b1611df not found: ID does not exist"
Jul 02 21:31:41.462817 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.462792    2564 scope.go:117] "RemoveContainer" containerID="5e60ea43f275ce6a6fbc3dd897b476ae2d8635918e2584027fdf9f560f5020cd"
Jul 02 21:31:41.463067 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:31:41.463050    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e60ea43f275ce6a6fbc3dd897b476ae2d8635918e2584027fdf9f560f5020cd\": container with ID starting with 5e60ea43f275ce6a6fbc3dd897b476ae2d8635918e2584027fdf9f560f5020cd not found: ID does not exist" containerID="5e60ea43f275ce6a6fbc3dd897b476ae2d8635918e2584027fdf9f560f5020cd"
Jul 02 21:31:41.463134 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.463075    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e60ea43f275ce6a6fbc3dd897b476ae2d8635918e2584027fdf9f560f5020cd"} err="failed to get container status \"5e60ea43f275ce6a6fbc3dd897b476ae2d8635918e2584027fdf9f560f5020cd\": rpc error: code = NotFound desc = could not find container \"5e60ea43f275ce6a6fbc3dd897b476ae2d8635918e2584027fdf9f560f5020cd\": container with ID starting with 5e60ea43f275ce6a6fbc3dd897b476ae2d8635918e2584027fdf9f560f5020cd not found: ID does not exist"
Jul 02 21:31:41.463134 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.463089    2564 scope.go:117] "RemoveContainer" containerID="fa062ad2e6156a34636710e10d094e493b3942a240b3fbe188ac69a0823d77cf"
Jul 02 21:31:41.463358 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:31:41.463338    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa062ad2e6156a34636710e10d094e493b3942a240b3fbe188ac69a0823d77cf\": container with ID starting with fa062ad2e6156a34636710e10d094e493b3942a240b3fbe188ac69a0823d77cf not found: ID does not exist" containerID="fa062ad2e6156a34636710e10d094e493b3942a240b3fbe188ac69a0823d77cf"
Jul 02 21:31:41.463404 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.463364    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa062ad2e6156a34636710e10d094e493b3942a240b3fbe188ac69a0823d77cf"} err="failed to get container status \"fa062ad2e6156a34636710e10d094e493b3942a240b3fbe188ac69a0823d77cf\": rpc error: code = NotFound desc = could not find container \"fa062ad2e6156a34636710e10d094e493b3942a240b3fbe188ac69a0823d77cf\": container with ID starting with fa062ad2e6156a34636710e10d094e493b3942a240b3fbe188ac69a0823d77cf not found: ID does not exist"
Jul 02 21:31:41.466230 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.466199    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-6p6t2"]
Jul 02 21:31:41.553769 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.553747    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"
Jul 02 21:31:41.605096 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.605069    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2n4r\" (UniqueName: \"kubernetes.io/projected/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-kube-api-access-k2n4r\") pod \"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b\" (UID: \"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b\") "
Jul 02 21:31:41.605286 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.605105    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-cabundle-cert\") pod \"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b\" (UID: \"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b\") "
Jul 02 21:31:41.605286 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.605145    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-kube-rbac-proxy-sar-config\") pod \"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b\" (UID: \"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b\") "
Jul 02 21:31:41.605401 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.605278    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-proxy-tls\") pod \"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b\" (UID: \"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b\") "
Jul 02 21:31:41.605401 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.605366    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-kserve-provision-location\") pod \"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b\" (UID: \"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b\") "
Jul 02 21:31:41.605494 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.605456    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" (UID: "d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:31:41.605494 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.605467    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" (UID: "d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:31:41.605707 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.605679    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" (UID: "d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:31:41.605757 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.605745    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-cabundle-cert\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:31:41.605800 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.605759    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:31:41.605800 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.605777    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-kserve-provision-location\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:31:41.607164 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.607147    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-kube-api-access-k2n4r" (OuterVolumeSpecName: "kube-api-access-k2n4r") pod "d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" (UID: "d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b"). InnerVolumeSpecName "kube-api-access-k2n4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:31:41.607164 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.607150    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" (UID: "d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:31:41.706651 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.706572    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2n4r\" (UniqueName: \"kubernetes.io/projected/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-kube-api-access-k2n4r\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:31:41.706651 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:41.706601    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b-proxy-tls\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:31:42.071375 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:42.071341    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07c2fae2-0515-4e71-82a5-e8da54ad9ea7" path="/var/lib/kubelet/pods/07c2fae2-0515-4e71-82a5-e8da54ad9ea7/volumes"
Jul 02 21:31:42.071847 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:42.071835    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d294529b-b6a5-4b4a-ac86-8db2ab88adae" path="/var/lib/kubelet/pods/d294529b-b6a5-4b4a-ac86-8db2ab88adae/volumes"
Jul 02 21:31:42.448422 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:42.448349    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"
Jul 02 21:31:42.448422 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:42.448361    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4" event={"ID":"d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b","Type":"ContainerDied","Data":"9b419c5c85f73b08cf4be2f385adf2719116ad895d64d16028b89d2ce50e945b"}
Jul 02 21:31:42.448422 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:42.448420    2564 scope.go:117] "RemoveContainer" containerID="984fbccad6aea1639bb87075e39b1e13e7de433cbab583ccb3d828f67b55e13e"
Jul 02 21:31:42.456559 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:42.456547    2564 scope.go:117] "RemoveContainer" containerID="0c0f5f408c80a5643c5880090c6c5798958a9768f982ecbab90903cf1de76818"
Jul 02 21:31:42.463792 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:42.463775    2564 scope.go:117] "RemoveContainer" containerID="1d350d04d3582134387b92edb51eff2cc8c386472b27a059c120030fd71022b6"
Jul 02 21:31:42.465821 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:42.465801    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"]
Jul 02 21:31:42.471749 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:42.471734    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-766fdfb6bb-9hls4"]
Jul 02 21:31:43.423094 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:43.423065    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn"
Jul 02 21:31:43.423736 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:43.423708    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn" podUID="83ed67a9-3f3d-4452-a21e-65b6eb3913ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 02 21:31:44.071187 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:44.071152    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" path="/var/lib/kubelet/pods/d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b/volumes"
Jul 02 21:31:47.297879 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:47.297841    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn" podUID="13a08355-135a-43a9-859d-b69891f40231" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 02 21:31:53.424413 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:53.424370    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn" podUID="83ed67a9-3f3d-4452-a21e-65b6eb3913ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 02 21:31:57.299098 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:31:57.299064    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn"
Jul 02 21:32:03.423927 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:32:03.423882    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn" podUID="83ed67a9-3f3d-4452-a21e-65b6eb3913ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 02 21:32:13.424240 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:32:13.424202    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn" podUID="83ed67a9-3f3d-4452-a21e-65b6eb3913ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 02 21:32:23.425262 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:32:23.425185    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn"
Jul 02 21:40:24.827041 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.826958    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn"]
Jul 02 21:40:24.827703 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.827212    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn" podUID="13a08355-135a-43a9-859d-b69891f40231" containerName="kserve-container" containerID="cri-o://bd6e088c024acb44a91c66d9aeff8c58efc03982b119b1246f141ddaa12c563a" gracePeriod=30
Jul 02 21:40:24.827703 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.827250    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn" podUID="13a08355-135a-43a9-859d-b69891f40231" containerName="kube-rbac-proxy" containerID="cri-o://63d27ba02ad4918da9c194f26fbb6f2ab0ccfd172e7b258af5a4881ad11316d2" gracePeriod=30
Jul 02 21:40:24.836438 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.836412    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz"]
Jul 02 21:40:24.836748 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.836736    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07c2fae2-0515-4e71-82a5-e8da54ad9ea7" containerName="kserve-container"
Jul 02 21:40:24.836792 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.836749    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="07c2fae2-0515-4e71-82a5-e8da54ad9ea7" containerName="kserve-container"
Jul 02 21:40:24.836792 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.836764    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" containerName="storage-initializer"
Jul 02 21:40:24.836792 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.836770    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" containerName="storage-initializer"
Jul 02 21:40:24.836792 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.836779    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07c2fae2-0515-4e71-82a5-e8da54ad9ea7" containerName="kube-rbac-proxy"
Jul 02 21:40:24.836792 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.836785    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="07c2fae2-0515-4e71-82a5-e8da54ad9ea7" containerName="kube-rbac-proxy"
Jul 02 21:40:24.836792 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.836794    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d294529b-b6a5-4b4a-ac86-8db2ab88adae" containerName="storage-initializer"
Jul 02 21:40:24.837020 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.836799    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d294529b-b6a5-4b4a-ac86-8db2ab88adae" containerName="storage-initializer"
Jul 02 21:40:24.837020 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.836807    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d294529b-b6a5-4b4a-ac86-8db2ab88adae" containerName="kserve-container"
Jul 02 21:40:24.837020 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.836812    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d294529b-b6a5-4b4a-ac86-8db2ab88adae" containerName="kserve-container"
Jul 02 21:40:24.837020 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.836833    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" containerName="kube-rbac-proxy"
Jul 02 21:40:24.837020 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.836839    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" containerName="kube-rbac-proxy"
Jul 02 21:40:24.837020 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.836850    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" containerName="kserve-container"
Jul 02 21:40:24.837020 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.836855    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" containerName="kserve-container"
Jul 02 21:40:24.837020 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.836863    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07c2fae2-0515-4e71-82a5-e8da54ad9ea7" containerName="storage-initializer"
Jul 02 21:40:24.837020 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.836869    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="07c2fae2-0515-4e71-82a5-e8da54ad9ea7" containerName="storage-initializer"
Jul 02 21:40:24.837020 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.836875    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d294529b-b6a5-4b4a-ac86-8db2ab88adae" containerName="kube-rbac-proxy"
Jul 02 21:40:24.837020 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.836880    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d294529b-b6a5-4b4a-ac86-8db2ab88adae" containerName="kube-rbac-proxy"
Jul 02 21:40:24.837020 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.836928    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" containerName="kserve-container"
Jul 02 21:40:24.837020 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.836934    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d294529b-b6a5-4b4a-ac86-8db2ab88adae" containerName="kserve-container"
Jul 02 21:40:24.837020 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.836940    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d30a040f-b5b2-4ef4-9cc9-a389f7c7c83b" containerName="kube-rbac-proxy"
Jul 02 21:40:24.837020 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.836947    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="07c2fae2-0515-4e71-82a5-e8da54ad9ea7" containerName="kserve-container"
Jul 02 21:40:24.837020 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.836953    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="07c2fae2-0515-4e71-82a5-e8da54ad9ea7" containerName="kube-rbac-proxy"
Jul 02 21:40:24.837020 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.836959    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d294529b-b6a5-4b4a-ac86-8db2ab88adae" containerName="kube-rbac-proxy"
Jul 02 21:40:24.840283 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.840265    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz"
Jul 02 21:40:24.842593 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.842567    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ee70d-kube-rbac-proxy-sar-config\""
Jul 02 21:40:24.842693 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.842612    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ee70d-predictor-serving-cert\""
Jul 02 21:40:24.849921 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.849899    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz"]
Jul 02 21:40:24.932428 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.932389    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4eedd1a9-82ce-49d5-8baa-9061c88c160f-proxy-tls\") pod \"success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz\" (UID: \"4eedd1a9-82ce-49d5-8baa-9061c88c160f\") " pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz"
Jul 02 21:40:24.932590 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.932437    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4eedd1a9-82ce-49d5-8baa-9061c88c160f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz\" (UID: \"4eedd1a9-82ce-49d5-8baa-9061c88c160f\") " pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz"
Jul 02 21:40:24.932590 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:24.932492    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfxmk\" (UniqueName: \"kubernetes.io/projected/4eedd1a9-82ce-49d5-8baa-9061c88c160f-kube-api-access-sfxmk\") pod \"success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz\" (UID: \"4eedd1a9-82ce-49d5-8baa-9061c88c160f\") " pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz"
Jul 02 21:40:25.033533 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:25.033500    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4eedd1a9-82ce-49d5-8baa-9061c88c160f-proxy-tls\") pod \"success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz\" (UID: \"4eedd1a9-82ce-49d5-8baa-9061c88c160f\") " pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz"
Jul 02 21:40:25.033735 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:25.033552    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4eedd1a9-82ce-49d5-8baa-9061c88c160f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz\" (UID: \"4eedd1a9-82ce-49d5-8baa-9061c88c160f\") " pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz"
Jul 02 21:40:25.033735 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:25.033588    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfxmk\" (UniqueName: \"kubernetes.io/projected/4eedd1a9-82ce-49d5-8baa-9061c88c160f-kube-api-access-sfxmk\") pod \"success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz\" (UID: \"4eedd1a9-82ce-49d5-8baa-9061c88c160f\") " pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz"
Jul 02 21:40:25.034310 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:25.034285    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4eedd1a9-82ce-49d5-8baa-9061c88c160f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz\" (UID: \"4eedd1a9-82ce-49d5-8baa-9061c88c160f\") " pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz"
Jul 02 21:40:25.036176 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:25.036143    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4eedd1a9-82ce-49d5-8baa-9061c88c160f-proxy-tls\") pod \"success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz\" (UID: \"4eedd1a9-82ce-49d5-8baa-9061c88c160f\") " pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz"
Jul 02 21:40:25.041410 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:25.041387    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfxmk\" (UniqueName: \"kubernetes.io/projected/4eedd1a9-82ce-49d5-8baa-9061c88c160f-kube-api-access-sfxmk\") pod \"success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz\" (UID: \"4eedd1a9-82ce-49d5-8baa-9061c88c160f\") " pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz"
Jul 02 21:40:25.151823 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:25.151741    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz"
Jul 02 21:40:25.280354 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:25.280311    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz"]
Jul 02 21:40:25.283213 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:40:25.283182    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eedd1a9_82ce_49d5_8baa_9061c88c160f.slice/crio-47262e80668409fbdd3b618c25d5928717cd9adbc682d7373e453bbede64f280 WatchSource:0}: Error finding container 47262e80668409fbdd3b618c25d5928717cd9adbc682d7373e453bbede64f280: Status 404 returned error can't find the container with id 47262e80668409fbdd3b618c25d5928717cd9adbc682d7373e453bbede64f280
Jul 02 21:40:25.602821 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:25.602790    2564 generic.go:356] "Generic (PLEG): container finished" podID="13a08355-135a-43a9-859d-b69891f40231" containerID="63d27ba02ad4918da9c194f26fbb6f2ab0ccfd172e7b258af5a4881ad11316d2" exitCode=2
Jul 02 21:40:25.603001 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:25.602856    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn" event={"ID":"13a08355-135a-43a9-859d-b69891f40231","Type":"ContainerDied","Data":"63d27ba02ad4918da9c194f26fbb6f2ab0ccfd172e7b258af5a4881ad11316d2"}
Jul 02 21:40:25.604593 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:25.604572    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz" event={"ID":"4eedd1a9-82ce-49d5-8baa-9061c88c160f","Type":"ContainerStarted","Data":"dbbe662fca94ae8d148f351d0515a34b5a719fe0c7ae21e109695e999c46d531"}
Jul 02 21:40:25.604720 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:25.604616    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz" event={"ID":"4eedd1a9-82ce-49d5-8baa-9061c88c160f","Type":"ContainerStarted","Data":"6ea387e9a7ce3918364d59828807265c06d493990878bae03f7d290df21cdaa7"}
Jul 02 21:40:25.604720 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:25.604640    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz" event={"ID":"4eedd1a9-82ce-49d5-8baa-9061c88c160f","Type":"ContainerStarted","Data":"47262e80668409fbdd3b618c25d5928717cd9adbc682d7373e453bbede64f280"}
Jul 02 21:40:25.604720 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:25.604662    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz"
Jul 02 21:40:25.604720 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:25.604715    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz"
Jul 02 21:40:25.606134 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:25.606096    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz" podUID="4eedd1a9-82ce-49d5-8baa-9061c88c160f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 02 21:40:25.622461 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:25.622408    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz" podStartSLOduration=1.622387492 podStartE2EDuration="1.622387492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:40:25.619889597 +0000 UTC m=+1286.457129672" watchObservedRunningTime="2026-07-02 21:40:25.622387492 +0000 UTC m=+1286.459627538"
Jul 02 21:40:26.608447 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:26.608404    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz" podUID="4eedd1a9-82ce-49d5-8baa-9061c88c160f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 02 21:40:27.292495 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:27.292454    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn" podUID="13a08355-135a-43a9-859d-b69891f40231" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 02 21:40:27.297806 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:27.297779    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn" podUID="13a08355-135a-43a9-859d-b69891f40231" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 02 21:40:28.178161 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:28.178138    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn"
Jul 02 21:40:28.260706 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:28.260676    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13a08355-135a-43a9-859d-b69891f40231-proxy-tls\") pod \"13a08355-135a-43a9-859d-b69891f40231\" (UID: \"13a08355-135a-43a9-859d-b69891f40231\") "
Jul 02 21:40:28.260841 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:28.260776    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9w6q\" (UniqueName: \"kubernetes.io/projected/13a08355-135a-43a9-859d-b69891f40231-kube-api-access-f9w6q\") pod \"13a08355-135a-43a9-859d-b69891f40231\" (UID: \"13a08355-135a-43a9-859d-b69891f40231\") "
Jul 02 21:40:28.260841 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:28.260809    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13a08355-135a-43a9-859d-b69891f40231-kube-rbac-proxy-sar-config\") pod \"13a08355-135a-43a9-859d-b69891f40231\" (UID: \"13a08355-135a-43a9-859d-b69891f40231\") "
Jul 02 21:40:28.261230 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:28.261198    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13a08355-135a-43a9-859d-b69891f40231-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "13a08355-135a-43a9-859d-b69891f40231" (UID: "13a08355-135a-43a9-859d-b69891f40231"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:40:28.262728 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:28.262697    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a08355-135a-43a9-859d-b69891f40231-kube-api-access-f9w6q" (OuterVolumeSpecName: "kube-api-access-f9w6q") pod "13a08355-135a-43a9-859d-b69891f40231" (UID: "13a08355-135a-43a9-859d-b69891f40231"). InnerVolumeSpecName "kube-api-access-f9w6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:40:28.262728 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:28.262711    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13a08355-135a-43a9-859d-b69891f40231-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "13a08355-135a-43a9-859d-b69891f40231" (UID: "13a08355-135a-43a9-859d-b69891f40231"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:40:28.361661 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:28.361633    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13a08355-135a-43a9-859d-b69891f40231-proxy-tls\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:40:28.361661 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:28.361657    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9w6q\" (UniqueName: \"kubernetes.io/projected/13a08355-135a-43a9-859d-b69891f40231-kube-api-access-f9w6q\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:40:28.361839 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:28.361668    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13a08355-135a-43a9-859d-b69891f40231-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:40:28.617454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:28.617371    2564 generic.go:356] "Generic (PLEG): container finished" podID="13a08355-135a-43a9-859d-b69891f40231" containerID="bd6e088c024acb44a91c66d9aeff8c58efc03982b119b1246f141ddaa12c563a" exitCode=0
Jul 02 21:40:28.617454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:28.617419    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn"
Jul 02 21:40:28.617454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:28.617443    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn" event={"ID":"13a08355-135a-43a9-859d-b69891f40231","Type":"ContainerDied","Data":"bd6e088c024acb44a91c66d9aeff8c58efc03982b119b1246f141ddaa12c563a"}
Jul 02 21:40:28.617682 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:28.617490    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn" event={"ID":"13a08355-135a-43a9-859d-b69891f40231","Type":"ContainerDied","Data":"7af616d316b89159ef61dd65d0834a770293bb0fdd3dc6fbbdd7a893162305c2"}
Jul 02 21:40:28.617682 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:28.617515    2564 scope.go:117] "RemoveContainer" containerID="63d27ba02ad4918da9c194f26fbb6f2ab0ccfd172e7b258af5a4881ad11316d2"
Jul 02 21:40:28.626459 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:28.626441    2564 scope.go:117] "RemoveContainer" containerID="bd6e088c024acb44a91c66d9aeff8c58efc03982b119b1246f141ddaa12c563a"
Jul 02 21:40:28.635624 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:28.635602    2564 scope.go:117] "RemoveContainer" containerID="63d27ba02ad4918da9c194f26fbb6f2ab0ccfd172e7b258af5a4881ad11316d2"
Jul 02 21:40:28.635899 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:40:28.635878    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63d27ba02ad4918da9c194f26fbb6f2ab0ccfd172e7b258af5a4881ad11316d2\": container with ID starting with 63d27ba02ad4918da9c194f26fbb6f2ab0ccfd172e7b258af5a4881ad11316d2 not found: ID does not exist" containerID="63d27ba02ad4918da9c194f26fbb6f2ab0ccfd172e7b258af5a4881ad11316d2"
Jul 02 21:40:28.635956 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:28.635909    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d27ba02ad4918da9c194f26fbb6f2ab0ccfd172e7b258af5a4881ad11316d2"} err="failed to get container status \"63d27ba02ad4918da9c194f26fbb6f2ab0ccfd172e7b258af5a4881ad11316d2\": rpc error: code = NotFound desc = could not find container \"63d27ba02ad4918da9c194f26fbb6f2ab0ccfd172e7b258af5a4881ad11316d2\": container with ID starting with 63d27ba02ad4918da9c194f26fbb6f2ab0ccfd172e7b258af5a4881ad11316d2 not found: ID does not exist"
Jul 02 21:40:28.635956 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:28.635926    2564 scope.go:117] "RemoveContainer" containerID="bd6e088c024acb44a91c66d9aeff8c58efc03982b119b1246f141ddaa12c563a"
Jul 02 21:40:28.636189 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:40:28.636167    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd6e088c024acb44a91c66d9aeff8c58efc03982b119b1246f141ddaa12c563a\": container with ID starting with bd6e088c024acb44a91c66d9aeff8c58efc03982b119b1246f141ddaa12c563a not found: ID does not exist" containerID="bd6e088c024acb44a91c66d9aeff8c58efc03982b119b1246f141ddaa12c563a"
Jul 02 21:40:28.636254 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:28.636197    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6e088c024acb44a91c66d9aeff8c58efc03982b119b1246f141ddaa12c563a"} err="failed to get container status \"bd6e088c024acb44a91c66d9aeff8c58efc03982b119b1246f141ddaa12c563a\": rpc error: code = NotFound desc = could not find container \"bd6e088c024acb44a91c66d9aeff8c58efc03982b119b1246f141ddaa12c563a\": container with ID starting with bd6e088c024acb44a91c66d9aeff8c58efc03982b119b1246f141ddaa12c563a not found: ID does not exist"
Jul 02 21:40:28.638714 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:28.638692    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn"]
Jul 02 21:40:28.642594 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:28.642574    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7615e-predictor-6fc7996664-4pfhn"]
Jul 02 21:40:30.072219 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:30.072178    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13a08355-135a-43a9-859d-b69891f40231" path="/var/lib/kubelet/pods/13a08355-135a-43a9-859d-b69891f40231/volumes"
Jul 02 21:40:31.612495 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:31.612463    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz"
Jul 02 21:40:31.612938 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:31.612877    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz" podUID="4eedd1a9-82ce-49d5-8baa-9061c88c160f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 02 21:40:41.613355 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:41.613313    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz" podUID="4eedd1a9-82ce-49d5-8baa-9061c88c160f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 02 21:40:50.723805 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:50.723768    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn"]
Jul 02 21:40:50.724238 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:50.723970    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn" podUID="83ed67a9-3f3d-4452-a21e-65b6eb3913ec" containerName="kserve-container" containerID="cri-o://5557d6e3e8f78e26bfdab5592c3c32b5887d44d713b64d4c097bfa4277c39d7d" gracePeriod=30
Jul 02 21:40:50.724238 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:50.724031    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn" podUID="83ed67a9-3f3d-4452-a21e-65b6eb3913ec" containerName="kube-rbac-proxy" containerID="cri-o://a22ae178fe4165b230ffe90d591d6700dcaf048fad2820be74748127b27ad1ed" gracePeriod=30
Jul 02 21:40:50.752026 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:50.751999    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s"]
Jul 02 21:40:50.752389 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:50.752376    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13a08355-135a-43a9-859d-b69891f40231" containerName="kserve-container"
Jul 02 21:40:50.752438 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:50.752392    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="13a08355-135a-43a9-859d-b69891f40231" containerName="kserve-container"
Jul 02 21:40:50.752438 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:50.752413    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13a08355-135a-43a9-859d-b69891f40231" containerName="kube-rbac-proxy"
Jul 02 21:40:50.752438 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:50.752418    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="13a08355-135a-43a9-859d-b69891f40231" containerName="kube-rbac-proxy"
Jul 02 21:40:50.752527 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:50.752466    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="13a08355-135a-43a9-859d-b69891f40231" containerName="kube-rbac-proxy"
Jul 02 21:40:50.752527 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:50.752476    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="13a08355-135a-43a9-859d-b69891f40231" containerName="kserve-container"
Jul 02 21:40:50.757128 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:50.757094    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s"
Jul 02 21:40:50.759702 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:50.759682    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1702f-predictor-serving-cert\""
Jul 02 21:40:50.759835 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:50.759817    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1702f-kube-rbac-proxy-sar-config\""
Jul 02 21:40:50.778623 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:50.778601    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s"]
Jul 02 21:40:50.849122 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:50.849081    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvrm4\" (UniqueName: \"kubernetes.io/projected/08b59517-c227-4b47-9824-7ce0c25e6707-kube-api-access-vvrm4\") pod \"success-200-isvc-1702f-predictor-d85f8dbd4-57r4s\" (UID: \"08b59517-c227-4b47-9824-7ce0c25e6707\") " pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s"
Jul 02 21:40:50.849275 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:50.849137    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08b59517-c227-4b47-9824-7ce0c25e6707-proxy-tls\") pod \"success-200-isvc-1702f-predictor-d85f8dbd4-57r4s\" (UID: \"08b59517-c227-4b47-9824-7ce0c25e6707\") " pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s"
Jul 02 21:40:50.849345 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:50.849270    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08b59517-c227-4b47-9824-7ce0c25e6707-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1702f-predictor-d85f8dbd4-57r4s\" (UID: \"08b59517-c227-4b47-9824-7ce0c25e6707\") " pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s"
Jul 02 21:40:50.949904 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:50.949874    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08b59517-c227-4b47-9824-7ce0c25e6707-proxy-tls\") pod \"success-200-isvc-1702f-predictor-d85f8dbd4-57r4s\" (UID: \"08b59517-c227-4b47-9824-7ce0c25e6707\") " pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s"
Jul 02 21:40:50.950075 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:40:50.950040    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-1702f-predictor-serving-cert: secret "success-200-isvc-1702f-predictor-serving-cert" not found
Jul 02 21:40:50.950075 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:50.950048    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08b59517-c227-4b47-9824-7ce0c25e6707-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1702f-predictor-d85f8dbd4-57r4s\" (UID: \"08b59517-c227-4b47-9824-7ce0c25e6707\") " pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s"
Jul 02 21:40:50.950205 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:40:50.950149    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08b59517-c227-4b47-9824-7ce0c25e6707-proxy-tls podName:08b59517-c227-4b47-9824-7ce0c25e6707 nodeName:}" failed. No retries permitted until 2026-07-02 21:40:51.450102858 +0000 UTC m=+1312.287342886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/08b59517-c227-4b47-9824-7ce0c25e6707-proxy-tls") pod "success-200-isvc-1702f-predictor-d85f8dbd4-57r4s" (UID: "08b59517-c227-4b47-9824-7ce0c25e6707") : secret "success-200-isvc-1702f-predictor-serving-cert" not found
Jul 02 21:40:50.950205 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:50.950182    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvrm4\" (UniqueName: \"kubernetes.io/projected/08b59517-c227-4b47-9824-7ce0c25e6707-kube-api-access-vvrm4\") pod \"success-200-isvc-1702f-predictor-d85f8dbd4-57r4s\" (UID: \"08b59517-c227-4b47-9824-7ce0c25e6707\") " pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s"
Jul 02 21:40:50.950686 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:50.950667    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08b59517-c227-4b47-9824-7ce0c25e6707-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1702f-predictor-d85f8dbd4-57r4s\" (UID: \"08b59517-c227-4b47-9824-7ce0c25e6707\") " pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s"
Jul 02 21:40:50.960105 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:50.960078    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvrm4\" (UniqueName: \"kubernetes.io/projected/08b59517-c227-4b47-9824-7ce0c25e6707-kube-api-access-vvrm4\") pod \"success-200-isvc-1702f-predictor-d85f8dbd4-57r4s\" (UID: \"08b59517-c227-4b47-9824-7ce0c25e6707\") " pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s"
Jul 02 21:40:51.454452 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:51.454419    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08b59517-c227-4b47-9824-7ce0c25e6707-proxy-tls\") pod \"success-200-isvc-1702f-predictor-d85f8dbd4-57r4s\" (UID: \"08b59517-c227-4b47-9824-7ce0c25e6707\") " pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s"
Jul 02 21:40:51.456656 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:51.456628    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08b59517-c227-4b47-9824-7ce0c25e6707-proxy-tls\") pod \"success-200-isvc-1702f-predictor-d85f8dbd4-57r4s\" (UID: \"08b59517-c227-4b47-9824-7ce0c25e6707\") " pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s"
Jul 02 21:40:51.613631 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:51.613590    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz" podUID="4eedd1a9-82ce-49d5-8baa-9061c88c160f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 02 21:40:51.667838 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:51.667805    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s"
Jul 02 21:40:51.712558 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:51.712488    2564 generic.go:356] "Generic (PLEG): container finished" podID="83ed67a9-3f3d-4452-a21e-65b6eb3913ec" containerID="a22ae178fe4165b230ffe90d591d6700dcaf048fad2820be74748127b27ad1ed" exitCode=2
Jul 02 21:40:51.712680 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:51.712563    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn" event={"ID":"83ed67a9-3f3d-4452-a21e-65b6eb3913ec","Type":"ContainerDied","Data":"a22ae178fe4165b230ffe90d591d6700dcaf048fad2820be74748127b27ad1ed"}
Jul 02 21:40:51.794564 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:51.794540    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s"]
Jul 02 21:40:51.796728 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:40:51.796695    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b59517_c227_4b47_9824_7ce0c25e6707.slice/crio-71f4b77073de5961a2d71b82adf653a14f3150e6b07ca0a96719a81f6c7a7667 WatchSource:0}: Error finding container 71f4b77073de5961a2d71b82adf653a14f3150e6b07ca0a96719a81f6c7a7667: Status 404 returned error can't find the container with id 71f4b77073de5961a2d71b82adf653a14f3150e6b07ca0a96719a81f6c7a7667
Jul 02 21:40:52.718487 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:52.718450    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s" event={"ID":"08b59517-c227-4b47-9824-7ce0c25e6707","Type":"ContainerStarted","Data":"88a824f30bdbca4ded28564ed8d523155f124c0c587b0ceece059311e2b57e40"}
Jul 02 21:40:52.718824 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:52.718533    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s" event={"ID":"08b59517-c227-4b47-9824-7ce0c25e6707","Type":"ContainerStarted","Data":"961012cfd3611a668fc02278e30babbaea7bba916ee986fd02b400d432176a4d"}
Jul 02 21:40:52.718824 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:52.718555    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s" event={"ID":"08b59517-c227-4b47-9824-7ce0c25e6707","Type":"ContainerStarted","Data":"71f4b77073de5961a2d71b82adf653a14f3150e6b07ca0a96719a81f6c7a7667"}
Jul 02 21:40:52.718824 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:52.718702    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s"
Jul 02 21:40:52.737124 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:52.737045    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s" podStartSLOduration=2.737025569 podStartE2EDuration="2.737025569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:40:52.735025704 +0000 UTC m=+1313.572265763" watchObservedRunningTime="2026-07-02 21:40:52.737025569 +0000 UTC m=+1313.574265616"
Jul 02 21:40:53.418594 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:53.418560    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn" podUID="83ed67a9-3f3d-4452-a21e-65b6eb3913ec" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 02 21:40:53.423888 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:53.423866    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn" podUID="83ed67a9-3f3d-4452-a21e-65b6eb3913ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 02 21:40:53.722455 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:53.722380    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s"
Jul 02 21:40:53.723594 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:53.723571    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s" podUID="08b59517-c227-4b47-9824-7ce0c25e6707" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 02 21:40:53.970872 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:53.970848    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn"
Jul 02 21:40:54.078972 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:54.078941    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83ed67a9-3f3d-4452-a21e-65b6eb3913ec-proxy-tls\") pod \"83ed67a9-3f3d-4452-a21e-65b6eb3913ec\" (UID: \"83ed67a9-3f3d-4452-a21e-65b6eb3913ec\") "
Jul 02 21:40:54.079167 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:54.078992    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83ed67a9-3f3d-4452-a21e-65b6eb3913ec-kube-rbac-proxy-sar-config\") pod \"83ed67a9-3f3d-4452-a21e-65b6eb3913ec\" (UID: \"83ed67a9-3f3d-4452-a21e-65b6eb3913ec\") "
Jul 02 21:40:54.079167 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:54.079106    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkft6\" (UniqueName: \"kubernetes.io/projected/83ed67a9-3f3d-4452-a21e-65b6eb3913ec-kube-api-access-mkft6\") pod \"83ed67a9-3f3d-4452-a21e-65b6eb3913ec\" (UID: \"83ed67a9-3f3d-4452-a21e-65b6eb3913ec\") "
Jul 02 21:40:54.079362 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:54.079333    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83ed67a9-3f3d-4452-a21e-65b6eb3913ec-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "83ed67a9-3f3d-4452-a21e-65b6eb3913ec" (UID: "83ed67a9-3f3d-4452-a21e-65b6eb3913ec"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:40:54.080977 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:54.080949    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83ed67a9-3f3d-4452-a21e-65b6eb3913ec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "83ed67a9-3f3d-4452-a21e-65b6eb3913ec" (UID: "83ed67a9-3f3d-4452-a21e-65b6eb3913ec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:40:54.081057 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:54.080999    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ed67a9-3f3d-4452-a21e-65b6eb3913ec-kube-api-access-mkft6" (OuterVolumeSpecName: "kube-api-access-mkft6") pod "83ed67a9-3f3d-4452-a21e-65b6eb3913ec" (UID: "83ed67a9-3f3d-4452-a21e-65b6eb3913ec"). InnerVolumeSpecName "kube-api-access-mkft6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:40:54.179984 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:54.179909    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkft6\" (UniqueName: \"kubernetes.io/projected/83ed67a9-3f3d-4452-a21e-65b6eb3913ec-kube-api-access-mkft6\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:40:54.179984 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:54.179939    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83ed67a9-3f3d-4452-a21e-65b6eb3913ec-proxy-tls\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:40:54.179984 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:54.179954    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83ed67a9-3f3d-4452-a21e-65b6eb3913ec-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:40:54.728029 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:54.727995    2564 generic.go:356] "Generic (PLEG): container finished" podID="83ed67a9-3f3d-4452-a21e-65b6eb3913ec" containerID="5557d6e3e8f78e26bfdab5592c3c32b5887d44d713b64d4c097bfa4277c39d7d" exitCode=0
Jul 02 21:40:54.728516 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:54.728050    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn"
Jul 02 21:40:54.728516 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:54.728076    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn" event={"ID":"83ed67a9-3f3d-4452-a21e-65b6eb3913ec","Type":"ContainerDied","Data":"5557d6e3e8f78e26bfdab5592c3c32b5887d44d713b64d4c097bfa4277c39d7d"}
Jul 02 21:40:54.728516 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:54.728148    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn" event={"ID":"83ed67a9-3f3d-4452-a21e-65b6eb3913ec","Type":"ContainerDied","Data":"afb59cc280bd0fa1d2aa9100d10eb6b40d9c5d1268572916b4438023013040a8"}
Jul 02 21:40:54.728516 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:54.728175    2564 scope.go:117] "RemoveContainer" containerID="a22ae178fe4165b230ffe90d591d6700dcaf048fad2820be74748127b27ad1ed"
Jul 02 21:40:54.728697 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:54.728606    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s" podUID="08b59517-c227-4b47-9824-7ce0c25e6707" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 02 21:40:54.738796 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:54.738779    2564 scope.go:117] "RemoveContainer" containerID="5557d6e3e8f78e26bfdab5592c3c32b5887d44d713b64d4c097bfa4277c39d7d"
Jul 02 21:40:54.746889 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:54.746864    2564 scope.go:117] "RemoveContainer" containerID="a22ae178fe4165b230ffe90d591d6700dcaf048fad2820be74748127b27ad1ed"
Jul 02 21:40:54.747180 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:40:54.747158    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a22ae178fe4165b230ffe90d591d6700dcaf048fad2820be74748127b27ad1ed\": container with ID starting with a22ae178fe4165b230ffe90d591d6700dcaf048fad2820be74748127b27ad1ed not found: ID does not exist" containerID="a22ae178fe4165b230ffe90d591d6700dcaf048fad2820be74748127b27ad1ed"
Jul 02 21:40:54.747281 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:54.747191    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22ae178fe4165b230ffe90d591d6700dcaf048fad2820be74748127b27ad1ed"} err="failed to get container status \"a22ae178fe4165b230ffe90d591d6700dcaf048fad2820be74748127b27ad1ed\": rpc error: code = NotFound desc = could not find container \"a22ae178fe4165b230ffe90d591d6700dcaf048fad2820be74748127b27ad1ed\": container with ID starting with a22ae178fe4165b230ffe90d591d6700dcaf048fad2820be74748127b27ad1ed not found: ID does not exist"
Jul 02 21:40:54.747281 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:54.747215    2564 scope.go:117] "RemoveContainer" containerID="5557d6e3e8f78e26bfdab5592c3c32b5887d44d713b64d4c097bfa4277c39d7d"
Jul 02 21:40:54.747487 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:40:54.747470    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5557d6e3e8f78e26bfdab5592c3c32b5887d44d713b64d4c097bfa4277c39d7d\": container with ID starting with 5557d6e3e8f78e26bfdab5592c3c32b5887d44d713b64d4c097bfa4277c39d7d not found: ID does not exist" containerID="5557d6e3e8f78e26bfdab5592c3c32b5887d44d713b64d4c097bfa4277c39d7d"
Jul 02 21:40:54.747534 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:54.747494    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5557d6e3e8f78e26bfdab5592c3c32b5887d44d713b64d4c097bfa4277c39d7d"} err="failed to get container status \"5557d6e3e8f78e26bfdab5592c3c32b5887d44d713b64d4c097bfa4277c39d7d\": rpc error: code = NotFound desc = could not find container \"5557d6e3e8f78e26bfdab5592c3c32b5887d44d713b64d4c097bfa4277c39d7d\": container with ID starting with 5557d6e3e8f78e26bfdab5592c3c32b5887d44d713b64d4c097bfa4277c39d7d not found: ID does not exist"
Jul 02 21:40:54.748079 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:54.748061    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn"]
Jul 02 21:40:54.750492 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:54.750471    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a3b0c-predictor-7ccb86bd58-qfxvn"]
Jul 02 21:40:56.071750 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:56.071719    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83ed67a9-3f3d-4452-a21e-65b6eb3913ec" path="/var/lib/kubelet/pods/83ed67a9-3f3d-4452-a21e-65b6eb3913ec/volumes"
Jul 02 21:40:59.733889 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:59.733856    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s"
Jul 02 21:40:59.734525 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:40:59.734496    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s" podUID="08b59517-c227-4b47-9824-7ce0c25e6707" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 02 21:41:01.613093 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:01.613055    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz" podUID="4eedd1a9-82ce-49d5-8baa-9061c88c160f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 02 21:41:09.734688 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:09.734648    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s" podUID="08b59517-c227-4b47-9824-7ce0c25e6707" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 02 21:41:11.613514 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:11.613487    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz"
Jul 02 21:41:19.734858 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:19.734817    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s" podUID="08b59517-c227-4b47-9824-7ce0c25e6707" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 02 21:41:29.734743 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:29.734692    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s" podUID="08b59517-c227-4b47-9824-7ce0c25e6707" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 02 21:41:35.076683 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.076647    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz"]
Jul 02 21:41:35.077146 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.076859    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz" podUID="4eedd1a9-82ce-49d5-8baa-9061c88c160f" containerName="kserve-container" containerID="cri-o://6ea387e9a7ce3918364d59828807265c06d493990878bae03f7d290df21cdaa7" gracePeriod=30
Jul 02 21:41:35.077146 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.076918    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz" podUID="4eedd1a9-82ce-49d5-8baa-9061c88c160f" containerName="kube-rbac-proxy" containerID="cri-o://dbbe662fca94ae8d148f351d0515a34b5a719fe0c7ae21e109695e999c46d531" gracePeriod=30
Jul 02 21:41:35.101283 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.101260    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd"]
Jul 02 21:41:35.101624 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.101611    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83ed67a9-3f3d-4452-a21e-65b6eb3913ec" containerName="kube-rbac-proxy"
Jul 02 21:41:35.101675 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.101627    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="83ed67a9-3f3d-4452-a21e-65b6eb3913ec" containerName="kube-rbac-proxy"
Jul 02 21:41:35.101675 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.101643    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83ed67a9-3f3d-4452-a21e-65b6eb3913ec" containerName="kserve-container"
Jul 02 21:41:35.101675 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.101649    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="83ed67a9-3f3d-4452-a21e-65b6eb3913ec" containerName="kserve-container"
Jul 02 21:41:35.101776 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.101702    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="83ed67a9-3f3d-4452-a21e-65b6eb3913ec" containerName="kube-rbac-proxy"
Jul 02 21:41:35.101776 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.101713    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="83ed67a9-3f3d-4452-a21e-65b6eb3913ec" containerName="kserve-container"
Jul 02 21:41:35.106421 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.106094    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd"
Jul 02 21:41:35.109164 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.109139    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-87dcc-predictor-serving-cert\""
Jul 02 21:41:35.109267 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.109257    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-87dcc-kube-rbac-proxy-sar-config\""
Jul 02 21:41:35.118441 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.118417    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd"]
Jul 02 21:41:35.217841 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.217810    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vf9s\" (UniqueName: \"kubernetes.io/projected/fd57752b-4d3f-4887-9433-f69b266be07c-kube-api-access-6vf9s\") pod \"success-200-isvc-87dcc-predictor-648644cd6-hvcgd\" (UID: \"fd57752b-4d3f-4887-9433-f69b266be07c\") " pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd"
Jul 02 21:41:35.218010 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.217916    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd57752b-4d3f-4887-9433-f69b266be07c-proxy-tls\") pod \"success-200-isvc-87dcc-predictor-648644cd6-hvcgd\" (UID: \"fd57752b-4d3f-4887-9433-f69b266be07c\") " pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd"
Jul 02 21:41:35.218104 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.218083    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd57752b-4d3f-4887-9433-f69b266be07c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-87dcc-predictor-648644cd6-hvcgd\" (UID: \"fd57752b-4d3f-4887-9433-f69b266be07c\") " pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd"
Jul 02 21:41:35.318984 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.318951    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd57752b-4d3f-4887-9433-f69b266be07c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-87dcc-predictor-648644cd6-hvcgd\" (UID: \"fd57752b-4d3f-4887-9433-f69b266be07c\") " pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd"
Jul 02 21:41:35.319201 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.319028    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vf9s\" (UniqueName: \"kubernetes.io/projected/fd57752b-4d3f-4887-9433-f69b266be07c-kube-api-access-6vf9s\") pod \"success-200-isvc-87dcc-predictor-648644cd6-hvcgd\" (UID: \"fd57752b-4d3f-4887-9433-f69b266be07c\") " pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd"
Jul 02 21:41:35.319201 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.319058    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd57752b-4d3f-4887-9433-f69b266be07c-proxy-tls\") pod \"success-200-isvc-87dcc-predictor-648644cd6-hvcgd\" (UID: \"fd57752b-4d3f-4887-9433-f69b266be07c\") " pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd"
Jul 02 21:41:35.319743 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.319718    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd57752b-4d3f-4887-9433-f69b266be07c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-87dcc-predictor-648644cd6-hvcgd\" (UID: \"fd57752b-4d3f-4887-9433-f69b266be07c\") " pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd"
Jul 02 21:41:35.321486 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.321461    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd57752b-4d3f-4887-9433-f69b266be07c-proxy-tls\") pod \"success-200-isvc-87dcc-predictor-648644cd6-hvcgd\" (UID: \"fd57752b-4d3f-4887-9433-f69b266be07c\") " pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd"
Jul 02 21:41:35.327798 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.327751    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vf9s\" (UniqueName: \"kubernetes.io/projected/fd57752b-4d3f-4887-9433-f69b266be07c-kube-api-access-6vf9s\") pod \"success-200-isvc-87dcc-predictor-648644cd6-hvcgd\" (UID: \"fd57752b-4d3f-4887-9433-f69b266be07c\") " pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd"
Jul 02 21:41:35.416434 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.416386    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd"
Jul 02 21:41:35.546154 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.546096    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd"]
Jul 02 21:41:35.549101 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:41:35.549075    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd57752b_4d3f_4887_9433_f69b266be07c.slice/crio-c3716122c19373a934b360c7933e5acd16290cebd741f8628bbaea5ec7b2680a WatchSource:0}: Error finding container c3716122c19373a934b360c7933e5acd16290cebd741f8628bbaea5ec7b2680a: Status 404 returned error can't find the container with id c3716122c19373a934b360c7933e5acd16290cebd741f8628bbaea5ec7b2680a
Jul 02 21:41:35.897628 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.897546    2564 generic.go:356] "Generic (PLEG): container finished" podID="4eedd1a9-82ce-49d5-8baa-9061c88c160f" containerID="dbbe662fca94ae8d148f351d0515a34b5a719fe0c7ae21e109695e999c46d531" exitCode=2
Jul 02 21:41:35.897628 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.897618    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz" event={"ID":"4eedd1a9-82ce-49d5-8baa-9061c88c160f","Type":"ContainerDied","Data":"dbbe662fca94ae8d148f351d0515a34b5a719fe0c7ae21e109695e999c46d531"}
Jul 02 21:41:35.899235 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.899211    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd" event={"ID":"fd57752b-4d3f-4887-9433-f69b266be07c","Type":"ContainerStarted","Data":"8509fcf9ada29fe5eb164446e744ab4f181ce5a4515d66b35b15e52d4f181b1d"}
Jul 02 21:41:35.899359 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.899263    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd" event={"ID":"fd57752b-4d3f-4887-9433-f69b266be07c","Type":"ContainerStarted","Data":"56889a8a3916ea9e04c770bc4433e6de9275b7697bd36e57240323ed0f09c177"}
Jul 02 21:41:35.899359 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.899289    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd"
Jul 02 21:41:35.899359 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.899308    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd" event={"ID":"fd57752b-4d3f-4887-9433-f69b266be07c","Type":"ContainerStarted","Data":"c3716122c19373a934b360c7933e5acd16290cebd741f8628bbaea5ec7b2680a"}
Jul 02 21:41:35.899359 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.899335    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd"
Jul 02 21:41:35.900631 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.900610    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd" podUID="fd57752b-4d3f-4887-9433-f69b266be07c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 02 21:41:35.916668 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:35.916624    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd" podStartSLOduration=0.916610039 podStartE2EDuration="916.610039ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:41:35.915207852 +0000 UTC m=+1356.752447898" watchObservedRunningTime="2026-07-02 21:41:35.916610039 +0000 UTC m=+1356.753850086"
Jul 02 21:41:36.608864 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:36.608818    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz" podUID="4eedd1a9-82ce-49d5-8baa-9061c88c160f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 02 21:41:36.903408 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:36.903323    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd" podUID="fd57752b-4d3f-4887-9433-f69b266be07c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 02 21:41:38.123690 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:38.123666    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz"
Jul 02 21:41:38.243993 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:38.243962    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4eedd1a9-82ce-49d5-8baa-9061c88c160f-kube-rbac-proxy-sar-config\") pod \"4eedd1a9-82ce-49d5-8baa-9061c88c160f\" (UID: \"4eedd1a9-82ce-49d5-8baa-9061c88c160f\") "
Jul 02 21:41:38.244172 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:38.244018    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfxmk\" (UniqueName: \"kubernetes.io/projected/4eedd1a9-82ce-49d5-8baa-9061c88c160f-kube-api-access-sfxmk\") pod \"4eedd1a9-82ce-49d5-8baa-9061c88c160f\" (UID: \"4eedd1a9-82ce-49d5-8baa-9061c88c160f\") "
Jul 02 21:41:38.244172 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:38.244041    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4eedd1a9-82ce-49d5-8baa-9061c88c160f-proxy-tls\") pod \"4eedd1a9-82ce-49d5-8baa-9061c88c160f\" (UID: \"4eedd1a9-82ce-49d5-8baa-9061c88c160f\") "
Jul 02 21:41:38.244430 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:38.244400    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eedd1a9-82ce-49d5-8baa-9061c88c160f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4eedd1a9-82ce-49d5-8baa-9061c88c160f" (UID: "4eedd1a9-82ce-49d5-8baa-9061c88c160f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:41:38.246153 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:38.246128    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eedd1a9-82ce-49d5-8baa-9061c88c160f-kube-api-access-sfxmk" (OuterVolumeSpecName: "kube-api-access-sfxmk") pod "4eedd1a9-82ce-49d5-8baa-9061c88c160f" (UID: "4eedd1a9-82ce-49d5-8baa-9061c88c160f"). InnerVolumeSpecName "kube-api-access-sfxmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:41:38.246241 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:38.246133    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eedd1a9-82ce-49d5-8baa-9061c88c160f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4eedd1a9-82ce-49d5-8baa-9061c88c160f" (UID: "4eedd1a9-82ce-49d5-8baa-9061c88c160f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:41:38.345423 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:38.345393    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4eedd1a9-82ce-49d5-8baa-9061c88c160f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:41:38.345423 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:38.345417    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfxmk\" (UniqueName: \"kubernetes.io/projected/4eedd1a9-82ce-49d5-8baa-9061c88c160f-kube-api-access-sfxmk\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:41:38.345423 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:38.345427    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4eedd1a9-82ce-49d5-8baa-9061c88c160f-proxy-tls\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:41:38.913754 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:38.913723    2564 generic.go:356] "Generic (PLEG): container finished" podID="4eedd1a9-82ce-49d5-8baa-9061c88c160f" containerID="6ea387e9a7ce3918364d59828807265c06d493990878bae03f7d290df21cdaa7" exitCode=0
Jul 02 21:41:38.913920 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:38.913780    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz"
Jul 02 21:41:38.913920 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:38.913806    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz" event={"ID":"4eedd1a9-82ce-49d5-8baa-9061c88c160f","Type":"ContainerDied","Data":"6ea387e9a7ce3918364d59828807265c06d493990878bae03f7d290df21cdaa7"}
Jul 02 21:41:38.913920 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:38.913871    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz" event={"ID":"4eedd1a9-82ce-49d5-8baa-9061c88c160f","Type":"ContainerDied","Data":"47262e80668409fbdd3b618c25d5928717cd9adbc682d7373e453bbede64f280"}
Jul 02 21:41:38.913920 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:38.913898    2564 scope.go:117] "RemoveContainer" containerID="dbbe662fca94ae8d148f351d0515a34b5a719fe0c7ae21e109695e999c46d531"
Jul 02 21:41:38.923162 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:38.923142    2564 scope.go:117] "RemoveContainer" containerID="6ea387e9a7ce3918364d59828807265c06d493990878bae03f7d290df21cdaa7"
Jul 02 21:41:38.930386 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:38.930370    2564 scope.go:117] "RemoveContainer" containerID="dbbe662fca94ae8d148f351d0515a34b5a719fe0c7ae21e109695e999c46d531"
Jul 02 21:41:38.930618 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:41:38.930598    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbbe662fca94ae8d148f351d0515a34b5a719fe0c7ae21e109695e999c46d531\": container with ID starting with dbbe662fca94ae8d148f351d0515a34b5a719fe0c7ae21e109695e999c46d531 not found: ID does not exist" containerID="dbbe662fca94ae8d148f351d0515a34b5a719fe0c7ae21e109695e999c46d531"
Jul 02 21:41:38.930728 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:38.930625    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbbe662fca94ae8d148f351d0515a34b5a719fe0c7ae21e109695e999c46d531"} err="failed to get container status \"dbbe662fca94ae8d148f351d0515a34b5a719fe0c7ae21e109695e999c46d531\": rpc error: code = NotFound desc = could not find container \"dbbe662fca94ae8d148f351d0515a34b5a719fe0c7ae21e109695e999c46d531\": container with ID starting with dbbe662fca94ae8d148f351d0515a34b5a719fe0c7ae21e109695e999c46d531 not found: ID does not exist"
Jul 02 21:41:38.930728 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:38.930643    2564 scope.go:117] "RemoveContainer" containerID="6ea387e9a7ce3918364d59828807265c06d493990878bae03f7d290df21cdaa7"
Jul 02 21:41:38.930884 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:41:38.930870    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ea387e9a7ce3918364d59828807265c06d493990878bae03f7d290df21cdaa7\": container with ID starting with 6ea387e9a7ce3918364d59828807265c06d493990878bae03f7d290df21cdaa7 not found: ID does not exist" containerID="6ea387e9a7ce3918364d59828807265c06d493990878bae03f7d290df21cdaa7"
Jul 02 21:41:38.930931 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:38.930888    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea387e9a7ce3918364d59828807265c06d493990878bae03f7d290df21cdaa7"} err="failed to get container status \"6ea387e9a7ce3918364d59828807265c06d493990878bae03f7d290df21cdaa7\": rpc error: code = NotFound desc = could not find container \"6ea387e9a7ce3918364d59828807265c06d493990878bae03f7d290df21cdaa7\": container with ID starting with 6ea387e9a7ce3918364d59828807265c06d493990878bae03f7d290df21cdaa7 not found: ID does not exist"
Jul 02 21:41:38.935364 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:38.935340    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz"]
Jul 02 21:41:38.939367 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:38.939347    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ee70d-predictor-78b4c5c7b6-kv4jz"]
Jul 02 21:41:39.734998 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:39.734963    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s"
Jul 02 21:41:40.076666 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:40.076586    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eedd1a9-82ce-49d5-8baa-9061c88c160f" path="/var/lib/kubelet/pods/4eedd1a9-82ce-49d5-8baa-9061c88c160f/volumes"
Jul 02 21:41:41.907899 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:41.907871    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd"
Jul 02 21:41:41.908361 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:41.908328    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd" podUID="fd57752b-4d3f-4887-9433-f69b266be07c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 02 21:41:51.908823 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:41:51.908781    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd" podUID="fd57752b-4d3f-4887-9433-f69b266be07c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 02 21:42:00.947408 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:00.947371    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s"]
Jul 02 21:42:00.947943 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:00.947654    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s" podUID="08b59517-c227-4b47-9824-7ce0c25e6707" containerName="kserve-container" containerID="cri-o://961012cfd3611a668fc02278e30babbaea7bba916ee986fd02b400d432176a4d" gracePeriod=30
Jul 02 21:42:00.947943 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:00.947677    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s" podUID="08b59517-c227-4b47-9824-7ce0c25e6707" containerName="kube-rbac-proxy" containerID="cri-o://88a824f30bdbca4ded28564ed8d523155f124c0c587b0ceece059311e2b57e40" gracePeriod=30
Jul 02 21:42:00.978248 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:00.978221    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz"]
Jul 02 21:42:00.978580 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:00.978568    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eedd1a9-82ce-49d5-8baa-9061c88c160f" containerName="kube-rbac-proxy"
Jul 02 21:42:00.978631 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:00.978581    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eedd1a9-82ce-49d5-8baa-9061c88c160f" containerName="kube-rbac-proxy"
Jul 02 21:42:00.978631 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:00.978597    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eedd1a9-82ce-49d5-8baa-9061c88c160f" containerName="kserve-container"
Jul 02 21:42:00.978631 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:00.978603    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eedd1a9-82ce-49d5-8baa-9061c88c160f" containerName="kserve-container"
Jul 02 21:42:00.978728 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:00.978655    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eedd1a9-82ce-49d5-8baa-9061c88c160f" containerName="kube-rbac-proxy"
Jul 02 21:42:00.978728 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:00.978665    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eedd1a9-82ce-49d5-8baa-9061c88c160f" containerName="kserve-container"
Jul 02 21:42:00.982380 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:00.982362    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz"
Jul 02 21:42:00.985026 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:00.985008    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8bf58-kube-rbac-proxy-sar-config\""
Jul 02 21:42:00.985173 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:00.985070    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8bf58-predictor-serving-cert\""
Jul 02 21:42:00.990589 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:00.990566    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz"]
Jul 02 21:42:01.036789 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:01.036759    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ae0814a-bb2b-4049-8980-2e1654c9dcda-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8bf58-predictor-695db95966-9w5pz\" (UID: \"6ae0814a-bb2b-4049-8980-2e1654c9dcda\") " pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz"
Jul 02 21:42:01.036941 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:01.036865    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ae0814a-bb2b-4049-8980-2e1654c9dcda-proxy-tls\") pod \"success-200-isvc-8bf58-predictor-695db95966-9w5pz\" (UID: \"6ae0814a-bb2b-4049-8980-2e1654c9dcda\") " pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz"
Jul 02 21:42:01.036941 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:01.036897    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4bnc\" (UniqueName: \"kubernetes.io/projected/6ae0814a-bb2b-4049-8980-2e1654c9dcda-kube-api-access-m4bnc\") pod \"success-200-isvc-8bf58-predictor-695db95966-9w5pz\" (UID: \"6ae0814a-bb2b-4049-8980-2e1654c9dcda\") " pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz"
Jul 02 21:42:01.138014 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:01.137976    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ae0814a-bb2b-4049-8980-2e1654c9dcda-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8bf58-predictor-695db95966-9w5pz\" (UID: \"6ae0814a-bb2b-4049-8980-2e1654c9dcda\") " pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz"
Jul 02 21:42:01.138229 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:01.138067    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ae0814a-bb2b-4049-8980-2e1654c9dcda-proxy-tls\") pod \"success-200-isvc-8bf58-predictor-695db95966-9w5pz\" (UID: \"6ae0814a-bb2b-4049-8980-2e1654c9dcda\") " pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz"
Jul 02 21:42:01.138229 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:01.138089    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4bnc\" (UniqueName: \"kubernetes.io/projected/6ae0814a-bb2b-4049-8980-2e1654c9dcda-kube-api-access-m4bnc\") pod \"success-200-isvc-8bf58-predictor-695db95966-9w5pz\" (UID: \"6ae0814a-bb2b-4049-8980-2e1654c9dcda\") " pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz"
Jul 02 21:42:01.138698 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:01.138672    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ae0814a-bb2b-4049-8980-2e1654c9dcda-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8bf58-predictor-695db95966-9w5pz\" (UID: \"6ae0814a-bb2b-4049-8980-2e1654c9dcda\") " pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz"
Jul 02 21:42:01.140474 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:01.140453    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ae0814a-bb2b-4049-8980-2e1654c9dcda-proxy-tls\") pod \"success-200-isvc-8bf58-predictor-695db95966-9w5pz\" (UID: \"6ae0814a-bb2b-4049-8980-2e1654c9dcda\") " pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz"
Jul 02 21:42:01.145564 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:01.145541    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4bnc\" (UniqueName: \"kubernetes.io/projected/6ae0814a-bb2b-4049-8980-2e1654c9dcda-kube-api-access-m4bnc\") pod \"success-200-isvc-8bf58-predictor-695db95966-9w5pz\" (UID: \"6ae0814a-bb2b-4049-8980-2e1654c9dcda\") " pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz"
Jul 02 21:42:01.293233 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:01.293202    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz"
Jul 02 21:42:01.418551 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:01.418526    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz"]
Jul 02 21:42:01.420290 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:42:01.420265    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae0814a_bb2b_4049_8980_2e1654c9dcda.slice/crio-7b7e5343d9a7294caccdbf40befaeda76b11642b7f84630fcf7c6432e46bf2ec WatchSource:0}: Error finding container 7b7e5343d9a7294caccdbf40befaeda76b11642b7f84630fcf7c6432e46bf2ec: Status 404 returned error can't find the container with id 7b7e5343d9a7294caccdbf40befaeda76b11642b7f84630fcf7c6432e46bf2ec
Jul 02 21:42:01.909092 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:01.909052    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd" podUID="fd57752b-4d3f-4887-9433-f69b266be07c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 02 21:42:02.017683 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:02.017638    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz" event={"ID":"6ae0814a-bb2b-4049-8980-2e1654c9dcda","Type":"ContainerStarted","Data":"4fc1687a8538f5a9ddf1cb752a9fe35bade403db0792dce317e8a8e88a332f62"}
Jul 02 21:42:02.018141 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:02.017736    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz" event={"ID":"6ae0814a-bb2b-4049-8980-2e1654c9dcda","Type":"ContainerStarted","Data":"9c41c2826870f5322cd66ce847cf00afc9141ed8ed82a8a0c86cfaa05b28ac42"}
Jul 02 21:42:02.018141 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:02.017774    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz"
Jul 02 21:42:02.018141 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:02.017802    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz"
Jul 02 21:42:02.018141 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:02.017828    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz" event={"ID":"6ae0814a-bb2b-4049-8980-2e1654c9dcda","Type":"ContainerStarted","Data":"7b7e5343d9a7294caccdbf40befaeda76b11642b7f84630fcf7c6432e46bf2ec"}
Jul 02 21:42:02.019071 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:02.019038    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz" podUID="6ae0814a-bb2b-4049-8980-2e1654c9dcda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 02 21:42:02.019577 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:02.019555    2564 generic.go:356] "Generic (PLEG): container finished" podID="08b59517-c227-4b47-9824-7ce0c25e6707" containerID="88a824f30bdbca4ded28564ed8d523155f124c0c587b0ceece059311e2b57e40" exitCode=2
Jul 02 21:42:02.019650 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:02.019616    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s" event={"ID":"08b59517-c227-4b47-9824-7ce0c25e6707","Type":"ContainerDied","Data":"88a824f30bdbca4ded28564ed8d523155f124c0c587b0ceece059311e2b57e40"}
Jul 02 21:42:02.036760 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:02.036710    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz" podStartSLOduration=2.036693939 podStartE2EDuration="2.036693939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:42:02.035351312 +0000 UTC m=+1382.872591358" watchObservedRunningTime="2026-07-02 21:42:02.036693939 +0000 UTC m=+1382.873933985"
Jul 02 21:42:03.023777 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:03.023742    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz" podUID="6ae0814a-bb2b-4049-8980-2e1654c9dcda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 02 21:42:04.030673 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:04.030536    2564 generic.go:356] "Generic (PLEG): container finished" podID="08b59517-c227-4b47-9824-7ce0c25e6707" containerID="961012cfd3611a668fc02278e30babbaea7bba916ee986fd02b400d432176a4d" exitCode=0
Jul 02 21:42:04.030673 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:04.030606    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s" event={"ID":"08b59517-c227-4b47-9824-7ce0c25e6707","Type":"ContainerDied","Data":"961012cfd3611a668fc02278e30babbaea7bba916ee986fd02b400d432176a4d"}
Jul 02 21:42:04.092189 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:04.092164    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s"
Jul 02 21:42:04.165339 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:04.165302    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvrm4\" (UniqueName: \"kubernetes.io/projected/08b59517-c227-4b47-9824-7ce0c25e6707-kube-api-access-vvrm4\") pod \"08b59517-c227-4b47-9824-7ce0c25e6707\" (UID: \"08b59517-c227-4b47-9824-7ce0c25e6707\") "
Jul 02 21:42:04.165515 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:04.165395    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08b59517-c227-4b47-9824-7ce0c25e6707-proxy-tls\") pod \"08b59517-c227-4b47-9824-7ce0c25e6707\" (UID: \"08b59517-c227-4b47-9824-7ce0c25e6707\") "
Jul 02 21:42:04.165515 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:04.165423    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08b59517-c227-4b47-9824-7ce0c25e6707-kube-rbac-proxy-sar-config\") pod \"08b59517-c227-4b47-9824-7ce0c25e6707\" (UID: \"08b59517-c227-4b47-9824-7ce0c25e6707\") "
Jul 02 21:42:04.165849 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:04.165823    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b59517-c227-4b47-9824-7ce0c25e6707-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "08b59517-c227-4b47-9824-7ce0c25e6707" (UID: "08b59517-c227-4b47-9824-7ce0c25e6707"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:42:04.167443 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:04.167416    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b59517-c227-4b47-9824-7ce0c25e6707-kube-api-access-vvrm4" (OuterVolumeSpecName: "kube-api-access-vvrm4") pod "08b59517-c227-4b47-9824-7ce0c25e6707" (UID: "08b59517-c227-4b47-9824-7ce0c25e6707"). InnerVolumeSpecName "kube-api-access-vvrm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:42:04.167551 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:04.167446    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08b59517-c227-4b47-9824-7ce0c25e6707-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "08b59517-c227-4b47-9824-7ce0c25e6707" (UID: "08b59517-c227-4b47-9824-7ce0c25e6707"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:42:04.266105 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:04.266072    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08b59517-c227-4b47-9824-7ce0c25e6707-proxy-tls\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:42:04.266105 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:04.266102    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08b59517-c227-4b47-9824-7ce0c25e6707-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:42:04.266349 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:04.266140    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvrm4\" (UniqueName: \"kubernetes.io/projected/08b59517-c227-4b47-9824-7ce0c25e6707-kube-api-access-vvrm4\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:42:05.037477 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:05.037437    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s" event={"ID":"08b59517-c227-4b47-9824-7ce0c25e6707","Type":"ContainerDied","Data":"71f4b77073de5961a2d71b82adf653a14f3150e6b07ca0a96719a81f6c7a7667"}
Jul 02 21:42:05.037477 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:05.037475    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s"
Jul 02 21:42:05.037938 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:05.037527    2564 scope.go:117] "RemoveContainer" containerID="88a824f30bdbca4ded28564ed8d523155f124c0c587b0ceece059311e2b57e40"
Jul 02 21:42:05.047497 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:05.047478    2564 scope.go:117] "RemoveContainer" containerID="961012cfd3611a668fc02278e30babbaea7bba916ee986fd02b400d432176a4d"
Jul 02 21:42:05.060570 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:05.060548    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s"]
Jul 02 21:42:05.065151 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:05.065130    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1702f-predictor-d85f8dbd4-57r4s"]
Jul 02 21:42:06.074377 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:06.074337    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08b59517-c227-4b47-9824-7ce0c25e6707" path="/var/lib/kubelet/pods/08b59517-c227-4b47-9824-7ce0c25e6707/volumes"
Jul 02 21:42:08.028364 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:08.028337    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz"
Jul 02 21:42:08.028817 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:08.028791    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz" podUID="6ae0814a-bb2b-4049-8980-2e1654c9dcda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 02 21:42:11.908354 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:11.908315    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd" podUID="fd57752b-4d3f-4887-9433-f69b266be07c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 02 21:42:18.028879 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:18.028843    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz" podUID="6ae0814a-bb2b-4049-8980-2e1654c9dcda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 02 21:42:21.909005 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:21.908973    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd"
Jul 02 21:42:28.029365 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:28.029317    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz" podUID="6ae0814a-bb2b-4049-8980-2e1654c9dcda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 02 21:42:38.029572 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:38.029536    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz" podUID="6ae0814a-bb2b-4049-8980-2e1654c9dcda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 02 21:42:48.029309 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:42:48.029278    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz"
Jul 02 21:50:50.011701 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:50.011670    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk"]
Jul 02 21:50:50.012189 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:50.012012    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b59517-c227-4b47-9824-7ce0c25e6707" containerName="kube-rbac-proxy"
Jul 02 21:50:50.012189 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:50.012022    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b59517-c227-4b47-9824-7ce0c25e6707" containerName="kube-rbac-proxy"
Jul 02 21:50:50.012189 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:50.012034    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b59517-c227-4b47-9824-7ce0c25e6707" containerName="kserve-container"
Jul 02 21:50:50.012189 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:50.012039    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b59517-c227-4b47-9824-7ce0c25e6707" containerName="kserve-container"
Jul 02 21:50:50.012189 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:50.012097    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="08b59517-c227-4b47-9824-7ce0c25e6707" containerName="kserve-container"
Jul 02 21:50:50.012189 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:50.012107    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="08b59517-c227-4b47-9824-7ce0c25e6707" containerName="kube-rbac-proxy"
Jul 02 21:50:50.015584 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:50.015563    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk"
Jul 02 21:50:50.017765 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:50.017747    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c9d16-predictor-serving-cert\""
Jul 02 21:50:50.018084 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:50.018065    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c9d16-kube-rbac-proxy-sar-config\""
Jul 02 21:50:50.020890 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:50.020868    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd"]
Jul 02 21:50:50.021103 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:50.021081    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd" podUID="fd57752b-4d3f-4887-9433-f69b266be07c" containerName="kserve-container" containerID="cri-o://56889a8a3916ea9e04c770bc4433e6de9275b7697bd36e57240323ed0f09c177" gracePeriod=30
Jul 02 21:50:50.021207 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:50.021186    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd" podUID="fd57752b-4d3f-4887-9433-f69b266be07c" containerName="kube-rbac-proxy" containerID="cri-o://8509fcf9ada29fe5eb164446e744ab4f181ce5a4515d66b35b15e52d4f181b1d" gracePeriod=30
Jul 02 21:50:50.025838 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:50.025817    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk"]
Jul 02 21:50:50.154847 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:50.154815    2564 generic.go:356] "Generic (PLEG): container finished" podID="fd57752b-4d3f-4887-9433-f69b266be07c" containerID="8509fcf9ada29fe5eb164446e744ab4f181ce5a4515d66b35b15e52d4f181b1d" exitCode=2
Jul 02 21:50:50.155000 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:50.154887    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd" event={"ID":"fd57752b-4d3f-4887-9433-f69b266be07c","Type":"ContainerDied","Data":"8509fcf9ada29fe5eb164446e744ab4f181ce5a4515d66b35b15e52d4f181b1d"}
Jul 02 21:50:50.168321 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:50.168244    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw4rg\" (UniqueName: \"kubernetes.io/projected/7c34c887-38b7-4dcd-8043-c726de59f2f7-kube-api-access-pw4rg\") pod \"success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk\" (UID: \"7c34c887-38b7-4dcd-8043-c726de59f2f7\") " pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk"
Jul 02 21:50:50.168321 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:50.168299    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c34c887-38b7-4dcd-8043-c726de59f2f7-proxy-tls\") pod \"success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk\" (UID: \"7c34c887-38b7-4dcd-8043-c726de59f2f7\") " pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk"
Jul 02 21:50:50.168554 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:50.168334    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c34c887-38b7-4dcd-8043-c726de59f2f7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk\" (UID: \"7c34c887-38b7-4dcd-8043-c726de59f2f7\") " pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk"
Jul 02 21:50:50.269279 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:50.269183    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c34c887-38b7-4dcd-8043-c726de59f2f7-proxy-tls\") pod \"success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk\" (UID: \"7c34c887-38b7-4dcd-8043-c726de59f2f7\") " pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk"
Jul 02 21:50:50.269279 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:50.269232    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c34c887-38b7-4dcd-8043-c726de59f2f7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk\" (UID: \"7c34c887-38b7-4dcd-8043-c726de59f2f7\") " pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk"
Jul 02 21:50:50.269568 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:50.269309    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw4rg\" (UniqueName: \"kubernetes.io/projected/7c34c887-38b7-4dcd-8043-c726de59f2f7-kube-api-access-pw4rg\") pod \"success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk\" (UID: \"7c34c887-38b7-4dcd-8043-c726de59f2f7\") " pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk"
Jul 02 21:50:50.269568 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:50:50.269362    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-serving-cert: secret "success-200-isvc-c9d16-predictor-serving-cert" not found
Jul 02 21:50:50.269568 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:50:50.269448    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c34c887-38b7-4dcd-8043-c726de59f2f7-proxy-tls podName:7c34c887-38b7-4dcd-8043-c726de59f2f7 nodeName:}" failed. No retries permitted until 2026-07-02 21:50:50.769429387 +0000 UTC m=+1911.606669432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7c34c887-38b7-4dcd-8043-c726de59f2f7-proxy-tls") pod "success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk" (UID: "7c34c887-38b7-4dcd-8043-c726de59f2f7") : secret "success-200-isvc-c9d16-predictor-serving-cert" not found
Jul 02 21:50:50.270026 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:50.270006    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c34c887-38b7-4dcd-8043-c726de59f2f7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk\" (UID: \"7c34c887-38b7-4dcd-8043-c726de59f2f7\") " pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk"
Jul 02 21:50:50.278323 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:50.278304    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw4rg\" (UniqueName: \"kubernetes.io/projected/7c34c887-38b7-4dcd-8043-c726de59f2f7-kube-api-access-pw4rg\") pod \"success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk\" (UID: \"7c34c887-38b7-4dcd-8043-c726de59f2f7\") " pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk"
Jul 02 21:50:50.773291 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:50.773244    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c34c887-38b7-4dcd-8043-c726de59f2f7-proxy-tls\") pod \"success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk\" (UID: \"7c34c887-38b7-4dcd-8043-c726de59f2f7\") " pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk"
Jul 02 21:50:50.775583 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:50.775562    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c34c887-38b7-4dcd-8043-c726de59f2f7-proxy-tls\") pod \"success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk\" (UID: \"7c34c887-38b7-4dcd-8043-c726de59f2f7\") " pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk"
Jul 02 21:50:50.928186 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:50.928150    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk"
Jul 02 21:50:51.057907 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:51.057832    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk"]
Jul 02 21:50:51.060199 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:50:51.060175    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c34c887_38b7_4dcd_8043_c726de59f2f7.slice/crio-19433b42ae973029d3a2cb6873c48923e9f7e04b23a852c2e97b3a2e22f5aeb6 WatchSource:0}: Error finding container 19433b42ae973029d3a2cb6873c48923e9f7e04b23a852c2e97b3a2e22f5aeb6: Status 404 returned error can't find the container with id 19433b42ae973029d3a2cb6873c48923e9f7e04b23a852c2e97b3a2e22f5aeb6
Jul 02 21:50:51.161443 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:51.161415    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk" event={"ID":"7c34c887-38b7-4dcd-8043-c726de59f2f7","Type":"ContainerStarted","Data":"cf2a879d94e638d025cacf861d52ae899a86743201e76ffe8a8dd8b1aa50b493"}
Jul 02 21:50:51.161561 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:51.161473    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk" event={"ID":"7c34c887-38b7-4dcd-8043-c726de59f2f7","Type":"ContainerStarted","Data":"19433b42ae973029d3a2cb6873c48923e9f7e04b23a852c2e97b3a2e22f5aeb6"}
Jul 02 21:50:51.903651 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:51.903606    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd" podUID="fd57752b-4d3f-4887-9433-f69b266be07c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 02 21:50:51.909014 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:51.908977    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd" podUID="fd57752b-4d3f-4887-9433-f69b266be07c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 02 21:50:52.167691 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:52.167590    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk" event={"ID":"7c34c887-38b7-4dcd-8043-c726de59f2f7","Type":"ContainerStarted","Data":"d297231d5b095d74d2c4892cd4679cf6e21714eac3d2be340d3c945b805ca5a4"}
Jul 02 21:50:52.168142 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:52.167825    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk"
Jul 02 21:50:52.168142 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:52.167989    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk"
Jul 02 21:50:52.169090 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:52.169064    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk" podUID="7c34c887-38b7-4dcd-8043-c726de59f2f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 21:50:52.188178 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:52.188138    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk" podStartSLOduration=3.188122068 podStartE2EDuration="3.188122068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:50:52.185694371 +0000 UTC m=+1913.022934440" watchObservedRunningTime="2026-07-02 21:50:52.188122068 +0000 UTC m=+1913.025362107"
Jul 02 21:50:53.164377 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:53.164356    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd"
Jul 02 21:50:53.173311 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:53.173285    2564 generic.go:356] "Generic (PLEG): container finished" podID="fd57752b-4d3f-4887-9433-f69b266be07c" containerID="56889a8a3916ea9e04c770bc4433e6de9275b7697bd36e57240323ed0f09c177" exitCode=0
Jul 02 21:50:53.173641 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:53.173323    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd"
Jul 02 21:50:53.173641 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:53.173374    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd" event={"ID":"fd57752b-4d3f-4887-9433-f69b266be07c","Type":"ContainerDied","Data":"56889a8a3916ea9e04c770bc4433e6de9275b7697bd36e57240323ed0f09c177"}
Jul 02 21:50:53.173641 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:53.173427    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd" event={"ID":"fd57752b-4d3f-4887-9433-f69b266be07c","Type":"ContainerDied","Data":"c3716122c19373a934b360c7933e5acd16290cebd741f8628bbaea5ec7b2680a"}
Jul 02 21:50:53.173641 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:53.173454    2564 scope.go:117] "RemoveContainer" containerID="8509fcf9ada29fe5eb164446e744ab4f181ce5a4515d66b35b15e52d4f181b1d"
Jul 02 21:50:53.173834 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:53.173729    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk" podUID="7c34c887-38b7-4dcd-8043-c726de59f2f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 21:50:53.181063 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:53.181045    2564 scope.go:117] "RemoveContainer" containerID="56889a8a3916ea9e04c770bc4433e6de9275b7697bd36e57240323ed0f09c177"
Jul 02 21:50:53.189730 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:53.189709    2564 scope.go:117] "RemoveContainer" containerID="8509fcf9ada29fe5eb164446e744ab4f181ce5a4515d66b35b15e52d4f181b1d"
Jul 02 21:50:53.190037 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:50:53.189993    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8509fcf9ada29fe5eb164446e744ab4f181ce5a4515d66b35b15e52d4f181b1d\": container with ID starting with 8509fcf9ada29fe5eb164446e744ab4f181ce5a4515d66b35b15e52d4f181b1d not found: ID does not exist" containerID="8509fcf9ada29fe5eb164446e744ab4f181ce5a4515d66b35b15e52d4f181b1d"
Jul 02 21:50:53.190037 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:53.190021    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8509fcf9ada29fe5eb164446e744ab4f181ce5a4515d66b35b15e52d4f181b1d"} err="failed to get container status \"8509fcf9ada29fe5eb164446e744ab4f181ce5a4515d66b35b15e52d4f181b1d\": rpc error: code = NotFound desc = could not find container \"8509fcf9ada29fe5eb164446e744ab4f181ce5a4515d66b35b15e52d4f181b1d\": container with ID starting with 8509fcf9ada29fe5eb164446e744ab4f181ce5a4515d66b35b15e52d4f181b1d not found: ID does not exist"
Jul 02 21:50:53.190165 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:53.190039    2564 scope.go:117] "RemoveContainer" containerID="56889a8a3916ea9e04c770bc4433e6de9275b7697bd36e57240323ed0f09c177"
Jul 02 21:50:53.190391 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:50:53.190358    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56889a8a3916ea9e04c770bc4433e6de9275b7697bd36e57240323ed0f09c177\": container with ID starting with 56889a8a3916ea9e04c770bc4433e6de9275b7697bd36e57240323ed0f09c177 not found: ID does not exist" containerID="56889a8a3916ea9e04c770bc4433e6de9275b7697bd36e57240323ed0f09c177"
Jul 02 21:50:53.190434 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:53.190402    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56889a8a3916ea9e04c770bc4433e6de9275b7697bd36e57240323ed0f09c177"} err="failed to get container status \"56889a8a3916ea9e04c770bc4433e6de9275b7697bd36e57240323ed0f09c177\": rpc error: code = NotFound desc = could not find container \"56889a8a3916ea9e04c770bc4433e6de9275b7697bd36e57240323ed0f09c177\": container with ID starting with 56889a8a3916ea9e04c770bc4433e6de9275b7697bd36e57240323ed0f09c177 not found: ID does not exist"
Jul 02 21:50:53.296765 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:53.296727    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd57752b-4d3f-4887-9433-f69b266be07c-kube-rbac-proxy-sar-config\") pod \"fd57752b-4d3f-4887-9433-f69b266be07c\" (UID: \"fd57752b-4d3f-4887-9433-f69b266be07c\") "
Jul 02 21:50:53.296950 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:53.296782    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vf9s\" (UniqueName: \"kubernetes.io/projected/fd57752b-4d3f-4887-9433-f69b266be07c-kube-api-access-6vf9s\") pod \"fd57752b-4d3f-4887-9433-f69b266be07c\" (UID: \"fd57752b-4d3f-4887-9433-f69b266be07c\") "
Jul 02 21:50:53.296950 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:53.296812    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd57752b-4d3f-4887-9433-f69b266be07c-proxy-tls\") pod \"fd57752b-4d3f-4887-9433-f69b266be07c\" (UID: \"fd57752b-4d3f-4887-9433-f69b266be07c\") "
Jul 02 21:50:53.297106 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:53.297077    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd57752b-4d3f-4887-9433-f69b266be07c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fd57752b-4d3f-4887-9433-f69b266be07c" (UID: "fd57752b-4d3f-4887-9433-f69b266be07c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:50:53.298941 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:53.298917    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd57752b-4d3f-4887-9433-f69b266be07c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fd57752b-4d3f-4887-9433-f69b266be07c" (UID: "fd57752b-4d3f-4887-9433-f69b266be07c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:50:53.298941 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:53.298927    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd57752b-4d3f-4887-9433-f69b266be07c-kube-api-access-6vf9s" (OuterVolumeSpecName: "kube-api-access-6vf9s") pod "fd57752b-4d3f-4887-9433-f69b266be07c" (UID: "fd57752b-4d3f-4887-9433-f69b266be07c"). InnerVolumeSpecName "kube-api-access-6vf9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:50:53.398362 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:53.398330    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd57752b-4d3f-4887-9433-f69b266be07c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:50:53.398362 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:53.398357    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vf9s\" (UniqueName: \"kubernetes.io/projected/fd57752b-4d3f-4887-9433-f69b266be07c-kube-api-access-6vf9s\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:50:53.398362 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:53.398370    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd57752b-4d3f-4887-9433-f69b266be07c-proxy-tls\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:50:53.495761 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:53.495727    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd"]
Jul 02 21:50:53.499398 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:53.499374    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-87dcc-predictor-648644cd6-hvcgd"]
Jul 02 21:50:54.072238 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:54.072208    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd57752b-4d3f-4887-9433-f69b266be07c" path="/var/lib/kubelet/pods/fd57752b-4d3f-4887-9433-f69b266be07c/volumes"
Jul 02 21:50:58.179268 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:58.179236    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk"
Jul 02 21:50:58.179824 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:50:58.179738    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk" podUID="7c34c887-38b7-4dcd-8043-c726de59f2f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 21:51:08.179821 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:08.179781    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk" podUID="7c34c887-38b7-4dcd-8043-c726de59f2f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 21:51:15.874123 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:15.874080    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz"]
Jul 02 21:51:15.874608 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:15.874345    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz" podUID="6ae0814a-bb2b-4049-8980-2e1654c9dcda" containerName="kserve-container" containerID="cri-o://9c41c2826870f5322cd66ce847cf00afc9141ed8ed82a8a0c86cfaa05b28ac42" gracePeriod=30
Jul 02 21:51:15.874608 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:15.874427    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz" podUID="6ae0814a-bb2b-4049-8980-2e1654c9dcda" containerName="kube-rbac-proxy" containerID="cri-o://4fc1687a8538f5a9ddf1cb752a9fe35bade403db0792dce317e8a8e88a332f62" gracePeriod=30
Jul 02 21:51:15.901724 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:15.901688    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5"]
Jul 02 21:51:15.902064 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:15.902052    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd57752b-4d3f-4887-9433-f69b266be07c" containerName="kube-rbac-proxy"
Jul 02 21:51:15.902107 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:15.902066    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd57752b-4d3f-4887-9433-f69b266be07c" containerName="kube-rbac-proxy"
Jul 02 21:51:15.902107 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:15.902078    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd57752b-4d3f-4887-9433-f69b266be07c" containerName="kserve-container"
Jul 02 21:51:15.902107 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:15.902083    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd57752b-4d3f-4887-9433-f69b266be07c" containerName="kserve-container"
Jul 02 21:51:15.902226 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:15.902158    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd57752b-4d3f-4887-9433-f69b266be07c" containerName="kserve-container"
Jul 02 21:51:15.902226 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:15.902168    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd57752b-4d3f-4887-9433-f69b266be07c" containerName="kube-rbac-proxy"
Jul 02 21:51:15.905377 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:15.905360    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5"
Jul 02 21:51:15.907888 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:15.907867    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-76a5f-predictor-serving-cert\""
Jul 02 21:51:15.908011 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:15.907907    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-76a5f-kube-rbac-proxy-sar-config\""
Jul 02 21:51:15.915281 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:15.915262    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5"]
Jul 02 21:51:15.986551 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:15.986524    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bfa0b5e-ac46-4503-a2d0-c0a679de6543-proxy-tls\") pod \"success-200-isvc-76a5f-predictor-68958f7c46-8ddl5\" (UID: \"8bfa0b5e-ac46-4503-a2d0-c0a679de6543\") " pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5"
Jul 02 21:51:15.986679 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:15.986566    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bfa0b5e-ac46-4503-a2d0-c0a679de6543-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-76a5f-predictor-68958f7c46-8ddl5\" (UID: \"8bfa0b5e-ac46-4503-a2d0-c0a679de6543\") " pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5"
Jul 02 21:51:15.986679 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:15.986629    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fnmt\" (UniqueName: \"kubernetes.io/projected/8bfa0b5e-ac46-4503-a2d0-c0a679de6543-kube-api-access-5fnmt\") pod \"success-200-isvc-76a5f-predictor-68958f7c46-8ddl5\" (UID: \"8bfa0b5e-ac46-4503-a2d0-c0a679de6543\") " pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5"
Jul 02 21:51:16.087659 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:16.087626    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fnmt\" (UniqueName: \"kubernetes.io/projected/8bfa0b5e-ac46-4503-a2d0-c0a679de6543-kube-api-access-5fnmt\") pod \"success-200-isvc-76a5f-predictor-68958f7c46-8ddl5\" (UID: \"8bfa0b5e-ac46-4503-a2d0-c0a679de6543\") " pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5"
Jul 02 21:51:16.087847 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:16.087709    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bfa0b5e-ac46-4503-a2d0-c0a679de6543-proxy-tls\") pod \"success-200-isvc-76a5f-predictor-68958f7c46-8ddl5\" (UID: \"8bfa0b5e-ac46-4503-a2d0-c0a679de6543\") " pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5"
Jul 02 21:51:16.087847 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:16.087749    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bfa0b5e-ac46-4503-a2d0-c0a679de6543-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-76a5f-predictor-68958f7c46-8ddl5\" (UID: \"8bfa0b5e-ac46-4503-a2d0-c0a679de6543\") " pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5"
Jul 02 21:51:16.087847 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:51:16.087831    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-serving-cert: secret "success-200-isvc-76a5f-predictor-serving-cert" not found
Jul 02 21:51:16.088033 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:51:16.087897    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfa0b5e-ac46-4503-a2d0-c0a679de6543-proxy-tls podName:8bfa0b5e-ac46-4503-a2d0-c0a679de6543 nodeName:}" failed. No retries permitted until 2026-07-02 21:51:16.587881601 +0000 UTC m=+1937.425121626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8bfa0b5e-ac46-4503-a2d0-c0a679de6543-proxy-tls") pod "success-200-isvc-76a5f-predictor-68958f7c46-8ddl5" (UID: "8bfa0b5e-ac46-4503-a2d0-c0a679de6543") : secret "success-200-isvc-76a5f-predictor-serving-cert" not found
Jul 02 21:51:16.088409 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:16.088386    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bfa0b5e-ac46-4503-a2d0-c0a679de6543-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-76a5f-predictor-68958f7c46-8ddl5\" (UID: \"8bfa0b5e-ac46-4503-a2d0-c0a679de6543\") " pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5"
Jul 02 21:51:16.095350 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:16.095325    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fnmt\" (UniqueName: \"kubernetes.io/projected/8bfa0b5e-ac46-4503-a2d0-c0a679de6543-kube-api-access-5fnmt\") pod \"success-200-isvc-76a5f-predictor-68958f7c46-8ddl5\" (UID: \"8bfa0b5e-ac46-4503-a2d0-c0a679de6543\") " pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5"
Jul 02 21:51:16.272167 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:16.272138    2564 generic.go:356] "Generic (PLEG): container finished" podID="6ae0814a-bb2b-4049-8980-2e1654c9dcda" containerID="4fc1687a8538f5a9ddf1cb752a9fe35bade403db0792dce317e8a8e88a332f62" exitCode=2
Jul 02 21:51:16.272332 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:16.272204    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz" event={"ID":"6ae0814a-bb2b-4049-8980-2e1654c9dcda","Type":"ContainerDied","Data":"4fc1687a8538f5a9ddf1cb752a9fe35bade403db0792dce317e8a8e88a332f62"}
Jul 02 21:51:16.592050 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:16.591960    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bfa0b5e-ac46-4503-a2d0-c0a679de6543-proxy-tls\") pod \"success-200-isvc-76a5f-predictor-68958f7c46-8ddl5\" (UID: \"8bfa0b5e-ac46-4503-a2d0-c0a679de6543\") " pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5"
Jul 02 21:51:16.594454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:16.594431    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bfa0b5e-ac46-4503-a2d0-c0a679de6543-proxy-tls\") pod \"success-200-isvc-76a5f-predictor-68958f7c46-8ddl5\" (UID: \"8bfa0b5e-ac46-4503-a2d0-c0a679de6543\") " pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5"
Jul 02 21:51:16.815594 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:16.815556    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5"
Jul 02 21:51:16.948621 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:16.948535    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5"]
Jul 02 21:51:16.951371 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:51:16.951345    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bfa0b5e_ac46_4503_a2d0_c0a679de6543.slice/crio-ece8497fb697fb0e600c3b34f320c8761573f0c4bfbde7a0c67b520fbb1d5da3 WatchSource:0}: Error finding container ece8497fb697fb0e600c3b34f320c8761573f0c4bfbde7a0c67b520fbb1d5da3: Status 404 returned error can't find the container with id ece8497fb697fb0e600c3b34f320c8761573f0c4bfbde7a0c67b520fbb1d5da3
Jul 02 21:51:17.277764 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:17.277728    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5" event={"ID":"8bfa0b5e-ac46-4503-a2d0-c0a679de6543","Type":"ContainerStarted","Data":"30a65ff3e290a9f448292426aea5bce543a8b17f3bd09cc41c16e0d095a6ae78"}
Jul 02 21:51:17.277937 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:17.277784    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5" event={"ID":"8bfa0b5e-ac46-4503-a2d0-c0a679de6543","Type":"ContainerStarted","Data":"91c4bc06164df295d99710b89e78c9a7e9f85327ba5398d49bb7681ff8cc4fe3"}
Jul 02 21:51:17.277937 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:17.277807    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5" event={"ID":"8bfa0b5e-ac46-4503-a2d0-c0a679de6543","Type":"ContainerStarted","Data":"ece8497fb697fb0e600c3b34f320c8761573f0c4bfbde7a0c67b520fbb1d5da3"}
Jul 02 21:51:17.277937 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:17.277831    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5"
Jul 02 21:51:17.297273 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:17.297094    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5" podStartSLOduration=2.297074405 podStartE2EDuration="2.297074405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:51:17.296653177 +0000 UTC m=+1938.133893225" watchObservedRunningTime="2026-07-02 21:51:17.297074405 +0000 UTC m=+1938.134314452"
Jul 02 21:51:18.024238 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:18.024194    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz" podUID="6ae0814a-bb2b-4049-8980-2e1654c9dcda" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 02 21:51:18.028758 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:18.028728    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz" podUID="6ae0814a-bb2b-4049-8980-2e1654c9dcda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 02 21:51:18.179924 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:18.179884    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk" podUID="7c34c887-38b7-4dcd-8043-c726de59f2f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 21:51:18.281892 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:18.281815    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5"
Jul 02 21:51:18.282902 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:18.282872    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5" podUID="8bfa0b5e-ac46-4503-a2d0-c0a679de6543" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 02 21:51:18.921639 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:18.921617    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz"
Jul 02 21:51:19.015377 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:19.015297    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4bnc\" (UniqueName: \"kubernetes.io/projected/6ae0814a-bb2b-4049-8980-2e1654c9dcda-kube-api-access-m4bnc\") pod \"6ae0814a-bb2b-4049-8980-2e1654c9dcda\" (UID: \"6ae0814a-bb2b-4049-8980-2e1654c9dcda\") "
Jul 02 21:51:19.015377 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:19.015354    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ae0814a-bb2b-4049-8980-2e1654c9dcda-proxy-tls\") pod \"6ae0814a-bb2b-4049-8980-2e1654c9dcda\" (UID: \"6ae0814a-bb2b-4049-8980-2e1654c9dcda\") "
Jul 02 21:51:19.015546 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:19.015397    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ae0814a-bb2b-4049-8980-2e1654c9dcda-kube-rbac-proxy-sar-config\") pod \"6ae0814a-bb2b-4049-8980-2e1654c9dcda\" (UID: \"6ae0814a-bb2b-4049-8980-2e1654c9dcda\") "
Jul 02 21:51:19.015788 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:19.015766    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae0814a-bb2b-4049-8980-2e1654c9dcda-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6ae0814a-bb2b-4049-8980-2e1654c9dcda" (UID: "6ae0814a-bb2b-4049-8980-2e1654c9dcda"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:51:19.017375 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:19.017353    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae0814a-bb2b-4049-8980-2e1654c9dcda-kube-api-access-m4bnc" (OuterVolumeSpecName: "kube-api-access-m4bnc") pod "6ae0814a-bb2b-4049-8980-2e1654c9dcda" (UID: "6ae0814a-bb2b-4049-8980-2e1654c9dcda"). InnerVolumeSpecName "kube-api-access-m4bnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:51:19.017375 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:19.017351    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ae0814a-bb2b-4049-8980-2e1654c9dcda-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6ae0814a-bb2b-4049-8980-2e1654c9dcda" (UID: "6ae0814a-bb2b-4049-8980-2e1654c9dcda"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:51:19.116475 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:19.116441    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4bnc\" (UniqueName: \"kubernetes.io/projected/6ae0814a-bb2b-4049-8980-2e1654c9dcda-kube-api-access-m4bnc\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:51:19.116475 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:19.116471    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ae0814a-bb2b-4049-8980-2e1654c9dcda-proxy-tls\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:51:19.116475 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:19.116481    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ae0814a-bb2b-4049-8980-2e1654c9dcda-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:51:19.287283 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:19.287213    2564 generic.go:356] "Generic (PLEG): container finished" podID="6ae0814a-bb2b-4049-8980-2e1654c9dcda" containerID="9c41c2826870f5322cd66ce847cf00afc9141ed8ed82a8a0c86cfaa05b28ac42" exitCode=0
Jul 02 21:51:19.287283 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:19.287272    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz"
Jul 02 21:51:19.287471 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:19.287293    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz" event={"ID":"6ae0814a-bb2b-4049-8980-2e1654c9dcda","Type":"ContainerDied","Data":"9c41c2826870f5322cd66ce847cf00afc9141ed8ed82a8a0c86cfaa05b28ac42"}
Jul 02 21:51:19.287471 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:19.287342    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz" event={"ID":"6ae0814a-bb2b-4049-8980-2e1654c9dcda","Type":"ContainerDied","Data":"7b7e5343d9a7294caccdbf40befaeda76b11642b7f84630fcf7c6432e46bf2ec"}
Jul 02 21:51:19.287471 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:19.287369    2564 scope.go:117] "RemoveContainer" containerID="4fc1687a8538f5a9ddf1cb752a9fe35bade403db0792dce317e8a8e88a332f62"
Jul 02 21:51:19.288150 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:19.288125    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5" podUID="8bfa0b5e-ac46-4503-a2d0-c0a679de6543" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 02 21:51:19.295786 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:19.295771    2564 scope.go:117] "RemoveContainer" containerID="9c41c2826870f5322cd66ce847cf00afc9141ed8ed82a8a0c86cfaa05b28ac42"
Jul 02 21:51:19.302825 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:19.302809    2564 scope.go:117] "RemoveContainer" containerID="4fc1687a8538f5a9ddf1cb752a9fe35bade403db0792dce317e8a8e88a332f62"
Jul 02 21:51:19.303054 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:51:19.303036    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fc1687a8538f5a9ddf1cb752a9fe35bade403db0792dce317e8a8e88a332f62\": container with ID starting with 4fc1687a8538f5a9ddf1cb752a9fe35bade403db0792dce317e8a8e88a332f62 not found: ID does not exist" containerID="4fc1687a8538f5a9ddf1cb752a9fe35bade403db0792dce317e8a8e88a332f62"
Jul 02 21:51:19.303097 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:19.303062    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fc1687a8538f5a9ddf1cb752a9fe35bade403db0792dce317e8a8e88a332f62"} err="failed to get container status \"4fc1687a8538f5a9ddf1cb752a9fe35bade403db0792dce317e8a8e88a332f62\": rpc error: code = NotFound desc = could not find container \"4fc1687a8538f5a9ddf1cb752a9fe35bade403db0792dce317e8a8e88a332f62\": container with ID starting with 4fc1687a8538f5a9ddf1cb752a9fe35bade403db0792dce317e8a8e88a332f62 not found: ID does not exist"
Jul 02 21:51:19.303097 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:19.303077    2564 scope.go:117] "RemoveContainer" containerID="9c41c2826870f5322cd66ce847cf00afc9141ed8ed82a8a0c86cfaa05b28ac42"
Jul 02 21:51:19.303346 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:51:19.303329    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c41c2826870f5322cd66ce847cf00afc9141ed8ed82a8a0c86cfaa05b28ac42\": container with ID starting with 9c41c2826870f5322cd66ce847cf00afc9141ed8ed82a8a0c86cfaa05b28ac42 not found: ID does not exist" containerID="9c41c2826870f5322cd66ce847cf00afc9141ed8ed82a8a0c86cfaa05b28ac42"
Jul 02 21:51:19.303391 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:19.303352    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c41c2826870f5322cd66ce847cf00afc9141ed8ed82a8a0c86cfaa05b28ac42"} err="failed to get container status \"9c41c2826870f5322cd66ce847cf00afc9141ed8ed82a8a0c86cfaa05b28ac42\": rpc error: code = NotFound desc = could not find container \"9c41c2826870f5322cd66ce847cf00afc9141ed8ed82a8a0c86cfaa05b28ac42\": container with ID starting with 9c41c2826870f5322cd66ce847cf00afc9141ed8ed82a8a0c86cfaa05b28ac42 not found: ID does not exist"
Jul 02 21:51:19.307653 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:19.307635    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz"]
Jul 02 21:51:19.311323 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:19.311302    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8bf58-predictor-695db95966-9w5pz"]
Jul 02 21:51:20.071589 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:20.071557    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ae0814a-bb2b-4049-8980-2e1654c9dcda" path="/var/lib/kubelet/pods/6ae0814a-bb2b-4049-8980-2e1654c9dcda/volumes"
Jul 02 21:51:24.292454 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:24.292422    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5"
Jul 02 21:51:24.292905 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:24.292802    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5" podUID="8bfa0b5e-ac46-4503-a2d0-c0a679de6543" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 02 21:51:28.180087 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:28.180047    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk" podUID="7c34c887-38b7-4dcd-8043-c726de59f2f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 21:51:34.293059 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:34.293017    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5" podUID="8bfa0b5e-ac46-4503-a2d0-c0a679de6543" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 02 21:51:38.180911 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:38.180878    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk"
Jul 02 21:51:44.293670 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:44.293629    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5" podUID="8bfa0b5e-ac46-4503-a2d0-c0a679de6543" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 02 21:51:54.293793 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:51:54.293708    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5" podUID="8bfa0b5e-ac46-4503-a2d0-c0a679de6543" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 02 21:52:00.258735 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:00.258700    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk"]
Jul 02 21:52:00.259197 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:00.258907    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk" podUID="7c34c887-38b7-4dcd-8043-c726de59f2f7" containerName="kserve-container" containerID="cri-o://cf2a879d94e638d025cacf861d52ae899a86743201e76ffe8a8dd8b1aa50b493" gracePeriod=30
Jul 02 21:52:00.259197 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:00.258976    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk" podUID="7c34c887-38b7-4dcd-8043-c726de59f2f7" containerName="kube-rbac-proxy" containerID="cri-o://d297231d5b095d74d2c4892cd4679cf6e21714eac3d2be340d3c945b805ca5a4" gracePeriod=30
Jul 02 21:52:00.270494 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:00.270466    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88"]
Jul 02 21:52:00.270891 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:00.270873    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ae0814a-bb2b-4049-8980-2e1654c9dcda" containerName="kube-rbac-proxy"
Jul 02 21:52:00.270891 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:00.270891    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ae0814a-bb2b-4049-8980-2e1654c9dcda" containerName="kube-rbac-proxy"
Jul 02 21:52:00.271023 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:00.270905    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ae0814a-bb2b-4049-8980-2e1654c9dcda" containerName="kserve-container"
Jul 02 21:52:00.271023 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:00.270911    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ae0814a-bb2b-4049-8980-2e1654c9dcda" containerName="kserve-container"
Jul 02 21:52:00.271023 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:00.270970    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ae0814a-bb2b-4049-8980-2e1654c9dcda" containerName="kserve-container"
Jul 02 21:52:00.271023 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:00.270981    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ae0814a-bb2b-4049-8980-2e1654c9dcda" containerName="kube-rbac-proxy"
Jul 02 21:52:00.275350 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:00.275332    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88"
Jul 02 21:52:00.277590 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:00.277563    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-67ac0-predictor-serving-cert\""
Jul 02 21:52:00.277710 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:00.277573    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-67ac0-kube-rbac-proxy-sar-config\""
Jul 02 21:52:00.285849 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:00.285827    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88"]
Jul 02 21:52:00.359032 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:00.359003    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53cf0706-89f4-4e4f-90fa-f798c6c55398-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88\" (UID: \"53cf0706-89f4-4e4f-90fa-f798c6c55398\") " pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88"
Jul 02 21:52:00.359225 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:00.359063    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53cf0706-89f4-4e4f-90fa-f798c6c55398-proxy-tls\") pod \"success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88\" (UID: \"53cf0706-89f4-4e4f-90fa-f798c6c55398\") " pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88"
Jul 02 21:52:00.359225 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:00.359164    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bzrs\" (UniqueName: \"kubernetes.io/projected/53cf0706-89f4-4e4f-90fa-f798c6c55398-kube-api-access-7bzrs\") pod \"success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88\" (UID: \"53cf0706-89f4-4e4f-90fa-f798c6c55398\") " pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88"
Jul 02 21:52:00.457368 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:00.457336    2564 generic.go:356] "Generic (PLEG): container finished" podID="7c34c887-38b7-4dcd-8043-c726de59f2f7" containerID="d297231d5b095d74d2c4892cd4679cf6e21714eac3d2be340d3c945b805ca5a4" exitCode=2
Jul 02 21:52:00.457540 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:00.457407    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk" event={"ID":"7c34c887-38b7-4dcd-8043-c726de59f2f7","Type":"ContainerDied","Data":"d297231d5b095d74d2c4892cd4679cf6e21714eac3d2be340d3c945b805ca5a4"}
Jul 02 21:52:00.459737 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:00.459718    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53cf0706-89f4-4e4f-90fa-f798c6c55398-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88\" (UID: \"53cf0706-89f4-4e4f-90fa-f798c6c55398\") " pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88"
Jul 02 21:52:00.459790 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:00.459759    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53cf0706-89f4-4e4f-90fa-f798c6c55398-proxy-tls\") pod \"success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88\" (UID: \"53cf0706-89f4-4e4f-90fa-f798c6c55398\") " pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88"
Jul 02 21:52:00.459790 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:00.459784    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bzrs\" (UniqueName: \"kubernetes.io/projected/53cf0706-89f4-4e4f-90fa-f798c6c55398-kube-api-access-7bzrs\") pod \"success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88\" (UID: \"53cf0706-89f4-4e4f-90fa-f798c6c55398\") " pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88"
Jul 02 21:52:00.460377 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:00.460354    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53cf0706-89f4-4e4f-90fa-f798c6c55398-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88\" (UID: \"53cf0706-89f4-4e4f-90fa-f798c6c55398\") " pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88"
Jul 02 21:52:00.462132 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:00.462101    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53cf0706-89f4-4e4f-90fa-f798c6c55398-proxy-tls\") pod \"success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88\" (UID: \"53cf0706-89f4-4e4f-90fa-f798c6c55398\") " pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88"
Jul 02 21:52:00.466956 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:00.466938    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bzrs\" (UniqueName: \"kubernetes.io/projected/53cf0706-89f4-4e4f-90fa-f798c6c55398-kube-api-access-7bzrs\") pod \"success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88\" (UID: \"53cf0706-89f4-4e4f-90fa-f798c6c55398\") " pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88"
Jul 02 21:52:00.586290 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:00.586254    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88"
Jul 02 21:52:00.716885 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:00.716861    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88"]
Jul 02 21:52:00.719096 ip-10-0-130-230 kubenswrapper[2564]: W0702 21:52:00.719067    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53cf0706_89f4_4e4f_90fa_f798c6c55398.slice/crio-d82ef8c869f37f1d9012d041a655b91fc180c2292364b60e914d1f5649ea4e20 WatchSource:0}: Error finding container d82ef8c869f37f1d9012d041a655b91fc180c2292364b60e914d1f5649ea4e20: Status 404 returned error can't find the container with id d82ef8c869f37f1d9012d041a655b91fc180c2292364b60e914d1f5649ea4e20
Jul 02 21:52:01.463788 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:01.463697    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88" event={"ID":"53cf0706-89f4-4e4f-90fa-f798c6c55398","Type":"ContainerStarted","Data":"ac3355110f2df5f9031959f0305ea50780ebd8d42b7524299cdc660ca8643705"}
Jul 02 21:52:01.464230 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:01.463812    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88" event={"ID":"53cf0706-89f4-4e4f-90fa-f798c6c55398","Type":"ContainerStarted","Data":"11df5af35cfb6225eb07228251a6792c418eca7b065c92d86665c320191a6a99"}
Jul 02 21:52:01.464230 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:01.463838    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88" event={"ID":"53cf0706-89f4-4e4f-90fa-f798c6c55398","Type":"ContainerStarted","Data":"d82ef8c869f37f1d9012d041a655b91fc180c2292364b60e914d1f5649ea4e20"}
Jul 02 21:52:01.464230 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:01.463864    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88"
Jul 02 21:52:01.480664 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:01.480615    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88" podStartSLOduration=1.4806009869999999 podStartE2EDuration="1.480600987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:52:01.479345854 +0000 UTC m=+1982.316585902" watchObservedRunningTime="2026-07-02 21:52:01.480600987 +0000 UTC m=+1982.317841034"
Jul 02 21:52:02.468577 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:02.468539    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88"
Jul 02 21:52:02.469841 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:02.469816    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88" podUID="53cf0706-89f4-4e4f-90fa-f798c6c55398" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 02 21:52:03.174268 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:03.174229    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk" podUID="7c34c887-38b7-4dcd-8043-c726de59f2f7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 02 21:52:03.409173 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:03.409147    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk"
Jul 02 21:52:03.473617 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:03.473535    2564 generic.go:356] "Generic (PLEG): container finished" podID="7c34c887-38b7-4dcd-8043-c726de59f2f7" containerID="cf2a879d94e638d025cacf861d52ae899a86743201e76ffe8a8dd8b1aa50b493" exitCode=0
Jul 02 21:52:03.473617 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:03.473585    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk"
Jul 02 21:52:03.473617 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:03.473595    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk" event={"ID":"7c34c887-38b7-4dcd-8043-c726de59f2f7","Type":"ContainerDied","Data":"cf2a879d94e638d025cacf861d52ae899a86743201e76ffe8a8dd8b1aa50b493"}
Jul 02 21:52:03.474102 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:03.473667    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk" event={"ID":"7c34c887-38b7-4dcd-8043-c726de59f2f7","Type":"ContainerDied","Data":"19433b42ae973029d3a2cb6873c48923e9f7e04b23a852c2e97b3a2e22f5aeb6"}
Jul 02 21:52:03.474102 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:03.473695    2564 scope.go:117] "RemoveContainer" containerID="d297231d5b095d74d2c4892cd4679cf6e21714eac3d2be340d3c945b805ca5a4"
Jul 02 21:52:03.474189 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:03.474131    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88" podUID="53cf0706-89f4-4e4f-90fa-f798c6c55398" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 02 21:52:03.481992 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:03.481976    2564 scope.go:117] "RemoveContainer" containerID="cf2a879d94e638d025cacf861d52ae899a86743201e76ffe8a8dd8b1aa50b493"
Jul 02 21:52:03.488143 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:03.488105    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c34c887-38b7-4dcd-8043-c726de59f2f7-proxy-tls\") pod \"7c34c887-38b7-4dcd-8043-c726de59f2f7\" (UID: \"7c34c887-38b7-4dcd-8043-c726de59f2f7\") "
Jul 02 21:52:03.488244 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:03.488185    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pw4rg\" (UniqueName: \"kubernetes.io/projected/7c34c887-38b7-4dcd-8043-c726de59f2f7-kube-api-access-pw4rg\") pod \"7c34c887-38b7-4dcd-8043-c726de59f2f7\" (UID: \"7c34c887-38b7-4dcd-8043-c726de59f2f7\") "
Jul 02 21:52:03.488244 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:03.488239    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c34c887-38b7-4dcd-8043-c726de59f2f7-kube-rbac-proxy-sar-config\") pod \"7c34c887-38b7-4dcd-8043-c726de59f2f7\" (UID: \"7c34c887-38b7-4dcd-8043-c726de59f2f7\") "
Jul 02 21:52:03.488661 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:03.488621    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c34c887-38b7-4dcd-8043-c726de59f2f7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7c34c887-38b7-4dcd-8043-c726de59f2f7" (UID: "7c34c887-38b7-4dcd-8043-c726de59f2f7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:52:03.489527 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:03.489508    2564 scope.go:117] "RemoveContainer" containerID="d297231d5b095d74d2c4892cd4679cf6e21714eac3d2be340d3c945b805ca5a4"
Jul 02 21:52:03.489797 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:52:03.489779    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d297231d5b095d74d2c4892cd4679cf6e21714eac3d2be340d3c945b805ca5a4\": container with ID starting with d297231d5b095d74d2c4892cd4679cf6e21714eac3d2be340d3c945b805ca5a4 not found: ID does not exist" containerID="d297231d5b095d74d2c4892cd4679cf6e21714eac3d2be340d3c945b805ca5a4"
Jul 02 21:52:03.489886 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:03.489808    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d297231d5b095d74d2c4892cd4679cf6e21714eac3d2be340d3c945b805ca5a4"} err="failed to get container status \"d297231d5b095d74d2c4892cd4679cf6e21714eac3d2be340d3c945b805ca5a4\": rpc error: code = NotFound desc = could not find container \"d297231d5b095d74d2c4892cd4679cf6e21714eac3d2be340d3c945b805ca5a4\": container with ID starting with d297231d5b095d74d2c4892cd4679cf6e21714eac3d2be340d3c945b805ca5a4 not found: ID does not exist"
Jul 02 21:52:03.489886 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:03.489828    2564 scope.go:117] "RemoveContainer" containerID="cf2a879d94e638d025cacf861d52ae899a86743201e76ffe8a8dd8b1aa50b493"
Jul 02 21:52:03.490087 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:03.490063    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c34c887-38b7-4dcd-8043-c726de59f2f7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7c34c887-38b7-4dcd-8043-c726de59f2f7" (UID: "7c34c887-38b7-4dcd-8043-c726de59f2f7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:52:03.490216 ip-10-0-130-230 kubenswrapper[2564]: E0702 21:52:03.490101    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf2a879d94e638d025cacf861d52ae899a86743201e76ffe8a8dd8b1aa50b493\": container with ID starting with cf2a879d94e638d025cacf861d52ae899a86743201e76ffe8a8dd8b1aa50b493 not found: ID does not exist" containerID="cf2a879d94e638d025cacf861d52ae899a86743201e76ffe8a8dd8b1aa50b493"
Jul 02 21:52:03.490216 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:03.490150    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf2a879d94e638d025cacf861d52ae899a86743201e76ffe8a8dd8b1aa50b493"} err="failed to get container status \"cf2a879d94e638d025cacf861d52ae899a86743201e76ffe8a8dd8b1aa50b493\": rpc error: code = NotFound desc = could not find container \"cf2a879d94e638d025cacf861d52ae899a86743201e76ffe8a8dd8b1aa50b493\": container with ID starting with cf2a879d94e638d025cacf861d52ae899a86743201e76ffe8a8dd8b1aa50b493 not found: ID does not exist"
Jul 02 21:52:03.490216 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:03.490204    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c34c887-38b7-4dcd-8043-c726de59f2f7-kube-api-access-pw4rg" (OuterVolumeSpecName: "kube-api-access-pw4rg") pod "7c34c887-38b7-4dcd-8043-c726de59f2f7" (UID: "7c34c887-38b7-4dcd-8043-c726de59f2f7"). InnerVolumeSpecName "kube-api-access-pw4rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:52:03.589848 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:03.589810    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw4rg\" (UniqueName: \"kubernetes.io/projected/7c34c887-38b7-4dcd-8043-c726de59f2f7-kube-api-access-pw4rg\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:52:03.589848 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:03.589839    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c34c887-38b7-4dcd-8043-c726de59f2f7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:52:03.589848 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:03.589849    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c34c887-38b7-4dcd-8043-c726de59f2f7-proxy-tls\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 21:52:03.794960 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:03.794929    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk"]
Jul 02 21:52:03.798875 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:03.798852    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c9d16-predictor-664b8c5c66-7g9zk"]
Jul 02 21:52:04.076605 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:04.071729    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c34c887-38b7-4dcd-8043-c726de59f2f7" path="/var/lib/kubelet/pods/7c34c887-38b7-4dcd-8043-c726de59f2f7/volumes"
Jul 02 21:52:04.293944 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:04.293917    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5"
Jul 02 21:52:08.478953 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:08.478926    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88"
Jul 02 21:52:08.479501 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:08.479419    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88" podUID="53cf0706-89f4-4e4f-90fa-f798c6c55398" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 02 21:52:18.479882 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:18.479840    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88" podUID="53cf0706-89f4-4e4f-90fa-f798c6c55398" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 02 21:52:28.480131 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:28.480067    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88" podUID="53cf0706-89f4-4e4f-90fa-f798c6c55398" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 02 21:52:38.480365 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:38.480328    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88" podUID="53cf0706-89f4-4e4f-90fa-f798c6c55398" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 02 21:52:48.480679 ip-10-0-130-230 kubenswrapper[2564]: I0702 21:52:48.480649    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88"
Jul 02 22:01:15.077693 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:15.077599    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88"]
Jul 02 22:01:15.080846 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:15.077940    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88" podUID="53cf0706-89f4-4e4f-90fa-f798c6c55398" containerName="kserve-container" containerID="cri-o://11df5af35cfb6225eb07228251a6792c418eca7b065c92d86665c320191a6a99" gracePeriod=30
Jul 02 22:01:15.080846 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:15.077968    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88" podUID="53cf0706-89f4-4e4f-90fa-f798c6c55398" containerName="kube-rbac-proxy" containerID="cri-o://ac3355110f2df5f9031959f0305ea50780ebd8d42b7524299cdc660ca8643705" gracePeriod=30
Jul 02 22:01:15.729754 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:15.729720    2564 generic.go:356] "Generic (PLEG): container finished" podID="53cf0706-89f4-4e4f-90fa-f798c6c55398" containerID="ac3355110f2df5f9031959f0305ea50780ebd8d42b7524299cdc660ca8643705" exitCode=2
Jul 02 22:01:15.729925 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:15.729789    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88" event={"ID":"53cf0706-89f4-4e4f-90fa-f798c6c55398","Type":"ContainerDied","Data":"ac3355110f2df5f9031959f0305ea50780ebd8d42b7524299cdc660ca8643705"}
Jul 02 22:01:18.327486 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:18.327459    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88"
Jul 02 22:01:18.417804 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:18.417712    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bzrs\" (UniqueName: \"kubernetes.io/projected/53cf0706-89f4-4e4f-90fa-f798c6c55398-kube-api-access-7bzrs\") pod \"53cf0706-89f4-4e4f-90fa-f798c6c55398\" (UID: \"53cf0706-89f4-4e4f-90fa-f798c6c55398\") "
Jul 02 22:01:18.417804 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:18.417757    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53cf0706-89f4-4e4f-90fa-f798c6c55398-kube-rbac-proxy-sar-config\") pod \"53cf0706-89f4-4e4f-90fa-f798c6c55398\" (UID: \"53cf0706-89f4-4e4f-90fa-f798c6c55398\") "
Jul 02 22:01:18.418037 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:18.417808    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53cf0706-89f4-4e4f-90fa-f798c6c55398-proxy-tls\") pod \"53cf0706-89f4-4e4f-90fa-f798c6c55398\" (UID: \"53cf0706-89f4-4e4f-90fa-f798c6c55398\") "
Jul 02 22:01:18.418205 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:18.418183    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53cf0706-89f4-4e4f-90fa-f798c6c55398-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "53cf0706-89f4-4e4f-90fa-f798c6c55398" (UID: "53cf0706-89f4-4e4f-90fa-f798c6c55398"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 22:01:18.419785 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:18.419765    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53cf0706-89f4-4e4f-90fa-f798c6c55398-kube-api-access-7bzrs" (OuterVolumeSpecName: "kube-api-access-7bzrs") pod "53cf0706-89f4-4e4f-90fa-f798c6c55398" (UID: "53cf0706-89f4-4e4f-90fa-f798c6c55398"). InnerVolumeSpecName "kube-api-access-7bzrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 22:01:18.419888 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:18.419791    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53cf0706-89f4-4e4f-90fa-f798c6c55398-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "53cf0706-89f4-4e4f-90fa-f798c6c55398" (UID: "53cf0706-89f4-4e4f-90fa-f798c6c55398"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 22:01:18.519107 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:18.519073    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bzrs\" (UniqueName: \"kubernetes.io/projected/53cf0706-89f4-4e4f-90fa-f798c6c55398-kube-api-access-7bzrs\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 22:01:18.519107 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:18.519100    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53cf0706-89f4-4e4f-90fa-f798c6c55398-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 22:01:18.519107 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:18.519137    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53cf0706-89f4-4e4f-90fa-f798c6c55398-proxy-tls\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 22:01:18.744486 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:18.744453    2564 generic.go:356] "Generic (PLEG): container finished" podID="53cf0706-89f4-4e4f-90fa-f798c6c55398" containerID="11df5af35cfb6225eb07228251a6792c418eca7b065c92d86665c320191a6a99" exitCode=0
Jul 02 22:01:18.744660 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:18.744500    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88"
Jul 02 22:01:18.744660 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:18.744545    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88" event={"ID":"53cf0706-89f4-4e4f-90fa-f798c6c55398","Type":"ContainerDied","Data":"11df5af35cfb6225eb07228251a6792c418eca7b065c92d86665c320191a6a99"}
Jul 02 22:01:18.744660 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:18.744603    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88" event={"ID":"53cf0706-89f4-4e4f-90fa-f798c6c55398","Type":"ContainerDied","Data":"d82ef8c869f37f1d9012d041a655b91fc180c2292364b60e914d1f5649ea4e20"}
Jul 02 22:01:18.744660 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:18.744632    2564 scope.go:117] "RemoveContainer" containerID="ac3355110f2df5f9031959f0305ea50780ebd8d42b7524299cdc660ca8643705"
Jul 02 22:01:18.753617 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:18.753595    2564 scope.go:117] "RemoveContainer" containerID="11df5af35cfb6225eb07228251a6792c418eca7b065c92d86665c320191a6a99"
Jul 02 22:01:18.761514 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:18.761491    2564 scope.go:117] "RemoveContainer" containerID="ac3355110f2df5f9031959f0305ea50780ebd8d42b7524299cdc660ca8643705"
Jul 02 22:01:18.761808 ip-10-0-130-230 kubenswrapper[2564]: E0702 22:01:18.761789    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac3355110f2df5f9031959f0305ea50780ebd8d42b7524299cdc660ca8643705\": container with ID starting with ac3355110f2df5f9031959f0305ea50780ebd8d42b7524299cdc660ca8643705 not found: ID does not exist" containerID="ac3355110f2df5f9031959f0305ea50780ebd8d42b7524299cdc660ca8643705"
Jul 02 22:01:18.761878 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:18.761818    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac3355110f2df5f9031959f0305ea50780ebd8d42b7524299cdc660ca8643705"} err="failed to get container status \"ac3355110f2df5f9031959f0305ea50780ebd8d42b7524299cdc660ca8643705\": rpc error: code = NotFound desc = could not find container \"ac3355110f2df5f9031959f0305ea50780ebd8d42b7524299cdc660ca8643705\": container with ID starting with ac3355110f2df5f9031959f0305ea50780ebd8d42b7524299cdc660ca8643705 not found: ID does not exist"
Jul 02 22:01:18.761878 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:18.761836    2564 scope.go:117] "RemoveContainer" containerID="11df5af35cfb6225eb07228251a6792c418eca7b065c92d86665c320191a6a99"
Jul 02 22:01:18.762078 ip-10-0-130-230 kubenswrapper[2564]: E0702 22:01:18.762058    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11df5af35cfb6225eb07228251a6792c418eca7b065c92d86665c320191a6a99\": container with ID starting with 11df5af35cfb6225eb07228251a6792c418eca7b065c92d86665c320191a6a99 not found: ID does not exist" containerID="11df5af35cfb6225eb07228251a6792c418eca7b065c92d86665c320191a6a99"
Jul 02 22:01:18.762143 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:18.762084    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11df5af35cfb6225eb07228251a6792c418eca7b065c92d86665c320191a6a99"} err="failed to get container status \"11df5af35cfb6225eb07228251a6792c418eca7b065c92d86665c320191a6a99\": rpc error: code = NotFound desc = could not find container \"11df5af35cfb6225eb07228251a6792c418eca7b065c92d86665c320191a6a99\": container with ID starting with 11df5af35cfb6225eb07228251a6792c418eca7b065c92d86665c320191a6a99 not found: ID does not exist"
Jul 02 22:01:18.766869 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:18.766842    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88"]
Jul 02 22:01:18.770969 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:18.770948    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-67ac0-predictor-6f7fc66ddb-54h88"]
Jul 02 22:01:20.071745 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:01:20.071713    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53cf0706-89f4-4e4f-90fa-f798c6c55398" path="/var/lib/kubelet/pods/53cf0706-89f4-4e4f-90fa-f798c6c55398/volumes"
Jul 02 22:08:35.379369 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:35.379290    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5"]
Jul 02 22:08:35.382364 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:35.379546    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5" podUID="8bfa0b5e-ac46-4503-a2d0-c0a679de6543" containerName="kserve-container" containerID="cri-o://91c4bc06164df295d99710b89e78c9a7e9f85327ba5398d49bb7681ff8cc4fe3" gracePeriod=30
Jul 02 22:08:35.382364 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:35.379556    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5" podUID="8bfa0b5e-ac46-4503-a2d0-c0a679de6543" containerName="kube-rbac-proxy" containerID="cri-o://30a65ff3e290a9f448292426aea5bce543a8b17f3bd09cc41c16e0d095a6ae78" gracePeriod=30
Jul 02 22:08:36.341746 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.341707    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-h8jrj/must-gather-dn4kl"]
Jul 02 22:08:36.342054 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.342042    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c34c887-38b7-4dcd-8043-c726de59f2f7" containerName="kserve-container"
Jul 02 22:08:36.342103 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.342055    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c34c887-38b7-4dcd-8043-c726de59f2f7" containerName="kserve-container"
Jul 02 22:08:36.342103 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.342068    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53cf0706-89f4-4e4f-90fa-f798c6c55398" containerName="kserve-container"
Jul 02 22:08:36.342103 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.342075    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="53cf0706-89f4-4e4f-90fa-f798c6c55398" containerName="kserve-container"
Jul 02 22:08:36.342103 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.342093    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c34c887-38b7-4dcd-8043-c726de59f2f7" containerName="kube-rbac-proxy"
Jul 02 22:08:36.342103 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.342099    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c34c887-38b7-4dcd-8043-c726de59f2f7" containerName="kube-rbac-proxy"
Jul 02 22:08:36.342296 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.342106    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53cf0706-89f4-4e4f-90fa-f798c6c55398" containerName="kube-rbac-proxy"
Jul 02 22:08:36.342296 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.342132    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="53cf0706-89f4-4e4f-90fa-f798c6c55398" containerName="kube-rbac-proxy"
Jul 02 22:08:36.342296 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.342193    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c34c887-38b7-4dcd-8043-c726de59f2f7" containerName="kserve-container"
Jul 02 22:08:36.342296 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.342201    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="53cf0706-89f4-4e4f-90fa-f798c6c55398" containerName="kube-rbac-proxy"
Jul 02 22:08:36.342296 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.342208    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c34c887-38b7-4dcd-8043-c726de59f2f7" containerName="kube-rbac-proxy"
Jul 02 22:08:36.342296 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.342217    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="53cf0706-89f4-4e4f-90fa-f798c6c55398" containerName="kserve-container"
Jul 02 22:08:36.345476 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.345459    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h8jrj/must-gather-dn4kl"
Jul 02 22:08:36.347790 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.347763    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h8jrj\"/\"kube-root-ca.crt\""
Jul 02 22:08:36.347939 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.347821    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h8jrj\"/\"openshift-service-ca.crt\""
Jul 02 22:08:36.348565 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.348548    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-h8jrj\"/\"default-dockercfg-v9q4m\""
Jul 02 22:08:36.354221 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.354187    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h8jrj/must-gather-dn4kl"]
Jul 02 22:08:36.425332 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.425300    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f1f31e3f-b483-427d-8477-26afdefe79dd-must-gather-output\") pod \"must-gather-dn4kl\" (UID: \"f1f31e3f-b483-427d-8477-26afdefe79dd\") " pod="openshift-must-gather-h8jrj/must-gather-dn4kl"
Jul 02 22:08:36.425721 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.425363    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r4k6\" (UniqueName: \"kubernetes.io/projected/f1f31e3f-b483-427d-8477-26afdefe79dd-kube-api-access-7r4k6\") pod \"must-gather-dn4kl\" (UID: \"f1f31e3f-b483-427d-8477-26afdefe79dd\") " pod="openshift-must-gather-h8jrj/must-gather-dn4kl"
Jul 02 22:08:36.494781 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.494750    2564 generic.go:356] "Generic (PLEG): container finished" podID="8bfa0b5e-ac46-4503-a2d0-c0a679de6543" containerID="30a65ff3e290a9f448292426aea5bce543a8b17f3bd09cc41c16e0d095a6ae78" exitCode=2
Jul 02 22:08:36.494947 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.494830    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5" event={"ID":"8bfa0b5e-ac46-4503-a2d0-c0a679de6543","Type":"ContainerDied","Data":"30a65ff3e290a9f448292426aea5bce543a8b17f3bd09cc41c16e0d095a6ae78"}
Jul 02 22:08:36.526327 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.526297    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f1f31e3f-b483-427d-8477-26afdefe79dd-must-gather-output\") pod \"must-gather-dn4kl\" (UID: \"f1f31e3f-b483-427d-8477-26afdefe79dd\") " pod="openshift-must-gather-h8jrj/must-gather-dn4kl"
Jul 02 22:08:36.526499 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.526385    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r4k6\" (UniqueName: \"kubernetes.io/projected/f1f31e3f-b483-427d-8477-26afdefe79dd-kube-api-access-7r4k6\") pod \"must-gather-dn4kl\" (UID: \"f1f31e3f-b483-427d-8477-26afdefe79dd\") " pod="openshift-must-gather-h8jrj/must-gather-dn4kl"
Jul 02 22:08:36.526641 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.526624    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f1f31e3f-b483-427d-8477-26afdefe79dd-must-gather-output\") pod \"must-gather-dn4kl\" (UID: \"f1f31e3f-b483-427d-8477-26afdefe79dd\") " pod="openshift-must-gather-h8jrj/must-gather-dn4kl"
Jul 02 22:08:36.534937 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.534904    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r4k6\" (UniqueName: \"kubernetes.io/projected/f1f31e3f-b483-427d-8477-26afdefe79dd-kube-api-access-7r4k6\") pod \"must-gather-dn4kl\" (UID: \"f1f31e3f-b483-427d-8477-26afdefe79dd\") " pod="openshift-must-gather-h8jrj/must-gather-dn4kl"
Jul 02 22:08:36.665925 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.665849    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h8jrj/must-gather-dn4kl"
Jul 02 22:08:36.794626 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.794532    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h8jrj/must-gather-dn4kl"]
Jul 02 22:08:36.797458 ip-10-0-130-230 kubenswrapper[2564]: W0702 22:08:36.797427    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1f31e3f_b483_427d_8477_26afdefe79dd.slice/crio-3cdcb266f5057f9cf38993fd47a7dc180a3912ab7d5dc9dab4b775a2b11ccc7a WatchSource:0}: Error finding container 3cdcb266f5057f9cf38993fd47a7dc180a3912ab7d5dc9dab4b775a2b11ccc7a: Status 404 returned error can't find the container with id 3cdcb266f5057f9cf38993fd47a7dc180a3912ab7d5dc9dab4b775a2b11ccc7a
Jul 02 22:08:36.799207 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:36.799185    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 22:08:37.501873 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:37.501827    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h8jrj/must-gather-dn4kl" event={"ID":"f1f31e3f-b483-427d-8477-26afdefe79dd","Type":"ContainerStarted","Data":"3cdcb266f5057f9cf38993fd47a7dc180a3912ab7d5dc9dab4b775a2b11ccc7a"}
Jul 02 22:08:38.854410 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:38.854302    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5"
Jul 02 22:08:38.948767 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:38.948670    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fnmt\" (UniqueName: \"kubernetes.io/projected/8bfa0b5e-ac46-4503-a2d0-c0a679de6543-kube-api-access-5fnmt\") pod \"8bfa0b5e-ac46-4503-a2d0-c0a679de6543\" (UID: \"8bfa0b5e-ac46-4503-a2d0-c0a679de6543\") "
Jul 02 22:08:38.948939 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:38.948824    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bfa0b5e-ac46-4503-a2d0-c0a679de6543-proxy-tls\") pod \"8bfa0b5e-ac46-4503-a2d0-c0a679de6543\" (UID: \"8bfa0b5e-ac46-4503-a2d0-c0a679de6543\") "
Jul 02 22:08:38.948939 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:38.948859    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bfa0b5e-ac46-4503-a2d0-c0a679de6543-kube-rbac-proxy-sar-config\") pod \"8bfa0b5e-ac46-4503-a2d0-c0a679de6543\" (UID: \"8bfa0b5e-ac46-4503-a2d0-c0a679de6543\") "
Jul 02 22:08:38.949366 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:38.949303    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bfa0b5e-ac46-4503-a2d0-c0a679de6543-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8bfa0b5e-ac46-4503-a2d0-c0a679de6543" (UID: "8bfa0b5e-ac46-4503-a2d0-c0a679de6543"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 22:08:38.951607 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:38.951570    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bfa0b5e-ac46-4503-a2d0-c0a679de6543-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8bfa0b5e-ac46-4503-a2d0-c0a679de6543" (UID: "8bfa0b5e-ac46-4503-a2d0-c0a679de6543"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 22:08:38.951607 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:38.951573    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bfa0b5e-ac46-4503-a2d0-c0a679de6543-kube-api-access-5fnmt" (OuterVolumeSpecName: "kube-api-access-5fnmt") pod "8bfa0b5e-ac46-4503-a2d0-c0a679de6543" (UID: "8bfa0b5e-ac46-4503-a2d0-c0a679de6543"). InnerVolumeSpecName "kube-api-access-5fnmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 22:08:39.049801 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:39.049761    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fnmt\" (UniqueName: \"kubernetes.io/projected/8bfa0b5e-ac46-4503-a2d0-c0a679de6543-kube-api-access-5fnmt\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 22:08:39.049801 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:39.049804    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bfa0b5e-ac46-4503-a2d0-c0a679de6543-proxy-tls\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 22:08:39.050053 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:39.049820    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bfa0b5e-ac46-4503-a2d0-c0a679de6543-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 22:08:39.512653 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:39.512616    2564 generic.go:356] "Generic (PLEG): container finished" podID="8bfa0b5e-ac46-4503-a2d0-c0a679de6543" containerID="91c4bc06164df295d99710b89e78c9a7e9f85327ba5398d49bb7681ff8cc4fe3" exitCode=0
Jul 02 22:08:39.512819 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:39.512703    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5"
Jul 02 22:08:39.512819 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:39.512708    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5" event={"ID":"8bfa0b5e-ac46-4503-a2d0-c0a679de6543","Type":"ContainerDied","Data":"91c4bc06164df295d99710b89e78c9a7e9f85327ba5398d49bb7681ff8cc4fe3"}
Jul 02 22:08:39.512819 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:39.512787    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5" event={"ID":"8bfa0b5e-ac46-4503-a2d0-c0a679de6543","Type":"ContainerDied","Data":"ece8497fb697fb0e600c3b34f320c8761573f0c4bfbde7a0c67b520fbb1d5da3"}
Jul 02 22:08:39.513022 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:39.512820    2564 scope.go:117] "RemoveContainer" containerID="30a65ff3e290a9f448292426aea5bce543a8b17f3bd09cc41c16e0d095a6ae78"
Jul 02 22:08:39.535340 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:39.535310    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5"]
Jul 02 22:08:39.539747 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:39.539719    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-76a5f-predictor-68958f7c46-8ddl5"]
Jul 02 22:08:40.072957 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:40.072927    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bfa0b5e-ac46-4503-a2d0-c0a679de6543" path="/var/lib/kubelet/pods/8bfa0b5e-ac46-4503-a2d0-c0a679de6543/volumes"
Jul 02 22:08:41.944707 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:41.944416    2564 scope.go:117] "RemoveContainer" containerID="91c4bc06164df295d99710b89e78c9a7e9f85327ba5398d49bb7681ff8cc4fe3"
Jul 02 22:08:42.826987 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:42.826960    2564 scope.go:117] "RemoveContainer" containerID="30a65ff3e290a9f448292426aea5bce543a8b17f3bd09cc41c16e0d095a6ae78"
Jul 02 22:08:42.827371 ip-10-0-130-230 kubenswrapper[2564]: E0702 22:08:42.827325    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30a65ff3e290a9f448292426aea5bce543a8b17f3bd09cc41c16e0d095a6ae78\": container with ID starting with 30a65ff3e290a9f448292426aea5bce543a8b17f3bd09cc41c16e0d095a6ae78 not found: ID does not exist" containerID="30a65ff3e290a9f448292426aea5bce543a8b17f3bd09cc41c16e0d095a6ae78"
Jul 02 22:08:42.827441 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:42.827386    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30a65ff3e290a9f448292426aea5bce543a8b17f3bd09cc41c16e0d095a6ae78"} err="failed to get container status \"30a65ff3e290a9f448292426aea5bce543a8b17f3bd09cc41c16e0d095a6ae78\": rpc error: code = NotFound desc = could not find container \"30a65ff3e290a9f448292426aea5bce543a8b17f3bd09cc41c16e0d095a6ae78\": container with ID starting with 30a65ff3e290a9f448292426aea5bce543a8b17f3bd09cc41c16e0d095a6ae78 not found: ID does not exist"
Jul 02 22:08:42.827441 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:42.827413    2564 scope.go:117] "RemoveContainer" containerID="91c4bc06164df295d99710b89e78c9a7e9f85327ba5398d49bb7681ff8cc4fe3"
Jul 02 22:08:42.827659 ip-10-0-130-230 kubenswrapper[2564]: E0702 22:08:42.827645    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91c4bc06164df295d99710b89e78c9a7e9f85327ba5398d49bb7681ff8cc4fe3\": container with ID starting with 91c4bc06164df295d99710b89e78c9a7e9f85327ba5398d49bb7681ff8cc4fe3 not found: ID does not exist" containerID="91c4bc06164df295d99710b89e78c9a7e9f85327ba5398d49bb7681ff8cc4fe3"
Jul 02 22:08:42.827706 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:42.827664    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c4bc06164df295d99710b89e78c9a7e9f85327ba5398d49bb7681ff8cc4fe3"} err="failed to get container status \"91c4bc06164df295d99710b89e78c9a7e9f85327ba5398d49bb7681ff8cc4fe3\": rpc error: code = NotFound desc = could not find container \"91c4bc06164df295d99710b89e78c9a7e9f85327ba5398d49bb7681ff8cc4fe3\": container with ID starting with 91c4bc06164df295d99710b89e78c9a7e9f85327ba5398d49bb7681ff8cc4fe3 not found: ID does not exist"
Jul 02 22:08:43.538286 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:43.538252    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h8jrj/must-gather-dn4kl" event={"ID":"f1f31e3f-b483-427d-8477-26afdefe79dd","Type":"ContainerStarted","Data":"3ebd81b5d8510eb9576af3327aec0593cb1ec914111bf1becf7864e297c984fb"}
Jul 02 22:08:43.538700 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:43.538313    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h8jrj/must-gather-dn4kl" event={"ID":"f1f31e3f-b483-427d-8477-26afdefe79dd","Type":"ContainerStarted","Data":"bd963c39f2ba053a29a43f0f9eec092e6876e22fd90bee4f69b638a1c621ff41"}
Jul 02 22:08:43.554814 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:08:43.554761    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-h8jrj/must-gather-dn4kl" podStartSLOduration=1.440071095 podStartE2EDuration="7.55474465s" totalImagesPullingTime="6.114673555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 22:08:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 22:08:43.552937599 +0000 UTC m=+2984.390177647" watchObservedRunningTime="2026-07-02 22:08:43.55474465 +0000 UTC m=+2984.391984696"
Jul 02 22:09:05.637789 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:05.637753    2564 generic.go:356] "Generic (PLEG): container finished" podID="f1f31e3f-b483-427d-8477-26afdefe79dd" containerID="bd963c39f2ba053a29a43f0f9eec092e6876e22fd90bee4f69b638a1c621ff41" exitCode=0
Jul 02 22:09:05.638361 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:05.637827    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h8jrj/must-gather-dn4kl" event={"ID":"f1f31e3f-b483-427d-8477-26afdefe79dd","Type":"ContainerDied","Data":"bd963c39f2ba053a29a43f0f9eec092e6876e22fd90bee4f69b638a1c621ff41"}
Jul 02 22:09:05.638361 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:05.638240    2564 scope.go:117] "RemoveContainer" containerID="bd963c39f2ba053a29a43f0f9eec092e6876e22fd90bee4f69b638a1c621ff41"
Jul 02 22:09:06.062588 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:06.062561    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-h8jrj_must-gather-dn4kl_f1f31e3f-b483-427d-8477-26afdefe79dd/gather/0.log"
Jul 02 22:09:06.675309 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:06.675276    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-28brx/must-gather-g4qp6"]
Jul 02 22:09:06.675719 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:06.675633    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bfa0b5e-ac46-4503-a2d0-c0a679de6543" containerName="kube-rbac-proxy"
Jul 02 22:09:06.675719 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:06.675644    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bfa0b5e-ac46-4503-a2d0-c0a679de6543" containerName="kube-rbac-proxy"
Jul 02 22:09:06.675719 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:06.675658    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bfa0b5e-ac46-4503-a2d0-c0a679de6543" containerName="kserve-container"
Jul 02 22:09:06.675719 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:06.675663    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bfa0b5e-ac46-4503-a2d0-c0a679de6543" containerName="kserve-container"
Jul 02 22:09:06.675853 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:06.675725    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bfa0b5e-ac46-4503-a2d0-c0a679de6543" containerName="kube-rbac-proxy"
Jul 02 22:09:06.675853 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:06.675736    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bfa0b5e-ac46-4503-a2d0-c0a679de6543" containerName="kserve-container"
Jul 02 22:09:06.681986 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:06.681965    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-28brx/must-gather-g4qp6"
Jul 02 22:09:06.685821 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:06.685328    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-28brx\"/\"default-dockercfg-4gx7c\""
Jul 02 22:09:06.685821 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:06.685548    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-28brx\"/\"kube-root-ca.crt\""
Jul 02 22:09:06.685821 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:06.685752    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-28brx\"/\"openshift-service-ca.crt\""
Jul 02 22:09:06.688705 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:06.688682    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-28brx/must-gather-g4qp6"]
Jul 02 22:09:06.793143 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:06.793082    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d41bc78e-9dab-428c-b6d5-1b2e1a849b0e-must-gather-output\") pod \"must-gather-g4qp6\" (UID: \"d41bc78e-9dab-428c-b6d5-1b2e1a849b0e\") " pod="openshift-must-gather-28brx/must-gather-g4qp6"
Jul 02 22:09:06.793293 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:06.793162    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvh7c\" (UniqueName: \"kubernetes.io/projected/d41bc78e-9dab-428c-b6d5-1b2e1a849b0e-kube-api-access-cvh7c\") pod \"must-gather-g4qp6\" (UID: \"d41bc78e-9dab-428c-b6d5-1b2e1a849b0e\") " pod="openshift-must-gather-28brx/must-gather-g4qp6"
Jul 02 22:09:06.893807 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:06.893768    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d41bc78e-9dab-428c-b6d5-1b2e1a849b0e-must-gather-output\") pod \"must-gather-g4qp6\" (UID: \"d41bc78e-9dab-428c-b6d5-1b2e1a849b0e\") " pod="openshift-must-gather-28brx/must-gather-g4qp6"
Jul 02 22:09:06.894009 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:06.893818    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvh7c\" (UniqueName: \"kubernetes.io/projected/d41bc78e-9dab-428c-b6d5-1b2e1a849b0e-kube-api-access-cvh7c\") pod \"must-gather-g4qp6\" (UID: \"d41bc78e-9dab-428c-b6d5-1b2e1a849b0e\") " pod="openshift-must-gather-28brx/must-gather-g4qp6"
Jul 02 22:09:06.894212 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:06.894187    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d41bc78e-9dab-428c-b6d5-1b2e1a849b0e-must-gather-output\") pod \"must-gather-g4qp6\" (UID: \"d41bc78e-9dab-428c-b6d5-1b2e1a849b0e\") " pod="openshift-must-gather-28brx/must-gather-g4qp6"
Jul 02 22:09:06.902052 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:06.902023    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvh7c\" (UniqueName: \"kubernetes.io/projected/d41bc78e-9dab-428c-b6d5-1b2e1a849b0e-kube-api-access-cvh7c\") pod \"must-gather-g4qp6\" (UID: \"d41bc78e-9dab-428c-b6d5-1b2e1a849b0e\") " pod="openshift-must-gather-28brx/must-gather-g4qp6"
Jul 02 22:09:06.993281 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:06.993255    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-28brx/must-gather-g4qp6"
Jul 02 22:09:07.123849 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:07.123826    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-28brx/must-gather-g4qp6"]
Jul 02 22:09:07.126124 ip-10-0-130-230 kubenswrapper[2564]: W0702 22:09:07.126084    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd41bc78e_9dab_428c_b6d5_1b2e1a849b0e.slice/crio-c127446ad854a9d7c822c6a087d5d85d8a5b87a71148a9483c057754d512bf4c WatchSource:0}: Error finding container c127446ad854a9d7c822c6a087d5d85d8a5b87a71148a9483c057754d512bf4c: Status 404 returned error can't find the container with id c127446ad854a9d7c822c6a087d5d85d8a5b87a71148a9483c057754d512bf4c
Jul 02 22:09:07.646787 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:07.646752    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-28brx/must-gather-g4qp6" event={"ID":"d41bc78e-9dab-428c-b6d5-1b2e1a849b0e","Type":"ContainerStarted","Data":"c127446ad854a9d7c822c6a087d5d85d8a5b87a71148a9483c057754d512bf4c"}
Jul 02 22:09:08.655175 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:08.655138    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-28brx/must-gather-g4qp6" event={"ID":"d41bc78e-9dab-428c-b6d5-1b2e1a849b0e","Type":"ContainerStarted","Data":"93a62781abb4eeb75662d5156ea300b795863e56ce87f24ecbff11b2fb6bb605"}
Jul 02 22:09:08.655608 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:08.655189    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-28brx/must-gather-g4qp6" event={"ID":"d41bc78e-9dab-428c-b6d5-1b2e1a849b0e","Type":"ContainerStarted","Data":"725b912ecebff2153807b8297510921bd73d214ad7087bdd75c8f1010f4cc113"}
Jul 02 22:09:08.671617 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:08.671550    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-28brx/must-gather-g4qp6" podStartSLOduration=2.029665985 podStartE2EDuration="2.671529712s" totalImagesPullingTime="641.863727ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 22:09:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 22:09:08.670692653 +0000 UTC m=+3009.507932701" watchObservedRunningTime="2026-07-02 22:09:08.671529712 +0000 UTC m=+3009.508769763"
Jul 02 22:09:09.128463 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:09.128430    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f8nf7_453c93ba-280d-45f1-8ade-994e6242ccc7/global-pull-secret-syncer/0.log"
Jul 02 22:09:09.331874 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:09.331839    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zd5dg_b988618a-9a30-40af-964a-6c68d01dacfb/konnectivity-agent/0.log"
Jul 02 22:09:09.382023 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:09.381952    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-230.ec2.internal_005b67a3b8a4751eb2ba150e034470d6/haproxy/0.log"
Jul 02 22:09:11.503389 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:11.503347    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-h8jrj/must-gather-dn4kl"]
Jul 02 22:09:11.503968 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:11.503927    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-h8jrj/must-gather-dn4kl" podUID="f1f31e3f-b483-427d-8477-26afdefe79dd" containerName="copy" containerID="cri-o://3ebd81b5d8510eb9576af3327aec0593cb1ec914111bf1becf7864e297c984fb" gracePeriod=2
Jul 02 22:09:11.509877 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:11.509844    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-h8jrj/must-gather-dn4kl"]
Jul 02 22:09:11.671466 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:11.671375    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-h8jrj_must-gather-dn4kl_f1f31e3f-b483-427d-8477-26afdefe79dd/copy/0.log"
Jul 02 22:09:11.672187 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:11.672036    2564 generic.go:356] "Generic (PLEG): container finished" podID="f1f31e3f-b483-427d-8477-26afdefe79dd" containerID="3ebd81b5d8510eb9576af3327aec0593cb1ec914111bf1becf7864e297c984fb" exitCode=143
Jul 02 22:09:11.879970 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:11.878930    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-h8jrj_must-gather-dn4kl_f1f31e3f-b483-427d-8477-26afdefe79dd/copy/0.log"
Jul 02 22:09:11.887134 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:11.886352    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h8jrj/must-gather-dn4kl"
Jul 02 22:09:11.891132 ip-10-0-130-230 kubenswrapper[2564]: E0702 22:09:11.888384    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-dn4kl\" is forbidden: User \"system:node:ip-10-0-130-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-h8jrj\": no relationship found between node 'ip-10-0-130-230.ec2.internal' and this object" podUID="f1f31e3f-b483-427d-8477-26afdefe79dd" pod="openshift-must-gather-h8jrj/must-gather-dn4kl"
Jul 02 22:09:11.950517 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:11.950479    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f1f31e3f-b483-427d-8477-26afdefe79dd-must-gather-output\") pod \"f1f31e3f-b483-427d-8477-26afdefe79dd\" (UID: \"f1f31e3f-b483-427d-8477-26afdefe79dd\") "
Jul 02 22:09:11.952061 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:11.952019    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1f31e3f-b483-427d-8477-26afdefe79dd-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "f1f31e3f-b483-427d-8477-26afdefe79dd" (UID: "f1f31e3f-b483-427d-8477-26afdefe79dd"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 22:09:11.953207 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:11.953178    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7r4k6\" (UniqueName: \"kubernetes.io/projected/f1f31e3f-b483-427d-8477-26afdefe79dd-kube-api-access-7r4k6\") pod \"f1f31e3f-b483-427d-8477-26afdefe79dd\" (UID: \"f1f31e3f-b483-427d-8477-26afdefe79dd\") "
Jul 02 22:09:11.954674 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:11.953880    2564 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f1f31e3f-b483-427d-8477-26afdefe79dd-must-gather-output\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 22:09:11.965714 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:11.964740    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1f31e3f-b483-427d-8477-26afdefe79dd-kube-api-access-7r4k6" (OuterVolumeSpecName: "kube-api-access-7r4k6") pod "f1f31e3f-b483-427d-8477-26afdefe79dd" (UID: "f1f31e3f-b483-427d-8477-26afdefe79dd"). InnerVolumeSpecName "kube-api-access-7r4k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 22:09:12.055462 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:12.055418    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r4k6\" (UniqueName: \"kubernetes.io/projected/f1f31e3f-b483-427d-8477-26afdefe79dd-kube-api-access-7r4k6\") on node \"ip-10-0-130-230.ec2.internal\" DevicePath \"\""
Jul 02 22:09:12.074277 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:12.074239    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1f31e3f-b483-427d-8477-26afdefe79dd" path="/var/lib/kubelet/pods/f1f31e3f-b483-427d-8477-26afdefe79dd/volumes"
Jul 02 22:09:12.679572 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:12.679545    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-h8jrj_must-gather-dn4kl_f1f31e3f-b483-427d-8477-26afdefe79dd/copy/0.log"
Jul 02 22:09:12.680652 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:12.680635    2564 scope.go:117] "RemoveContainer" containerID="3ebd81b5d8510eb9576af3327aec0593cb1ec914111bf1becf7864e297c984fb"
Jul 02 22:09:12.681083 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:12.680866    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h8jrj/must-gather-dn4kl"
Jul 02 22:09:12.695851 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:12.695567    2564 scope.go:117] "RemoveContainer" containerID="bd963c39f2ba053a29a43f0f9eec092e6876e22fd90bee4f69b638a1c621ff41"
Jul 02 22:09:13.680086 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:13.680006    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae/alertmanager/0.log"
Jul 02 22:09:13.738230 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:13.738199    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae/config-reloader/0.log"
Jul 02 22:09:13.793728 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:13.793699    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae/kube-rbac-proxy-web/0.log"
Jul 02 22:09:13.851882 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:13.851826    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae/kube-rbac-proxy/0.log"
Jul 02 22:09:13.910435 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:13.910407    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae/kube-rbac-proxy-metric/0.log"
Jul 02 22:09:13.976034 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:13.975948    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae/prom-label-proxy/0.log"
Jul 02 22:09:14.033597 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:14.033567    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fdb3f3b2-f215-464e-83a4-9ab4ca8d15ae/init-config-reloader/0.log"
Jul 02 22:09:14.130151 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:14.130093    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-4ghfd_4a25465f-d7ff-4356-8bb1-ab8ef7a166a7/kube-state-metrics/0.log"
Jul 02 22:09:14.155091 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:14.155057    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-4ghfd_4a25465f-d7ff-4356-8bb1-ab8ef7a166a7/kube-rbac-proxy-main/0.log"
Jul 02 22:09:14.184581 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:14.184554    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-4ghfd_4a25465f-d7ff-4356-8bb1-ab8ef7a166a7/kube-rbac-proxy-self/0.log"
Jul 02 22:09:14.248648 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:14.248599    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-vqqmh_1a350267-d9ef-4568-9cd2-027f8dbc7448/monitoring-plugin/0.log"
Jul 02 22:09:14.452065 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:14.452039    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mc95h_c00f52df-cbd9-4514-9124-c4bf250131b9/node-exporter/0.log"
Jul 02 22:09:14.487990 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:14.487962    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mc95h_c00f52df-cbd9-4514-9124-c4bf250131b9/kube-rbac-proxy/0.log"
Jul 02 22:09:14.523824 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:14.523737    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mc95h_c00f52df-cbd9-4514-9124-c4bf250131b9/init-textfile/0.log"
Jul 02 22:09:14.563341 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:14.563305    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-zr8d2_32e0b40a-797e-4f77-addc-87a3076aef35/kube-rbac-proxy-main/0.log"
Jul 02 22:09:14.591415 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:14.591385    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-zr8d2_32e0b40a-797e-4f77-addc-87a3076aef35/kube-rbac-proxy-self/0.log"
Jul 02 22:09:14.620549 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:14.620519    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-zr8d2_32e0b40a-797e-4f77-addc-87a3076aef35/openshift-state-metrics/0.log"
Jul 02 22:09:14.669237 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:14.669204    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f/prometheus/0.log"
Jul 02 22:09:14.689862 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:14.689831    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f/config-reloader/0.log"
Jul 02 22:09:14.720313 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:14.720285    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f/thanos-sidecar/0.log"
Jul 02 22:09:14.748534 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:14.748508    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f/kube-rbac-proxy-web/0.log"
Jul 02 22:09:14.772805 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:14.772774    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f/kube-rbac-proxy/0.log"
Jul 02 22:09:14.797015 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:14.796933    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f/kube-rbac-proxy-thanos/0.log"
Jul 02 22:09:14.821711 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:14.821673    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cbc8aee9-2e8e-497c-8bf0-e3c5b3bcf85f/init-config-reloader/0.log"
Jul 02 22:09:14.856186 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:14.856160    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-sxkx5_14f107b6-85d1-4377-912f-442b949a0b1b/prometheus-operator/0.log"
Jul 02 22:09:14.880415 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:14.880371    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-sxkx5_14f107b6-85d1-4377-912f-442b949a0b1b/kube-rbac-proxy/0.log"
Jul 02 22:09:14.957447 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:14.957395    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dc9c85ff5-grzn4_6599453e-5370-4a05-a765-52ad046d1955/telemeter-client/0.log"
Jul 02 22:09:14.995085 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:14.995045    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dc9c85ff5-grzn4_6599453e-5370-4a05-a765-52ad046d1955/reload/0.log"
Jul 02 22:09:15.036731 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:15.036698    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dc9c85ff5-grzn4_6599453e-5370-4a05-a765-52ad046d1955/kube-rbac-proxy/0.log"
Jul 02 22:09:15.104582 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:15.104506    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78949869bb-xq265_f95a85ad-d07b-4346-8f57-1afa13eb330e/thanos-query/0.log"
Jul 02 22:09:15.144397 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:15.144343    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78949869bb-xq265_f95a85ad-d07b-4346-8f57-1afa13eb330e/kube-rbac-proxy-web/0.log"
Jul 02 22:09:15.192312 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:15.192265    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78949869bb-xq265_f95a85ad-d07b-4346-8f57-1afa13eb330e/kube-rbac-proxy/0.log"
Jul 02 22:09:15.227332 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:15.227301    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78949869bb-xq265_f95a85ad-d07b-4346-8f57-1afa13eb330e/prom-label-proxy/0.log"
Jul 02 22:09:15.269329 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:15.269301    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78949869bb-xq265_f95a85ad-d07b-4346-8f57-1afa13eb330e/kube-rbac-proxy-rules/0.log"
Jul 02 22:09:15.305067 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:15.305025    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78949869bb-xq265_f95a85ad-d07b-4346-8f57-1afa13eb330e/kube-rbac-proxy-metrics/0.log"
Jul 02 22:09:17.346659 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:17.346626    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-28brx/perf-node-gather-daemonset-wskzt"]
Jul 02 22:09:17.347672 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:17.347644    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1f31e3f-b483-427d-8477-26afdefe79dd" containerName="gather"
Jul 02 22:09:17.347672 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:17.347670    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1f31e3f-b483-427d-8477-26afdefe79dd" containerName="gather"
Jul 02 22:09:17.347672 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:17.347691    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1f31e3f-b483-427d-8477-26afdefe79dd" containerName="copy"
Jul 02 22:09:17.347936 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:17.347699    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1f31e3f-b483-427d-8477-26afdefe79dd" containerName="copy"
Jul 02 22:09:17.347936 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:17.347771    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1f31e3f-b483-427d-8477-26afdefe79dd" containerName="copy"
Jul 02 22:09:17.347936 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:17.347782    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1f31e3f-b483-427d-8477-26afdefe79dd" containerName="gather"
Jul 02 22:09:17.350713 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:17.350690    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-28brx/perf-node-gather-daemonset-wskzt"
Jul 02 22:09:17.370475 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:17.370445    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-28brx/perf-node-gather-daemonset-wskzt"]
Jul 02 22:09:17.410378 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:17.408617    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb33a61c-e2b7-40fb-ac4e-e291783b0614-podres\") pod \"perf-node-gather-daemonset-wskzt\" (UID: \"cb33a61c-e2b7-40fb-ac4e-e291783b0614\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-wskzt"
Jul 02 22:09:17.410378 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:17.408720    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb33a61c-e2b7-40fb-ac4e-e291783b0614-proc\") pod \"perf-node-gather-daemonset-wskzt\" (UID: \"cb33a61c-e2b7-40fb-ac4e-e291783b0614\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-wskzt"
Jul 02 22:09:17.410378 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:17.408759    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb33a61c-e2b7-40fb-ac4e-e291783b0614-sys\") pod \"perf-node-gather-daemonset-wskzt\" (UID: \"cb33a61c-e2b7-40fb-ac4e-e291783b0614\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-wskzt"
Jul 02 22:09:17.410378 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:17.408790    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb33a61c-e2b7-40fb-ac4e-e291783b0614-lib-modules\") pod \"perf-node-gather-daemonset-wskzt\" (UID: \"cb33a61c-e2b7-40fb-ac4e-e291783b0614\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-wskzt"
Jul 02 22:09:17.410378 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:17.408818    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nx55\" (UniqueName: \"kubernetes.io/projected/cb33a61c-e2b7-40fb-ac4e-e291783b0614-kube-api-access-7nx55\") pod \"perf-node-gather-daemonset-wskzt\" (UID: \"cb33a61c-e2b7-40fb-ac4e-e291783b0614\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-wskzt"
Jul 02 22:09:17.509426 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:17.509377    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb33a61c-e2b7-40fb-ac4e-e291783b0614-podres\") pod \"perf-node-gather-daemonset-wskzt\" (UID: \"cb33a61c-e2b7-40fb-ac4e-e291783b0614\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-wskzt"
Jul 02 22:09:17.509634 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:17.509506    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb33a61c-e2b7-40fb-ac4e-e291783b0614-proc\") pod \"perf-node-gather-daemonset-wskzt\" (UID: \"cb33a61c-e2b7-40fb-ac4e-e291783b0614\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-wskzt"
Jul 02 22:09:17.509634 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:17.509548    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb33a61c-e2b7-40fb-ac4e-e291783b0614-sys\") pod \"perf-node-gather-daemonset-wskzt\" (UID: \"cb33a61c-e2b7-40fb-ac4e-e291783b0614\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-wskzt"
Jul 02 22:09:17.509634 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:17.509581    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb33a61c-e2b7-40fb-ac4e-e291783b0614-lib-modules\") pod \"perf-node-gather-daemonset-wskzt\" (UID: \"cb33a61c-e2b7-40fb-ac4e-e291783b0614\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-wskzt"
Jul 02 22:09:17.509634 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:17.509609    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nx55\" (UniqueName: \"kubernetes.io/projected/cb33a61c-e2b7-40fb-ac4e-e291783b0614-kube-api-access-7nx55\") pod \"perf-node-gather-daemonset-wskzt\" (UID: \"cb33a61c-e2b7-40fb-ac4e-e291783b0614\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-wskzt"
Jul 02 22:09:17.509883 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:17.509648    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb33a61c-e2b7-40fb-ac4e-e291783b0614-proc\") pod \"perf-node-gather-daemonset-wskzt\" (UID: \"cb33a61c-e2b7-40fb-ac4e-e291783b0614\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-wskzt"
Jul 02 22:09:17.509883 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:17.509742    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb33a61c-e2b7-40fb-ac4e-e291783b0614-sys\") pod \"perf-node-gather-daemonset-wskzt\" (UID: \"cb33a61c-e2b7-40fb-ac4e-e291783b0614\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-wskzt"
Jul 02 22:09:17.509883 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:17.509837    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb33a61c-e2b7-40fb-ac4e-e291783b0614-lib-modules\") pod \"perf-node-gather-daemonset-wskzt\" (UID: \"cb33a61c-e2b7-40fb-ac4e-e291783b0614\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-wskzt"
Jul 02 22:09:17.509883 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:17.509869    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb33a61c-e2b7-40fb-ac4e-e291783b0614-podres\") pod \"perf-node-gather-daemonset-wskzt\" (UID: \"cb33a61c-e2b7-40fb-ac4e-e291783b0614\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-wskzt"
Jul 02 22:09:17.518505 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:17.518479    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nx55\" (UniqueName: \"kubernetes.io/projected/cb33a61c-e2b7-40fb-ac4e-e291783b0614-kube-api-access-7nx55\") pod \"perf-node-gather-daemonset-wskzt\" (UID: \"cb33a61c-e2b7-40fb-ac4e-e291783b0614\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-wskzt"
Jul 02 22:09:17.664529 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:17.664444    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-28brx/perf-node-gather-daemonset-wskzt"
Jul 02 22:09:17.838423 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:17.837984    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-28brx/perf-node-gather-daemonset-wskzt"]
Jul 02 22:09:17.840801 ip-10-0-130-230 kubenswrapper[2564]: W0702 22:09:17.840764    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcb33a61c_e2b7_40fb_ac4e_e291783b0614.slice/crio-5ecc07031b50646f986198caba4ea33b672aca62005bed4e86f7e7237517707b WatchSource:0}: Error finding container 5ecc07031b50646f986198caba4ea33b672aca62005bed4e86f7e7237517707b: Status 404 returned error can't find the container with id 5ecc07031b50646f986198caba4ea33b672aca62005bed4e86f7e7237517707b
Jul 02 22:09:18.726701 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:18.726668    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-28brx/perf-node-gather-daemonset-wskzt" event={"ID":"cb33a61c-e2b7-40fb-ac4e-e291783b0614","Type":"ContainerStarted","Data":"e2bd8b84ffdf58a4a49d1ddba1a11faafbd76b83763712bf7fcdda5b77c43cb5"}
Jul 02 22:09:18.727136 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:18.726726    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-28brx/perf-node-gather-daemonset-wskzt" event={"ID":"cb33a61c-e2b7-40fb-ac4e-e291783b0614","Type":"ContainerStarted","Data":"5ecc07031b50646f986198caba4ea33b672aca62005bed4e86f7e7237517707b"}
Jul 02 22:09:18.727136 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:18.726767    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-28brx/perf-node-gather-daemonset-wskzt"
Jul 02 22:09:18.745511 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:18.745441    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-28brx/perf-node-gather-daemonset-wskzt" podStartSLOduration=1.745420058 podStartE2EDuration="1.745420058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 22:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 22:09:18.742743983 +0000 UTC m=+3019.579984033" watchObservedRunningTime="2026-07-02 22:09:18.745420058 +0000 UTC m=+3019.582660106"
Jul 02 22:09:18.920022 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:18.919990    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tcvc8_4e7ae876-0cf7-4171-a256-8205bbd810f7/dns/0.log"
Jul 02 22:09:18.943828 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:18.943798    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tcvc8_4e7ae876-0cf7-4171-a256-8205bbd810f7/kube-rbac-proxy/0.log"
Jul 02 22:09:19.017131 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:19.017087    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6k66x_188c6c1c-be85-48e2-9cbb-9ab9722bb2b4/dns-node-resolver/0.log"
Jul 02 22:09:19.574401 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:19.574368    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ctb57_426d3178-3121-44cc-b885-d8b7a3b902a9/node-ca/0.log"
Jul 02 22:09:20.785415 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:20.785381    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4scqp_65655d15-640b-4caf-8715-750aeb8b1887/serve-healthcheck-canary/0.log"
Jul 02 22:09:21.275728 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:21.275702    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kjzfh_ecf34a8b-a6e9-48cf-9097-74ac6ab7c362/kube-rbac-proxy/0.log"
Jul 02 22:09:21.298011 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:21.297984    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kjzfh_ecf34a8b-a6e9-48cf-9097-74ac6ab7c362/exporter/0.log"
Jul 02 22:09:21.320159 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:21.320127    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kjzfh_ecf34a8b-a6e9-48cf-9097-74ac6ab7c362/extractor/0.log"
Jul 02 22:09:23.592757 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:23.592729    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-67d95fdc48-dv69s_35aebdb0-c85d-4cb4-ae73-0451f2992d50/manager/0.log"
Jul 02 22:09:23.808844 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:23.808814    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-2lrcz_cf41621d-1fee-44e5-bd4f-0a077bf581fb/s3-init/0.log"
Jul 02 22:09:23.827387 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:23.827353    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-2lrcz_cf41621d-1fee-44e5-bd4f-0a077bf581fb/download-hf-model/0.log"
Jul 02 22:09:24.748950 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:24.748065    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-28brx/perf-node-gather-daemonset-wskzt"
Jul 02 22:09:29.490433 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:29.490405    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tm8gq_b0f5fcb9-715f-4dc1-915b-ae229225e5c2/kube-multus-additional-cni-plugins/0.log"
Jul 02 22:09:29.514128 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:29.514088    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tm8gq_b0f5fcb9-715f-4dc1-915b-ae229225e5c2/egress-router-binary-copy/0.log"
Jul 02 22:09:29.535907 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:29.535877    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tm8gq_b0f5fcb9-715f-4dc1-915b-ae229225e5c2/cni-plugins/0.log"
Jul 02 22:09:29.560022 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:29.559995    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tm8gq_b0f5fcb9-715f-4dc1-915b-ae229225e5c2/bond-cni-plugin/0.log"
Jul 02 22:09:29.583017 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:29.582990    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tm8gq_b0f5fcb9-715f-4dc1-915b-ae229225e5c2/routeoverride-cni/0.log"
Jul 02 22:09:29.606243 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:29.606218    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tm8gq_b0f5fcb9-715f-4dc1-915b-ae229225e5c2/whereabouts-cni-bincopy/0.log"
Jul 02 22:09:29.630886 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:29.630860    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tm8gq_b0f5fcb9-715f-4dc1-915b-ae229225e5c2/whereabouts-cni/0.log"
Jul 02 22:09:29.872581 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:29.872507    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-cg58w_dcd2b06f-76c0-4b1c-af0d-03f5d780e9dd/kube-multus/0.log"
Jul 02 22:09:29.919497 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:29.919470    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7rzx4_7fc43c39-ea1c-47da-b658-7834776917a8/network-metrics-daemon/0.log"
Jul 02 22:09:29.940583 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:29.940548    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7rzx4_7fc43c39-ea1c-47da-b658-7834776917a8/kube-rbac-proxy/0.log"
Jul 02 22:09:31.064966 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:31.064926    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hlmv7_53db7e45-3dab-4292-930c-8628a2f51800/ovn-controller/0.log"
Jul 02 22:09:31.084880 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:31.084855    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hlmv7_53db7e45-3dab-4292-930c-8628a2f51800/ovn-acl-logging/0.log"
Jul 02 22:09:31.109380 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:31.109353    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hlmv7_53db7e45-3dab-4292-930c-8628a2f51800/kube-rbac-proxy-node/0.log"
Jul 02 22:09:31.131692 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:31.131657    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hlmv7_53db7e45-3dab-4292-930c-8628a2f51800/kube-rbac-proxy-ovn-metrics/0.log"
Jul 02 22:09:31.153725 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:31.153689    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hlmv7_53db7e45-3dab-4292-930c-8628a2f51800/northd/0.log"
Jul 02 22:09:31.177674 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:31.177647    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hlmv7_53db7e45-3dab-4292-930c-8628a2f51800/nbdb/0.log"
Jul 02 22:09:31.202066 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:31.202040    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hlmv7_53db7e45-3dab-4292-930c-8628a2f51800/sbdb/0.log"
Jul 02 22:09:31.252532 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:31.252502    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hlmv7_53db7e45-3dab-4292-930c-8628a2f51800/ovnkube-controller/0.log"
Jul 02 22:09:32.495833 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:32.495808    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-65mqk_238eb474-294d-49e5-8086-6b9cf9711ac6/network-check-target-container/0.log"
Jul 02 22:09:33.657687 ip-10-0-130-230 kubenswrapper[2564]: I0702 22:09:33.657617    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-dhs4b_39ea2cc9-ac41-4474-99dd-1ce8c95d8b87/iptables-alerter/0.log"